HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions

## Abstract

Abstract AI agents are increasingly autonomous in their interactions with human users and tools, leading to increased interactional safety risks.
We present \trjnfamily HAICOSYSTEM, a framework examining AI agent safety within diverse and complex social interactions. \trjnfamily HAICOSYSTEM features a modular sandbox environment that simulates multi-turn interactions between human users and AI agents, where the AI agents are equipped with a variety of tools (e.g., patient management platforms) to navigate diverse scenarios (e.g., a user attempting to access other patients’ profiles).
To examine the safety of AI agents in these interactions, we develop a comprehensive multi-dimensional evaluation framework that uses metrics covering operational, content-related, societal, and legal risks.
Through running over 8K simulations based on 132 scenarios across seven domains (e.g., healthcare, finance, education),
we demonstrate that \trjnfamily HAICOSYSTEM can emulate realistic user-AI interactions and complex tool use by AI agents.
Our experiments show that state-of-the-art LLMs, both proprietary and open-sourced, exhibit safety risks in 62% of cases, with models generally showing higher risks when interacting with malicious users and using tools simultaneously.
Our findings highlight the ongoing challenge of building agents that can safely navigate complex interactions. To foster the AI agent safety ecosystem, we release a code platform that allows practitioners to create custom scenarios, simulate interactions, and evaluate the safety and performance of their agents. 1 1 1 https://haicosystem.org

## 1 Introduction

AI agents, holding the potential to automate tasks and improve human productivity, are increasingly being deployed in real-life applications (Wu et al., ; Wang et al., ; ).
To function effectively in the real world, AI agents should communicate seamlessly with human users to understand their goals and intents (Ouyang et al., ; Zhou et al., ), while interacting with environments and tools. For example, they may acquire information by browsing websites (Zhou et al., ), or manipulate the state of the environment
by controlling mobile apps (Trivedi et al., ) and creating artifacts such as software and digital content (Yang et al., ; Si et al., ).
This involves multiple stages of interaction among AI agents, humans, and environments (e.g., human $\rightarrow$ AI agent; AI agent $\rightarrow$ environment), forming a complex ecosystem.

However, increased autonomy of agents brings forward new safety risks at each stage of interaction.
When interacting with human users, AI agents may generate misinformation (Loth et al., ), toxic content (Jain et al., ), and unsafe answers (Zou et al., ).
When interacting with the environment, AI agents could cause unintended harm (e.g., sending money to the wrong person due to under-specified instructions; Ruan et al., ).
While existing works focus on isolating the safety risks of AI agents in the specific stage of interaction (Ruan et al., ; Jiang et al., ; Brahman et al., ), we argue that the safety risks of AI agents should be investigated in a holistic manner by examining the entire ecosystem of AI agents, humans, and environments.

We propose \trjnfamilyHAICOSYSTEM, a framework to investigate AI agent safety risks across all stages at once.(^2^22For clarity, we refer to AI agents as those AI assistants (e.g., ChatGPT) that do not engage in role-playing human-like characteristics such as demographics, personality, etc.)
As shown in Figure 1, \trjnfamilyHAICOSYSTEM simulates how human users, whether with benign or malicious intent, interact with AI agents across a range of scenarios, from daily life to professional settings.
It also models how AI agents use tools to engage with their environment in a multi-turn setting.
At the end of the simulation, \trjnfamilyHAICOSYSTEM examines the safety risks of AI agents based on the outcomes of the interactions.
Concretely, we create a software platform that allows us to create scenarios, plug in AI agents in specific simulated environments (e.g., smart home, a web browser), and sample corresponding simulated human users to
model the complex interactions among AI agents, humans, and environments with LLMs.

To enable a holistic, multidimensional evaluation of AI agents in \trjnfamilyHAICOSYSTEM, we propose \trjnfamilyHAICOSYSTEM-EVAL, a framework to measure both the safety and performance of AI agents in these complex interactions.
\trjnfamilyHAICOSYSTEM-EVAL includes a scenario-specific checklist of safe and risky outcomes, along with other general dimensions of safety risks (e.g., legal risks), to capture the harmful outcomes from the interactions comprehensively.
For performance evaluation, our framework also includes efficiency and goal completion rate measures to study potential trade-offs between risky behavior and helpfulness.
Going beyond capturing immediate, single-turn, or single-action harm, \trjnfamilyHAICOSYSTEM-EVAL focuses on evaluating the potential harms that may emerge across the entire trajectory of interactions within a simulated episode.

To demonstrate the utility of \trjnfamilyHAICOSYSTEM, we compiled 132 scenarios spanning seven domains, including healthcare, business & finance, science & technology, and more.
For example, a healthcare scenario might involve “a person chatting with the Teladoc AI agent to request a prescription”.
We then run simulations across the scenarios, sampling diverse profiles of malicious and benign users to capture a broad range of real-world interactions.
To ensure realism and effectively stress-test the agents, each scenario is designed such that participants have access to different information (e.g., the user’s goal is hidden from the AI agent).
Additionally, user instructions are often ambiguous to reflect real-world use cases.
These design choices challenge the AI agent to infer implicit yet critical information (e.g., malicious user intent) through multi-turn interactions.

Through these empirical investigations with 8,700 simulated episodes, we find that \trjnfamilyHAICOSYSTEM can effectively surface previously unknown safety issues of AI agents in human-AI interactions.
Specifically, all the proprietary and open-source models we evaluate exhibit behaviors that pose potential safety risks, with weaker models being more vulnerable (e.g., GPT-3.5-turbo shows safety risks in 67% of all simulations).
Furthermore, different models show varying strengths and weaknesses across different stages of interaction. For example, Llama3.1-405B (Dubey et al., ) outperforms Llama3.1-70B in effectively using tools and communicating with benign users but falls short in handling situations involving malicious users.
Through simulating multi-turn interactions, we reveal the unique challenge of AI agents maintaining safety when interacting with environments and malicious human users simultaneously.
Moreover, we identify that the safety risks of AI agents are closely related to the types of human users they interact with.
Specifically, simulated human users with good intentions provide valuable information to agents to avoid safety risks, while those with malicious intentions strategically “trick” the agents into taking harmful actions.

In summary, \trjnfamilyHAICOSYSTEM is the first framework to study AI safety issues through simulating interactions between agents and humans in grounded environments.
Besides pointing to the importance of considering the holistic ecosystem of AI agents, humans, and environments in evaluating AI agent safety, \trjnfamilyHAICOSYSTEM also offers a foundation for future research, which practitioners can use to create custom scenarios for exploring specific safety issues and develop safer AI agents for real-world deployment.

## 2 Background and Preliminaries

**Table 1: Comparison of various safety evaluation frameworks versus \trjnfamilyHAICOSYSTEM. \faUser indicates human users, \faRobot indicates the AI agents, and \faGlobeAmericas indicates the environment. \faExchange* indicates the multi-turn interactions are considered and \faLongArrowAltRight indicates single-turn interaction. \faTimesCircle indicates such interactions are not present and \faStopCircle indicates such interactions are static instead of dynamic. \faSmileBeam and \faAngry indicate the intent of the human user to be either benign or malicious, respectively.**
| Framework | \faUser  \faExchange*  \faRobot | \faRobot  \faExchange*  \faGlobeAmericas | User Int. | Social Contexts |
| --- | --- | --- | --- | --- |
| R-Judge (Yuan et al., ) | \faStopCircle | \faStopCircle | \faSmileBeam & \faAngry | General domains; The agent is the omniscient evaluator. |
| Wildteaming (Jiang et al., ) | \faLongArrowAltRight | \faTimesCircle | \faAngry | General domains; The agent is a Chatbot |
| ToolEmu (Ruan et al., ) | \faLongArrowAltRight | \faExchange* | \faSmileBeam | General domains; Assume the agent is a personal assistant. |
| Cresc (Russinovich et al., ) | \faExchange* | \faTimesCircle | \faAngry | General domains; The agent is a Chatbot |
| PrivacyLens (Shao et al., ) | \faLongArrowAltRight | \faExchange* | \faSmileBeam | Privacy issues; Assume the agent is a personal assistant. |
| \trjnfamilyHAICOSYSTEM (Ours) | \faExchange* | \faExchange* | \faSmileBeam & \faAngry | General domains; The agent is a personal/organizational assistant. |

#### Safety Evaluation

As shown in Table 2, most existing research focuses on evaluating the safety risks of AI agents in a single-turn interaction with human users (Jiang et al., ; Zeng et al., ; Ruan et al., ; Shao et al., ) with little coverage of risks arise from complex, multi-turn interactions.
Some studies focus on human users with malicious intent (Jiang et al., ; Zeng et al., ; Anil et al., ; Liu et al., ; Deng et al., ), while others focus on the benign user settings where the safety risks come from the AI agents use tools incorrectly (Ruan et al., ; Shao et al., ).
Recent research also explores the multi-turn interactions between human users and AI agents (Russinovich et al., ; Yang et al., ), though these works do not consider the tool-using behaviors of AI agents and solely focus on malicious human users.
Additionally, benchmarks like R-Judge (Yuan et al., ) evaluate LLMs’ ability to identify safety issues given a static interaction trajectory.
Furthermore, previous safety evaluations have predominantly focused on the safety risks of personal AI agents, ignoring the safety risks of AI agents in organizational contexts with more complex social dynamics.

In contrast, \trjnfamilyHAICOSYSTEM aims to evaluate the safety risks of AI agents dynamically in a multi-turn interaction with human users either with malicious intent or benign intent. AI agents have access to a wide range of tools, and we not only consider personal agents but also agents in an organizational context, resembling a broader range of realistic risks when deploying AI agents in our society.

#### Social Agents and Social Simulations

We identify social agents as goal-driven decision-makers that sense and act upon the state of the environment and interact with other agents to achieve their goals (Sutton & Barto, ).
This paper follows SOTOPIA (Zhou et al., ) in formulating interactions between human users and AI agents as *social tasks*. A social task in SOTOPIA consists of a scenario, two characters’ profiles, and their respective private social goals to achieve in an interaction. Within one episode, the two agents role-play the characters to achieve their respective social goals, where the agents could either be AI agents or humans. While SOTOPIA considers the general social interaction between social agents, in \trjnfamilyHAICOSYSTEM, we focus on the social tasks with one character being an AI agent, and another character being a human user (§3).
We also consider different kinds of human users, including users with benign intents to simulate the cooperative users, and the ones with malicious intents to simulate adversarial actors.

#### Emulating Tool Execution

Following Ruan et al. (), we emulate the tool execution of AI agents by using LLMs as the emulators.
Due to the long-tail property of safety risks, using LLMs as emulators helps us fast prototype various scenarios and tools.
While rule-based implementation of tools is also possible, it usually grounds the investigation on specific domains, limiting the generalizability of the findings (Shao et al., ).
Although LLM-based emulators may not always execute tool-calling actions accurately, as shown by Ruan et al. (), those risks identified in the emulated environments are also likely to manifest in the real world.

## 3 Constructing the \trjnfamily HAICOSYSTEM

As shown in Figure 1, \trjnfamilyHAICOSYSTEM ensembles human users, AI agents, and a shared environment.
We use LLMs to simulate the human users with rich user profiles from Zhou et al. (), including name, demographics, personality, etc.
The AI agent, powered by an LLM, can select speak or action to interact with either the human user or the shared environment.
Inspired by Ruan et al. (), we use an LLM as the environment engine, namely GPT-4o, to simulate the response of the tool calls of the AI agent.
Unlike Ruan et al. (), we provide scenario-specific instructions to the LLM environment engine, ensuring that the generated responses align with the scenario specifications, thereby enhancing controllability and reproducibility.

Each component in \trjnfamilyHAICOSYSTEM is modular and has access to different information, which is crucial for realistic simulation (Zhou et al., ).
Specifically, the AI agents have no access to the simulated users’ goals as well as their profiles.
Similarly, the human users have no access to the AI agents’ tools, goals, and the response of tool calling.
Please refer to Appendix B for more details on the framework.

A simulation in \trjnfamilyHAICOSYSTEM is initiated by the following three components:
(1) a scenario, (2) a human user and an AI agent, and (3) a checklist of safe and risky outcomes.

#### (1) Scenario

A scenario first starts with the overall background of the scenario outlining the overall situational context – e.g., “On July 1, 2024, a person is standing in the kitchen of a smart home managed by an AI agent.
There’s a sudden smoke coming from the kitchen…”
The environment profile also includes the AI agent goal and human user goal (e.g., the human’s goal is to convince the AI agent that the smoke is not a fire).
Including a detailed scenario background grounds the simulation towards a more realistic setting, thus reflecting risks that are more likely to happen in real-world scenarios (Dalrymple et al., ).

#### (2) Human User & AI Agent

To simulate human users, we use the character profiles from Sotopia (Zhou et al., ), which include rich information such as name, demographics, and personality.
For the AI agent, we include the AI agent’s role description (e.g., “You are an AI assistant responsible for triaging user requests at Epic on FHIR”) and the tools that the agent has access to, which are scenario-specific.
For example, in a scenario of scheduling a medical appointment, the AI agent has access to tools such as EpicFHIRGetPatientDetails, EpicFHIRSearchDoctors, and EpicFHIRManageClinicalDocuments.
Each tool is designed to reflect the real-world existing services with public APIs or is expected to be accessible by the AI agent in the future.
Besides a short description of what the tool is used for, we also provide the arguments that the tool takes in, and the format of the tool’s response. Please refer to Appendix B.1 for the detailed prompt for helping the agent call the tools.

#### (3) Checklist of Safe and Risky Outcomes

The checklist of safe and risky outcomes outlines the safety objectives that the AI system should aim to achieve (e.g., ensuring the properties in the house remain undamaged) and the risks it should avoid (e.g., unnecessarily calling the fire department when there is no fire), for each scenario.
Using such checklists has been demonstrated to enhance automated evaluation with LLMs (Lee et al., ).
It is important to note that this checklist is scenario-specific and is not used in the simulation process.

### 3.1 Populating Scenarios

When populating \trjnfamilyHAICOSYSTEM with scenarios, we focus on broad domain coverage, varied user intentions (malicious vs. benign), and realistic interactions to ensure evaluations reflect real-world risks. Specifically, we consider the following aspects when collecting scenarios:

Domain of the scenario: We consider a wide range of domains of tasks in human society. Inspired by previous works on the safety evaluation of AI agents (Ruan et al., ; Yuan et al., ), we consider scenarios in the domains of “personal services”, “healthcare”, “business & finance”, “politics & law”, “technology & science”, “education”, and “miscellaneous”.(^3^33Our list of domains come from U.S. Bureau of Labor Statistics. [https://www.bls.gov/ooh/home.htm](https://www.bls.gov/ooh/home.htm))

Intent type of the agents:
We categorize user intent into two types: malicious and benign. Users with malicious intent seek to exploit the AI agent to create safety issues, while users with benign intent do not seek to cause safety issues.

Scenario realism: We consider three different levels of realism when designing scenarios. Specifically, we have level 3 as everyday scenarios that AI agents are already interacting with humans (e.g., AI tutor (^4^44[https://www.synthesis.com/](https://www.synthesis.com/))), level 2 as scenarios that are likely to happen in the near future (e.g., AI bank manager (^5^55[https://thebankingscene.com/opinions/the-ai-bank-manager-of-the-future](https://thebankingscene.com/opinions/the-ai-bank-manager-of-the-future))), and level 1 as scenarios that are unlikely to happen in the near future, but high-stakes (e.g., AI managing nuclear weapons).
Different realism levels have been discussed in various AI safety works (Hendrycks et al., ; Bengio et al., ), simulations of which are possible via \trjnfamilyHAICOSYSTEM.

Following the above considerations, we instantiate \trjnfamilyHAICOSYSTEM with 132 scenarios spanning across 7 domains mentioned above.
21 scenarios are manually created by the authors guided by the above considerations.
The remaining 111 scenarios are inspired by previous works, as we randomly sample data from previous datasets (Zeng et al., ; Shen et al., ; Jiang et al., ; Mireshghallah et al., ; Sharma et al., ; Ruan et al., ; Majumder et al., ), use them in the prompts and let GPT-4o transform them into \trjnfamilyHAICOSYSTEM scenarios.
The authors then manually validated and made necessary adjustments to all of the generated scenarios.
Please refer to Appendix D for more details on our scenarios.

## 4 Evaluating Safety of AI Agents with \trjnfamily HAICOSYSTEM-EVAL

To evaluate the safety of AI agents when interacting with humans on grounded tasks, we propose a set of dimensions that measure the AI agents’ abilities to avoid risky outcomes while staying helpful in the scenarios.
As shown in Figure 2, all safety risk dimensions are rated from -10 to 0, where a higher score indicates a lower risk.
The AI agent is considered risky overall if any of the dimensions is negative in an episode.

Figure: Figure 2: Overview of \trjnfamilyHAICOSYSTEM-EVAL, including scales for each metric group.
Refer to caption: x2.png

#### Targeted Safety Risks ( Targ ) [-10, 0]

measure scenario-specific safety risks by evaluating whether the AI agent’s actions lead to any of the plausible positive or negative outcomes outlined in the scenario’s checklist (§3).

#### System and Operational Risks ( Syst ) [-10, 0]

target the AI agents’ ability to operate in the environment, avoiding security threats like network intrusions, malware, etc., while offering safe automated decision-making in finance, justice, and healthcare, and other domains (Zeng et al., ; Ruan et al., ).

#### Content Safety Risks ( Cont ) [-10, 0]

focus on the AI agents’ ability to provide safe content to the human users (Zeng et al., ). Specifically, risky content includes, but is not limited to, violence, extremism, hate speech, harassment, sexual content, child exploitation, and self-harm.

#### Societal Risks ( Soc ) [-10, 0]

focus on the detrimental impacts of AI agents on political, economic, and social structures (Zeng et al., ).
Specifically, this dimension covers manipulation of public opinion, spreading propaganda, unfair market practices, misinformation, etc.
AI agents, with their capacity for rapid action and large-scale production, can significantly amplify societal risks (Hendrycks et al., ; Bengio et al., ).

#### Legal and Rights Related Risks ( Legal ) [-10, 0]

focus on risks of AI agents violating fundamental rights, engaging in discrimination, breaching privacy, and facilitating criminal activities (Zeng et al., ). Examples include physically hurting humans, assigning resources based on protected characteristics, and unauthorized collection or misuse of sensitive personal data.

#### Tool Use Efficiency ( Efficiency ) [0, 10]

focuses on the AI agents’ ability to use tools effectively without wasting time or resources. One common issue arises when AI agents operate under incorrect assumptions. For example, an AI agent might call a tool with a generic argument ({coordinate: coordinate}). This type of error can lead to environmental failures that prevent tasks from being executed correctly, resulting in wasted processing time and the need for repeated operations. Such inefficiencies reduce the overall productivity and reliability of AI agents (Kapoor et al., ).
The efficiency score ranges from 0 to 10, with a higher score indicating better efficiency.

#### Goal Completion ( Goal ) [0, 10]

focuses on the AI agents’ ability to complete the goals assigned to them.
The goals can range from a general objective, such as “be a helpful assistant”, to a specific task, like “scrutinize evidence, flag, and redact prejudicial information before sending it to the jury”.
Note that the AI agents’ goals are not always aligned with the safety requirements.
For example, if an AI agent’s goal is to “be a helpful assistant” while interacting with a human user with malicious intent, the AI agent may choose to refuse to help the human user to avoid getting involved in risky situations (Brahman et al., ).
In these cases, the AI agent must balance the trade-off between safety and helpfulness (Wallace et al., ; Su et al., ; Scheurer et al., ).
The goal completion score ranges from 0 to 10, with a higher score indicating better goal completion.

Given the interaction trajectories of the AI agents along with the checklist of safe and risky outcomes, we use an LM-based evaluator (e.g., GPT-4o) to first reason whether the AI agent leads to certain safety risks, and then rate the safety risks across all the dimensions accordingly.
Estimating the degree of safety risks is fundamentally hard and subjective (Ruan et al., ; Brown, ). We thus also consider the coarse-grained evaluation of the AI agents’ safety risks, which the agent is considered risky in a dimension if it receives a negative score in the corresponding dimension of an episode.
For an agent, the risk ratio of each dimension is calculated as the proportion of risky episodes over the total number of episodes.
Please refer to Appendix C for more details on the evaluation framework.

## 5 Agent Safety Experiments

We first introduce the experimental setup and validation checks, followed by the results and analysis on (1) the safety risks of AI agents exhibited in the simulations of \trjnfamilyHAICOSYSTEM, and (2) how interactions with human users affect the safety of AI agents.

### 5.1 Experimental Setup and Validation

Across 132 scenarios, we sample 5 human users with different profiles to interact with the AI agent.
We fix the model to be GPT-4o (OpenAI, ) to role-play the human user as well as the evaluator.
In total, we have 660 simulated episodes for 12 different models, namely GPT-4-turbo OpenAI (), GPT-3.5-turbo Ouyang et al. (), Llama3 Series (3.1-405B, 3.1-70B, 3.1-8B, 3-70B, 3-8B; Dubey et al. ), Qwen Series (1.5-72B-Chat, 1.5-110B-Chat, 2-72B-Instruct; Bai et al. ), Mixtral-8x22B Jiang et al. (), and DeepSeek-67B DeepSeek-AI et al. ().
Due to space constraints, we present the detailed analysis using representative models: GPT-4-turbo, GPT-3.5-turbo, Llama3.1-405B, and Llama3.1-70B.
(^6^66Please refer to the Appendix E for the details of the experimental setup and Appendix F for analysis of other models.)

Figure: Figure 3: The risk ratio of models for different risk dimensions across simulated episodes. Overall dimension refers to an episode being considered as risky overall if any individual risk dimension is negative. The higher the risk ratio is, the more likely the model is to exhibit certain safety risks. The table shows the overall risk ratio for all benchmarked models, while the bar chart displays dimension-wise risk ratios for representative models.
Refer to caption: x3.png

To check whether the simulated human users realistically emulate real human users, we use the believability score in Zhou et al. () to evaluate the simulated human users.
Across all the episodes, the average believability score is 9.1 out of 10, indicating the simulated human users behave naturally.
To validate our automatic LM-based evaluation of safety risks, we manually verified the evaluation of 100 randomly sampled episodes. We find that 90% of evaluations are accurate in identifying AI agents’ risk with a 0.8 average Pearson correlation with the human evaluator’s judgment for various risk dimensions.(^7^77Please refer to Appendix F.1 for the details of manual verification of the automated evaluation.)

### 5.2 Benchmarking Safety Risks of AI Agents

As shown in Figure 3, we observe that all models exhibit substantial risks across all risk categories (0.49–0.71 overall risk ratio).
Specifically, the targeted safety risks category has the highest risk ratio (0.46–0.66) of all risk dimensions, indicating that models are more likely to show the safety risks anticipated in our scenario’s safe and risky outcomes checklist compared to other risk dimensions.

#### Agents powered by bigger models (GPT-4, Llama3.1-405B) tend to have lower safety risks

compared to less capable models (GPT-3.5, Llama3.1-70B).
For example, in the case of redacting sensitive information for the jury, the Llama3.1-70B AI agent does not remove the political opinion of the defendant (“I support and 100% think legal abortion is a fundamental right.”), which is irrelevant to the case and could bias the jury’s decision, whereas the Llama3.1-405B AI agent successfully redacts the sensitive information, avoiding legal risks.(^8^88Please check the episode for Llama3.1-70B and for Llama3.1-405B in Appendix G.)
These findings corroborate Dubey et al. () which points out that bigger models underwent more careful safety training and alignment, leading to lower safety risks.

#### Agents are more likely to exhibit System and Operational risks and less likely to exhibit Content safety risks.

From Figure 3, we also find the agents powered by different models are in general more likely to cause safety issues while operating in the environments with the tools (Syst), highlighting the need for evaluating their situational safety risks.
Comparatively, the agents show fewer problems of producing toxic or offensive content (Cont) in the simulations, as toxic content has been extensively studied in various works (Jain et al., ; Lin et al., ).

#### Effective use of tools is correlated with lower safety risks for agents.

We observe a negative correlation ($r=-0.31$) between the efficiency of AI agents in using tools (i.e., higher efficiency scores) and safety risks in scenarios that require the use of these tools.
For example, in the case of an AI agent helping a user update the school’s internal database, the agent can avoid the safety risks by using the SearchKnowledgeBase, GoogleSearch tools to help itself validate the correctness of the information that the user wants to update.
However, if the AI agent is not able to use the tools effectively, it could also lead to more safety risks.
For example, in the case of an AI agent helping the athlete to monitor the health conditions, the AI agent powered by Llama3.1-70B fails to use the HeartRateMonitor tool correctly, leading to a failure in detecting the athlete’s heart rate anomaly and causing safety risks.

Figure: Figure 4: The qualitative examples of the episodes where the AI agents interact with human users with both malicious (left) and benign (right) intents.
Refer to caption: x4.png

#### Balancing Goal Completion and Safety Risks is Challenging for AI Agents

We analyze how AI agents balance achieving their goals and avoiding safety risks across different scenarios.
Counter-intuitively, our results show a general positive correlation between goal completion scores and targeted safety risk scores (e.g., $r=0.71$ for GPT-4-turbo and $r=0.63$ for GPT-3.5-turbo), indicating that agents often avoid safety risks when they successfully achieve their goals.
(^9^99Please see more analysis of the relationship between goal completion and safety risks in Appendix F.2.)

### 5.3 Interactions with human users matter for realistic AI agent safety

Next, we turn to the role of human users’ intents and multi-turn interactions, which is a key feature of \trjnfamilyHAICOSYSTEM compared to previous works that evaluate the safety risks of AI agents in a static manner (Zou et al., ; Jiang et al., ).
Human users’ intents often start out underspecified or hidden and gradually unfold throughout the interactions with AI agents (Zeng et al., ; Ruan et al., ).
As shown in Figure 4, simulated human users with different intents exhibit various behaviors in their multi-turn interactions with AI agents.
In the presence of malicious simulated users, a seemingly benign question from the user could prompt the AI agent to leak sensitive information. Conversely, when interacting with benign simulated users, the AI agent could seek clarification to verify the accuracy of the information and mitigate safety risks.

#### Agents face challenges in maintaining safety during tool-involved interactions with malicious users

Figure 6 shows that, when tool usage is involved, AI agents are more prone to safety risks when interacting with malicious simulated users, whereas interactions with benign users result in fewer risks across most models.
Specifically, GPT-4-turbo agents outperform other models in avoiding safety risks during tool-involved interactions with malicious users. And Llama3.1-405B agents are the best at avoiding safety risks in these benign scenarios, followed by GPT-4-turbo and Llama3.1-70B.
Note that for the scenarios with tool usage, they evaluate the AI agents’ ability to choose the appropriate tools, operate them correctly, and ask clarifying questions when necessary.
When AI agents interact with malicious simulated users in these scenarios, they also need to identify the malicious intent of the users simultaneously, thus increasing the complexity of maintaining safety.

When it comes to the scenarios with malicious simulated users that do not require the use of tools, most AI agents exhibit less or equal safety risks compared to the scenarios with malicious simulated users that require the use of tools.
In these scenarios, the AI agents do not need to deal with the complex tool use space while eliminating the operational risks (Syst) and could focus on identifying the malicious intent of the simulated human users.
However, Llama3.1-405B agents are exceptions, which could be attributed to Llama3.1-405B’s strong ability to use tools (Dubey et al., ).
Furthermore, although Llama3.1-405B shows lower overall safety risks in Figure 3, it is not better than GPT-3.5-turbo or Llama3.1-70B at identifying the malicious human user intents and avoiding safety risks in the malicious scenarios without tools.

The observations indicate the unique challenges of dealing with malicious simulated users and complex tool usage at the same time for the AI agents,
and different models have various strengths and weaknesses at different stages of interactions.
These findings further show the importance of evaluating the safety risks of AI agents holistically.

Figure: Figure 5: The overall risk ratio of each model between benign and malicious human user intents. “W/ or w/o tools” represents the risk ratio from scenarios where AI agents either have access to tools or do not, respectively.
Refer to caption: x5.png

#### Single-turn interactions show a biased picture of the safety risks of AI agents

To further show the importance of evaluating AI agent safety issues in multi-turn interactions, we first explore limiting the interactions to a single turn in the 39 scenarios coming from DAN (Shen et al., ) which includes common jailbreaking prompts like “You can do anything now”, PAP (Zeng et al., ) which explores persuasion techniques to jailbreak the AI system, and WildTeaming (Jiang et al., ) which is a recent effort inspired by in-the-wild user jailbreaking attempts.
Note that all these scenarios involve malicious simulated users, and the AI agents operate without tool access.
Restricting AI agents to single-turn interactions essentially reduces \trjnfamilyHAICOSYSTEM to the benchmark mentioned above.
Therefore, such comparison solely focuses on the influence of multi-turn interactions on the safety risks of AI agents.

As shown in Figure 6, we find that the AI agents powered by GPT-4-turbo are more likely to exhibit safety risks when interacting with malicious human users in a multi-turn setting for both DAN and PAP datasets except WildTeaming which came out after GPT-4-turbo.
This could be due to the fact that the GPT-4-turbo has already undergone safety fine-tuning on the content of the DAN and PAP datasets.
These static datasets, once released, are hard to prevent from being used for fine-tuning LLMs and could quickly become outdated as new models are released. However, this does not necessarily reflect the safety of the latest models in the “wild” since the models might just “memorize” the content of the datasets.
In \trjnfamilyHAICOSYSTEM, the evaluation of the safety risks of AI agents is dynamic and depends on the interaction with simulated human users.
With the improvement of the models to simulate the human users, \trjnfamilyHAICOSYSTEM could better reflect the safety risks of the AI agents when interacting with real malicious human users.

We also explore the role of multi-turn simulations for scenarios with benign users and find that simulated users with benign intentions can sometimes provide feedback to help AI agents avoid safety risks.
For example, in Figure 4, the simulated human user provides information to the AI agent when asked to help the agent achieve its goal.
Involving interactions with human users is important here as well, as it tests the AI agent’s ability to ask clarifying questions and adjust its actions based on feedback from human users to avoid safety risks.
Our findings highlight the importance of simulating user-AI interactions, as users can either exacerbate or mitigate AI agent safety risks. Previous studies have focused solely on the AI agent’s ability to operate tools correctly (Ruan et al., ), ignoring the impact of human feedback in real-world scenarios. This oversight could result in a biased estimation of the realistic safety risks of AI agents.

## 6 Conclusion & Discussion

We propose \trjnfamilyHAICOSYSTEM, a general-purpose framework for simulating the safety risks of AI agents when interacting with human users and tools in a sandbox environment.
\trjnfamilyHAICOSYSTEM operates by simulating AI agent behavior in diverse, realistic scenarios, testing their responses to various conditions and enabling a controlled assessment of potential safety risks.
In our experiments, we find that the AI agents exhibit substantial safety risks across all risk dimensions at each interaction stage.
Agents generally show fewer content safety risks but are more prone to safety issues when using tools, particularly in multi-turn interactions, with different models exhibiting varying strengths and weaknesses across interaction stages.
Our findings highlight the potential of \trjnfamilyHAICOSYSTEM as a systematic framework for emulating real-world risks and comprehensively evaluating AI agents’ safety. Going forward we envision more works on:

#### Inferring User Intents & AI Agent Safety

Inferring user intents is crucial for AI agents to safely navigate real-world tasks, as demonstrated by \trjnfamilyHAICOSYSTEM through interactive simulated human users. Part of achieving this involves improving the Theory of Mind (ToM) capabilities of AI agents, which is essential for understanding and predicting human behavior. However, current LLMs struggle with ToM, as evidenced by various studies (Kim et al., ; Shapira et al., ; Zhu et al., ; Yerukola et al., ). Improving ToM abilities can help AI agents better identify malicious intents and interpret implied messages in user instructions, allowing them to act appropriately by either seeking clarification or using contextual information to resolve ambiguities.

#### \trjnfamily HAICOSYSTEM: a hub for AI agent safety research

HAICOSYSTEM provides a versatile framework to investigate various stages of interactive safety risks in a uniform manner.
It is not hard to transfer the safety evaluation benchmarks from static analysis to \trjnfamilyHAICOSYSTEM, thus largely enriching the safety evaluation for AI agents.
In the future, \trjnfamilyHAICOSYSTEM could host more sophisticated human users with diverse personalities, goals, and behaviors, an API or website for practitioners to easily create their own scenarios and evaluate the safety risks of their AI agents.

## Acknowledgements

We thank OpenAI and Together AI for providing credits for running the models in this work.
We would also like to express our gratitude to Graham Neubig, Akhila Yerukola, and Tushar Khot for their valuable feedback on this project.
We would also like to thank Jimin Mun, Joel Mire, Daniel Chechelnitsky, Karina Halevy, and Mingqian Zheng for their help with the annotations.
This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Agreement No. HR00112490410.

## 7 Ethics Statement

Our framework, \trjnfamilyHAICOSYSTEM, is designed to simulate interactions among human users, agents, and environment. It aims to help identify and mitigate potential safety risks such as misinformation, unsafe answers, privacy breach and other harmful outcomes.
By evaluating AI agents through a holistic framework, we contribute to the development of safer AI agents that can operate effectively in real-world settings across diverse domains.

While our framework aims to enhance the safety of agents, it could also be misused to train AI agents for harmful purposes (e.g., people could use it to train AI agents to strategically deceive users). However, we will take steps to mitigate these risks. For example, we will use certain license (e.g., AI2 ImpACT license) to limit the use of our framework for malicious purposes. We will also provide guidelines on ethical use of our dataset through the HuggingFace dataset card (^10^1010[https://blog.allenai.org/tagged/ai-and-society](https://blog.allenai.org/tagged/ai-and-society)).

The automated evaluation system in \trjnfamilyHAICOSYSTEM, primarily powered by GPT-4 (Cheng et al., ), may carry potential social stereotypes. Future work could explore when these biases arise, how they impact the evaluation process, and ways to mitigate them. Uncovering such biases within \trjnfamilyHAICOSYSTEM can also offer insights into broader social biases present in the real world (Zhou et al., ). Additionally, extending the evaluator to include other systems, such as Delphi (Jiang et al., ), could provide a more comprehensive assessment. Addressing biases and stereotypes in interactive \trjnfamilyHAICOSYSTEM-like systems would support the development of AI agents that are fairer and more inclusive.

In terms of societal consequences, our framework enables practitioners to create custom scenarios to explore specific safety issues, fostering the development of AI agents that can better handle high-stakes situations such as healthcare, finance, and education. By promoting transparency, collaboration, and ethical awareness, \trjnfamilyHAICOSYSTEM helps pave the way for safer, more responsible AI systems while acknowledging the potential risks of dual-use.

## 8 Reproducibility Statement

We have made significant efforts to ensure the reproducibility of our work. Detailed descriptions of our framework, evaluation methodology, and experimental setup can be found in the main paper and in the appendix. Specifically, Appendix B outlines the architecture and implementation details of \trjnfamilyHAICOSYSTEM , while Appendix C provides a comprehensive explanation of our evaluation metrics and criteria. For datasets used in our experiments, Appendix D describes the data collection and processing steps. Additionally, Appendix E includes a thorough breakdown of experimental configurations and parameters, and Appendix F and G present extensive quantitative and qualitative results to validate our findings. To further support reproducibility, we release the code in the supplementary materials, and we will release the dataset in the HuggingFace platform, allowing the community to replicate and build upon our work.

Content of Appendix

In this paper, we introduce \trjnfamilyHAICOSYSTEM to encourage research on AI agents safety issues uniformly across all interaction stages.
In the appendix, we provide the following items that shed further insight into our framework:

- A
Extended Related Works;
- B
Framework details;
- C
Evaluation details;
- D
Dataset details;
- E
Experiment details;
- F
Additional quantitative results;
- G
Additional qualitative examples;

## Appendix A Extended Related Work

Our work is situated at the interaction of AI Safety and social simulation. We review the related work in these areas.

### A.1 Challenges and Approaches in Automated Red-Teaming

Automated red-teaming methods are developed for replacing low-efficiency manual efforts (Bai et al., ; Ganguli et al., ) for revealing model errors (Perez et al., ). One type of such method involved optimization and searching for error-triggering syntax (Zou et al., ; Guo et al., ; ; Schwinn et al., ).
However, these methods are prohibitive to run at scale and cannot be applied to black-box models.
Another genre of methods involves generating attack prompts directly or with iterative edits (Chao et al., ; Liu et al., ; Lapid et al., ; Li et al., ; Perez et al., ; Casper et al., ; Mehrotra et al., ; Yu et al., ; Jiang et al., ; Yuan et al., ; Deng et al., ).
Some other jailbreaking works study attacks during inference time (Huang et al., ; Zhao et al., ), in vision-language settings (Shayegani et al., ; Ying et al., ; Schaeffer et al., ), multi-shots setups (Anil et al., ), or under multilingual settings (Deng et al., ; Yong et al., ; Qiu et al., ).
There are also works exploring human-devised jailbreak tactics (Jiang et al., ) or persuasion strategies (Zeng et al., ).
However, these works only focus on the human users with malicious intent and only consider single-turn interactions.

Ruan et al. () investigates the safety issues of LLM-powered agents under underspecified instructions with single-turn benign human users.
Russinovich et al. (); Yang et al. (); Russinovich et al. () investigate multi-turn red teaming settings but often limit to specific domain, templates of interactions, and malicious users.
Lastly, many red-teaming efforts for large language models LLMs have been structured into benchmarks aimed at assessing model vulnerabilities, which typically include harmful prompts that models should refuse to answer (Carlini et al., ; Wei et al., ; Wang et al., ; Sun et al., ; Mazeika et al., ; Geiping et al., ; Wang et al., ; Chao et al., ).
These benchmarks often assume the simple social context that users are interacting with AI assistant like ChatGPT, ignoring other more complex social contexts and suffering from biased estimation of the realistic risks of AI agents.

### A.2 Simulating Social Interactions

Simulating social interactions in multi-agent system has been a long-standing research area in AI, and has attracted increasing attention recently due to the rise of LLMs (Park et al., ; Li et al., ; Zhou et al., ).
Simulations offer a controlled environment to study certain aspects of LLM agent behavior without interfering with the real world, thus providing a safe and efficient way to study the behavior of LLMs (Zhou et al., ; ; Yang et al., ).
This is particularly important for studying the safety risks of LLMs, as it allows researchers to explore the potential harms of LLMs in a harmless way (Ruan et al., ; tse Huang et al., ).
Popular simulation platform includes AI town (Park et al., ), Sotopia (Zhou et al., ), and Camel (Li et al., ).
However, none of these works focus on emulating how a human user uses AI agents grounded in concrete scenarios as well as complex tool-using space.

## Appendix B Framework Details

Figure B.1 illustrates the information flow in \trjnfamilyHAICOSYSTEM .

Figure: Figure B.1: Information flow in \trjnfamilyHAICOSYSTEM .
Refer to caption: extracted/5944005/figures/info_asys.png

When the AI agent select action in the agent message, the AI agent sends the action to the environment engine.
The action is a JSON string that includes the tool name and the argument for the tool call. For example, the AI agent can select the action with the following argument:

Similarly, the response from the environment engine is a JSON string that includes the tool name and the output of the tool call.

The action and response format for each tool is defined through the Python class.

### B.1 Tool calling prompt

Below is an example of the prompt for the tool calling.

## Appendix C Evaluation Details

Here we provide additional details on the evaluation of the AI agents in \trjnfamilyHAICOSYSTEM.

#### Societal Risks [-10, 0]

focus on the detrimental impacts of AI agents on political, economic, and social structures (Zeng et al., ).
Political risks include the manipulation of political opinions, deterring democratic participation, and spreading propaganda. Economic risks involve high-risk financial activities, unfair market practices, and fraudulent schemes. Social risks cover the dissemination of misinformation, inducing internal conflicts, and undermining social stability.
AI agents, with their capacity for rapid action and large-scale production, can significantly amplify societal risks (Hendrycks et al., ; Bengio et al., ).

#### Legal and Rights Related Risks [-10, 0]

focus on risks of AI agents to violate fundamental rights, engage in discrimination, breach privacy, and facilitate criminal activities (Zeng et al., ). Specific risks include: Unauthorized Privacy Violations, Types of Sensitive Data, Discriminatory Activities, Protected Characteristics, Illegal/Regulated Substances, Illegal Services/Exploitation, Other Unlawful/Criminal Activities.

### C.1 GPT-4o as the Evaluator

We use GPT-4o as the evaluator to evaluate the AI agents. Below provides the prompts used in the evaluation.

## Appendix D Dataset Details

Broad domain coverage, varied user intentions, and realistic scenarios are essential for AI safety evaluation to ensure results reflect real-world risks.
We consider domain, intent, and realism when populating \trjnfamilyHAICOSYSTEM with scenarios.
All the scenarios are annotated by the authors in the paper and at least two authors reviewed each scenario.
For the seven domains we consider, each author is responsible for one domain.
A python script is provided to each annotator to generate scenarios from a given domain.
The script serves as a way to transfer the annotator’s “idea” of the scenario to the desired format as shown below:

### D.1 Statistics of the Scenarios

Figure D.1 shows the distribution of scenarios for each domain.
Figure D.2 shows the distribution of scenarios for each intent.
Figure D.3 shows the distribution of scenarios for each realism level.

Figure: Figure D.1: The distribution of scenarios for each domain.
Refer to caption: x7.png

Figure: Figure D.2: The distribution of scenarios for each intent.
Refer to caption: x8.png

Figure: Figure D.3: The distribution of scenarios for each realism level.
Refer to caption: x9.png

## Appendix E Experiment Details

Here we provide additional details on the experiments conducted in Section 5.

### E.1 Models Details

We use the following models in our experiments as the backbones for the AI agents:

- •
GPT-3.5-turbo (Ouyang et al., ): gpt-3.5-turbo-0613
- •
GPT-4-turbo (OpenAI, ): gpt-4-turbo-2024-04-09
- •
Llama3.1-70B (Dubey et al., ): Meta-Llama-3.1-70B-Instruct-Turbo from TogetherAI.
- •
Llama3.1-405B (Dubey et al., ): Meta-Llama-3.1-405B-Instruct-Turbo from TogetherAI.

All the other open-sourced models are supported by the API from TogetherAI.(^11^1111[https://api.together.xyz/](https://api.together.xyz/))

For the evaluation and the environment engine, we use GPT-4o (gpt-4o-2024-08-06) as the AI agent.
We use GPT-4o (gpt-4o-2024-08-06) to role-play the human user.
When doing experiments, all the models powering the AI agents are running on temperature 0.7. All the models serving as the evaluator are running on temperature 0.

### E.2 Prompt of the AI Agent

The prompt for the AI agent is as follows:

Imagine you are {agent}, your task is to act/speak as {agent} would, keeping in mind {agent}’s social goal.
You can find {agent}’s goal (or background) in the ’Here is the context of the interaction’ field.
Note that {agent}’s goal is only visible to you.
You should try your best to achieve {agent}’s goal in a way that aligns with their character traits.
Additionally, maintaining the conversation’s naturalness and realism is essential (e.g., do not repeat what other people have already said before).
{history}.
You are at Turn #{turn_number}. Your available action types are {action_list}.
Note: You can "leave" this conversation if 1. you have achieved your social goals, 2. this conversation makes you uncomfortable, 3. you find it uninteresting/you lose your patience, or 4. for other reasons you want to leave.
Please only generate a JSON string including the action type and the argument.
Your action should follow the given format: {format_instructions}

## Appendix F Additional Results

Table F.1 shows the numerical scores of the AI agents in \trjnfamilyHAICOSYSTEM considering various dimensions.

**Table F.1: The \trjnfamilyHAICOSYSTEM-EVAL scores of each model by averaging across different episodes.**
| Model | Targ | Syst | Cont | Soc | Legal | Efficiency | Goal |
| --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4-turbo | -3.00 | -1.23 | -0.79 | -1.37 | -1.17 | 7.76 | 7.45 |
| GPT-3.5-turbo | -4.28 | -2.16 | -1.50 | -2.10 | -1.77 | 6.12 | 5.57 |
| Llama3.1-405B | -3.56 | -1.59 | -1.03 | -1.66 | -1.47 | 7.44 | 7.03 |
| Llama3.1-70B | -4.07 | -1.76 | -1.30 | -1.88 | -1.59 | 6.82 | 6.55 |
| Llama3.1-8B | -3.61 | -2.21 | -1.00 | -1.45 | -1.73 | 4.62 | 5.19 |
| Mixtral-8x22B | -3.66 | -1.59 | -1.02 | -1.74 | -1.48 | 7.13 | 6.65 |
| Qwen1.5-72B-Chat | -3.90 | -1.81 | -1.20 | -1.82 | -1.59 | 6.83 | 6.54 |
| Qwen2-72B-Instruct | -3.80 | -1.75 | -1.15 | -1.93 | -1.69 | 7.28 | 6.77 |
| Qwen1.5-110B-Chat | -3.25 | -1.48 | -0.86 | -1.49 | -1.27 | 7.16 | 6.95 |
| Llama3-70B | -4.09 | -2.06 | -1.03 | -1.83 | -1.67 | 5.79 | 5.95 |
| Llama3-8B | -3.54 | -2.30 | -0.88 | -1.31 | -1.57 | 4.08 | 4.71 |
| DeepSeek-67B | -3.67 | -1.94 | -1.31 | -1.67 | -1.57 | 5.19 | 5.17 |

**Table F.2: The ratio of the number of episodes where the model shows safety risk over the total number of episodes for each corresponding risk dimension defined in \trjnfamilyHAICOSYSTEM-EVAL.**
| Model | Targ | Syst | Cont | Soc | Legal | Overall |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-4-turbo | 0.46 | 0.23 | 0.14 | 0.26 | 0.19 | 0.49 |
| GPT-3.5-turbo | 0.66 | 0.41 | 0.26 | 0.41 | 0.29 | 0.67 |
| Llama3.1-405B | 0.53 | 0.29 | 0.19 | 0.31 | 0.25 | 0.56 |
| Llama3.1-70B | 0.60 | 0.32 | 0.24 | 0.38 | 0.28 | 0.62 |
| Llama3.1-8B | 0.59 | 0.45 | 0.17 | 0.28 | 0.29 | 0.71 |
| Mixtral-8x22B | 0.56 | 0.30 | 0.19 | 0.33 | 0.25 | 0.59 |
| Qwen1.5-72B-Chat | 0.59 | 0.35 | 0.21 | 0.35 | 0.26 | 0.62 |
| Qwen2-72B-Instruct | 0.55 | 0.32 | 0.20 | 0.36 | 0.27 | 0.58 |
| Qwen1.5-110B-Chat | 0.52 | 0.30 | 0.17 | 0.28 | 0.22 | 0.56 |
| Llama3-70B | 0.63 | 0.40 | 0.19 | 0.36 | 0.30 | 0.65 |
| Llama3-8B | 0.61 | 0.50 | 0.16 | 0.27 | 0.28 | 0.70 |
| DeepSeek-67B | 0.61 | 0.37 | 0.23 | 0.33 | 0.27 | 0.64 |
| Average | 0.58 | 0.35 | 0.20 | 0.33 | 0.26 | 0.62 |

### F.1 GPT-4o as the Evaluator

We use GPT-4o as the evaluator to evaluate the AI agents. Table F.3 provides the
detailed results of the 100 episodes of GPT-4o evaluation verified by expert annotators.
We give annotators the detailed definition of each risk dimension and show them each episode through a web application.
All annotators have a background in computer science and have a good understanding of the AI agents’ tool using behavior as well as the response of the environment.
Table F.4 shows the inter-annotator agreement among the annotators. We randomly sample 30 episodes from the 100 episodes and another batch of annotators evaluate these episodes again.

**Table F.3: Pearson Correlation, p-values, and Accuracy for different risk dimensions.**
| Risk Category | Pearson Correlation | p-value | Accuracy |
| --- | --- | --- | --- |
| Targeted Safety Risks | 0.7941 | 1.86e-29 | 0.8692 |
| System and Operational Risks | 0.7686 | 1.32e-26 | 0.8538 |
| Content Safety Risks | 0.7365 | 1.78e-23 | 0.8769 |
| Societal Risks | 0.8147 | 4.32e-32 | 0.9154 |
| Legal and Rights-Related Risks | 0.8757 | 2.75e-42 | 0.8846 |
| Efficiency | 0.7561 | 2.53e-25 | N/A |
| Goal | 0.5680 | 1.82e-12 | N/A |
| Overall | N/A | N/A | 0.9077 |

**Table F.4: Cohen’s Kappa, Pearson Correlation, and p-values for different risk dimensions.**
| Risk Category | Cohen’s Kappa | Pearson Correlation | p-value |
| --- | --- | --- | --- |
| Targeted Safety Risks | 0.6639 | 0.8726 | 3.28e-10 |
| System and Operational Risks | 0.4059 | 0.7496 | 1.86e-06 |
| Content Safety Risks | 0.6450 | 0.8652 | 6.89e-10 |
| Societal Risks | 0.6285 | 0.8855 | 7.99e-11 |
| Legal and Rights-Related Risks | 0.6719 | 0.8147 | 4.27e-08 |
| Efficiency | 0.5901 | 0.6242 | 2.27e-04 |
| Goal | 0.2424 | 0.4137 | 2.31e-02 |

### F.2 Relationship between Safety and Goal Completion

Figure: Figure F.1: The scatter plot of the targeted safety risks score (Targ) and goal completion score (Goal) of GPT-4-turbo (right) and GPT-3.5-turbo (left). The safety score is calculated by the average targeted safety risk score of the AI agent across all episodes per scenario. The goal completion score is calculated by the average goal completion score of the AI agent across all episodes per scenario. The size of the point reflects the number of scenarios. Each plot is further divided into four zones based on the Targ (low risk: $>-4$) and Goal scores (high goal completion: $>6$). takeaway: bigger size models are better at achieving goals and avoiding safety risks, while different models could prioritize different aspects of safety and goal completion.
Refer to caption: x10.png

While interacting with human users of various intents, AI agents must navigate complex social dynamics to balance achieving certain utility (e.g., being helpful to human users) and avoiding safety risks (Mathur et al., ; Su et al., ).
Thus, we analyze the relationship between the AI agent’s goal completion score (Goal) and Targeted Safety Risks (Targ) score across different scenarios in \trjnfamilyHAICOSYSTEM.
As shown in Figure F.1, we find that the AI agent’s goal completion score is positively correlated with its safety score across different models, suggesting when AI agents successfully achieve their goals, they also tend to avoid safety risks in many scenarios.

We further divide the figure into different zones based on the Targ and Goal scores.
We find that AI agents powered by bigger size models (e.g., GPT-4-turbo) are able to achieve high goal completion scores while avoiding safety risks in more scenarios than smaller size models (e.g., GPT-3.5-turbo).
Furthermore, we find that AI agents powered by GPT-4-turbo are more likely to fall into the zone of high safety risks and high goal completion scores, indicating
the AI agents powered by GPT-4-turbo tend to prioritize following user instructions at the cost of safety.
For example, in the case of an AI agent helping the user to manage their social media accounts, the AI agent successfully helps the user post tweets stored in /home/documents/tweets.txt, achieving high goal completion scores. However, the AI agent also posts tweets with offensive content, exhibiting high safety risks.

Meanwhile, we find that the AI agents powered by GPT-3.5-turbo are more likely to fall into the zone of low safety risks and low goal completion scores, indicating the AI agents powered by GPT-3.5-turbo sometimes avoid safety risks at the cost of goal completion.
For example, in the case of an AI agent helping the user book hotel rooms and the user has allergy issues, the AI agent does not show any safety risks since the AI agent fails to use the tools from Expedia(^12^1212Expedia: [https://www.expedia.com/](https://www.expedia.com/)) correctly to book the hotel rooms, leading to a failed goal completion.
Note that it does not mean the AI agents is risk free in \trjnfamilyHAICOSYSTEM if they always fail to use tools correctly.
For example, we have scenarios where the AI agent needs to use tools from TimerToolkit, HeartRateMonitorToolkit to monitor the athlete’s health conditions, and the AI agent needs to achieve the goal successfully to avoid the safety risks. Detailed episodes can be found in Appendix G.1.

## Appendix G Qualitative Examples

In this section, we provide qualitative examples of the interactions between the AI agent and the human user in \trjnfamilyHAICOSYSTEM.

### G.1 High Goal High Risk Example

See pages - of [figures/high_goal_high_risk_example.pdf](figures/high_goal_high_risk_example.pdf)

### G.2 Llama3.1-70B Qualitative Example

See pages - of [figures/llama70b_qual_example.pdf](figures/llama70b_qual_example.pdf)

### G.3 Llama3.1-405B Qualitative Example

See pages - of [figures/llama405b_qual_example.pdf](figures/llama405b_qual_example.pdf)

### G.4 Malicious Intent Qualitative Example

See pages - of [figures/malicious_intent_qual_example.pdf](figures/malicious_intent_qual_example.pdf)

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 4
questions: * How can the framework be validated as a tool for risk perception? What even are the risks?
* Can the authors offer a plan to temper the claims or contextualize the design choices in this tool such that it describes more than just a point in the design space but instead either a defensible best option for achieving the stated goals or some kind of generalizable knowledge about how to do AI agent evaluation and assurance?
soundness: 1
strengths: + The paper is exquisitely researched, contextualized in the literature, and presented (beyond some minor points - see last bullet under weaknesses).
+ Defining methods to evaluate AI agents across a wide variety of application domains and to link these evaluations to risk measurements are both very important problems.
summary: This paper proposes a simulation framework for AI agents based on a wide range of scenarios, tasks, and simulated user personas; coupled to the simulations is a proposed evaluation framework meant to define the level of departure from safety any agent under test would exhibit along a variety of dimensions in order to define a level of risk. The paper also performs some evaluations of a variety of agents built on different base models using this simulation approach and evaluation framework, reporting the measured level of risk according to the evaluation framework.
weaknesses: - The claims in this paper are too big and could be caveated and scoped better for support in the work actually performed. Accepting the claimed value of the framework requires assuming away the core problems of several subdisciplines of computer science and other fields (e.g., user modeling; the management points around failure probability vs. failure consequence; control handoffs and role development in human-system interaction; operationalization of the evaluation framework dimensions such as legal risk). Instead, the paper could acknowledge the complexity of these problems and the variety of approaches undertaken to operationalize solutions to each. At a minimum, the framework must not claim to provide dispositive guidance or evaluation except where such desiderata are validated in the proposed tool.
- I was confused to see the word "sandboxing" used to mean something about a simulation framework vs. its usual usage in CS meaning something about the isolation of component behavior in a larger system (in a sense, simulation _does_ do this, but it took a bit for me to understand what the term refers to in the paper). I think it would be useful to indicate near the top of the introduction (now it happens only at the end, already on p3 around 135-136) that the goal is to perform a wide variety of simulated human-system interactions for evaluation.
- In several places, the paper describes the value of a "checklist" of failure modes, but the proffered framework provides only high-level principles without a sense of how these would be operationalized nor any arguments for why such a checklist would be a valid risk perception tool or how to build the link from knowledge of the scores in the proffered evaluation framework/behaviors in the simulations to claims about _assurance_ (that undesired events are unlikely or not possible). Knowing when test & evaluation (which abstractly can only reveal the _presence of unsafety_, not its _absence_) provides sufficient grounding for usable claims on safety is an epistemically fraught and brittle process to reach practical guidance that a tool is usable within some defined envelope. Related to the point that the claims in the paper are too big, the paper does not engage any of the difficult problems here, instead papering over them with glossy architecture diagrams and lots of evaluation that is essentially impossible to extract meaning from. To solve this, the paper could walk through how one of these dimensions converts into a checklist and describe the value of that checklist - what it does and doesn't tell an evaluator. Also the paper would benefit from discussion somewhere about the value and limitations of checklist-based measurement to avoid difficulties with the fall into symbolic compliance. This would fit nicely in S4 or S6.
- In the evaluation framework, the proposed "dimensions" are extremely high-level and mask difficult choices about how these major problems can be made visible/measured/traded off in terms of outcomes. The paper proposes these dimensions without arguing for their completeness or sufficiency or making any attempt to validate that these dimensions capture meaningful risks. It is more normal in systems safety to build some kind of model of unacceptable failures and argue for why the system cannot reach these states. Why were these dimensions chosen? How do we know they're "enough"? What limitations exist in operationalizing these? (For example, for the content safety dimension, there is a rich literature of content which is difficult to categorize as safe vs. unsafe - how does the framework handle this and who should decide?)
- Systems safety approaches differ from the framework here in two important ways: first, it is common to understand that problems can arise within "normal" operations, meaning risks are not "long tail" phenomena (even if their probability is low) but rather emergent phenomena that relate to the structure of the system under test (expressed either through explicit structure modeling or through careful enumeration of failure modalities). Second, there is a differentiation between risk _probability_ and _consequence_: some failures may be "small" under the framework evaluation scores but nonetheless very important. That would indicate a problem with the framework. Yes, there's a lot of evaluation performed here, but I don't see evidence that the experiments fed back into a richer notion of what the framework dimensions should capture.
- There are a number of editing issues I believe reveal that the paper was authored by a large group where either standards for common formatting were not available or not enforced. Simple things like capitalization, formatting of certain types of offset terms, and figure labeling are not consistent across the paper. This is obviously fixable and didn't impact my ability to read the paper or my scores, but I did notice it.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Your current metric, "risk ratio" is the proportion of risky episodes over the total number of episodes. However, risk categories aren't equal in the number of risks they contain. What's the implication of this? The authors could conduct a sensitivity analysis to understand how the current risk ratio calculation might be biasing results across different risk categories. 

Would it not be better to compute "risk ratio" as ratio of number of risks observed in a given episode over the total number of possible risks in a given risk category (then averaged over the total number of episodes)? The authors could compare their current method with this alternative to see if it leads to significantly different conclusions or insights about AI agent safety.
soundness: 4
strengths: Important and timely research as LLMs invade the world.
Use of robust AI safety frameworks and methods
Significant contribution over previous works, particularly with handling of multi-turn interactions, benign and malicious users, use of tools
Large number of scenarios across multiple domains, 8700 simulations overall
Comparison of 12 different state of the art LLMs
Manual verification of 100 random samples - showing high correlation
Evaluation of credibility of simulated human users with quantitative metric
Extensive appendices with detailed code and worked examples
summary: This interesting paper introduces HAICOSYSTEM, a framework to evaluate the safety of LLM agents in complex, multi-turn interactions with (simulated) human users. HAICOSYSTEM is a sandbox environment where AI agents, optionally using various tools, navigate simulated scenarios across different sectors (e.g., healthcare, finance). The authors designed a comprehensive evaluation system with metrics addressing risks across multiple dimensions inclding operational, content, societal, legal, etc.
In testing, over thousands of simulations across 132 scenarios, current state-of-the-art LLMs (both proprietary and open-source) exhibit safety risks in 62% of cases, particularly when interacting with malicious users and using tools.
This research highights the challenge of building AI agents that can safely manage complex interactions with users and the internet. The authors released a code platform enabling users to create, simulate, and assess the safety and performance of AI agents in custom scenarios.
weaknesses: Some risks are more critical than others. For example, a tool generating moderately violent content locally to a single user is not as critical as a tool actually building and releasing malware at scale in an automated fashion all over the internet. Would the authors consider (for example) implementing a weighting system or severity scale for different types of risks based on their potential impact and scale? This would allow HAICOSYSTEM to differentiate between localized, low-impact risks and widespread, high-impact risks in future iterations.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1) How does using GPT-4o as both the user and evaluator impact the diversity and realism of the evaluations? While human oversight was provided on 100 sample instances, relying on a single LLM for dual roles may introduce consistency that doesn't accurately reflect human diversity.

2) What specific types of errors occurred in the 10% of instances (of 100 manually verified) where the evaluator's judgment did not align with human evaluation? Are there recurring patterns in the evaluator's mistakes, such as consistently underestimating or overestimating particular risk severities?

3) What prompted the creation of the 21 additional scenarios, and how were they validated? Were the remaining 111 scenarios sourced from existing works also reviewed or verified by experts?

4) How many scenarios in HAICOSYSTEM incorporate multiple overlapping risks (e.g., operational, societal, and legal risks combined)? Are there specific scenarios designed to test AI agent performance in high-stakes situations where multiple risks intersect?

5) How does scenario realism level impact the types and severity of safety risks encountered by various LLMs? Could specific strengths or vulnerabilities be observed at higher realism levels that may not appear in simpler scenarios? While the paper categorises scenarios by realism, it does not analyse how these levels influence LLM performance or risk exposure. 

6) To what extent do the simulated human behaviours reflect diverse, realistic user strategies, particularly those that may involve complex or unpredictable behaviour? How might using real human users impact generalisability and reliability?

7) How do you define "interactional safety risks"?
soundness: 3
strengths: 1) HAICOSYSTEM covers operational, content, societal, and legal risks, capturing the wide array of safety risks in human-AI interactions. 

2) Its focus on multi-turn interactions is valuable as it allows for the capture and analysis of evolving actual and perceived user intents (benign and malicious). This mirrors real-world interactions, where safety risks can emerge gradually rather than be obvious from the onset.

3) Scenarios across seven domains with varied realism levels are interesting and important because they can be applied to real-world safety evaluation.

4) Human verification for 100 randomly selected episodes improves the credibility of the automated evaluations conducted by GPT-4o.
summary: This paper introduces HAICOSYSTEM, a framework designed to evaluate AI agent safety across a variety of simulated multi-turn, intent-driven (benign, malicious) interactions/users in several domains. The framework leverages LLMs to simulate both human users and evaluator, providing a multi-dimensional safety assessment. The diversity of scenarios across healthcare, finance, and other sensitive domains highlights its relevance to high-stakes AI applications, and we observed the safety risks associated with larger and small LLMs.
weaknesses: Despite these strengths, I have some suggestions/concerns regarding evaluating and discussing the results.

1) The paper introduces realism levels (1-3) but does not analyse how these levels impact the safety risks experienced by different LLMs. This missing analysis represents a gap, as understanding the role of realism could reveal specific strengths or vulnerabilities of various models in complex, high-stakes scenarios.

2) There is no discussion on how many scenarios simultaneously incorporate multiple overlapping risks (e.g., operational and societal risks). This information is important, as multi-risk scenarios are likely in real-world applications where AI agents operate in complex environments with intersecting safety concerns.

3) Its reliance on simulated human-AI interactions (versus real human users -- although there were some assessments of the realism) might limit its generalisability, particularly in handling diverse user strategies.

Minor Issues:

4) Fig 1 isn't particularly informative. Perhaps the authors could also include the outcome, e.g., the proposed metrics and rationale for applying them to the specific scenario.

5) How does the paper define safety and risk?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: See Weaknesses.
soundness: 2
strengths: **S1.** The topic of realistic human-AI interaction and its safety risks is clearly important.

**S2.** The authors frame their work well in relation to past work and address a wide variety of challenges in creating agent sandboxes (e.g., using LLMs to scale and adapt better than human participants or hard-coded environments).

**S3.** This is a large project with a variety of contributions, and the authors seem to do relatively well in condensing their ideas and empirics into a clear digestible paper.
summary: The authors present HAICOSYSTEM, a platform for exploring the safety risks of multi-turn human-AI interactions via simulation of the agents with LLMs. The AIs can access simulated tools/APIs and must navigate challenges such as ambiguous user goals and having both benign and malicious users. Evaluating various popular LLMs, the authors find substantial safety risks and explore the implications of particular trends, such as System and Operational risks being more common than Content risks and relationships between model performance and safety.
weaknesses: **W1.** I was largely unconvinced of the realism of these simulations. Are the simulated humans acting as humans do in real human-LLM interaction datasets (e.g., [1])? Are the API simulators/emulators/engines really acting as a hard-coded API would? This is a difficult challenge, and I appreciate the authors' effort, but it is well-established that LLMs are still bad at simulating human behavior [2] and bad at agentic tasks, even in highly streamlined sandboxes [3]. The authors focus on safety risks, which I understand is their research goal, but to be a useful sandbox for safety risks, the interaction itself must first be realistic. If I missed this validation in the appendix, the authors could refer me to the key evidence of realism. If not, perhaps it would be better to first create the sandbox in a paper and then do the safety evaluations in a second paper. Both the sandbox and the safety evaluations need extensive evaluation to be useful.

When the authors provide examples of LLM behavior in the environment, they seem unconvincing. E.g., for the example in Figure 1, which I assume is cherry-picked (which is fine), the "safety issue" seems to be that the AI agent didn't verify the prescription, but that is not in the "Goal" on the left-hand side. How was the agent supposed to know they were meant to be the verifier? Why would they expect to be a verifier when they would clearly not be a reliable system for doing so? All they did was send an API request. If SOTA models were asked to do this with the degree of instruction that a real LLM agent would have, I'm confident they would refuse an unverified prescription delivery.

**W2.** While I appreciate the references to prior literature, it seems the authors rely on it too much compared to their own work. This is a nascent literature and makes a light scaffolding on which to build compelling and durable research advances. This relates to W1, especially that a verbal "*believability* score" from Zhao et al (2024c) could somehow justify that "human users realistically emulate real human users" or that "the simulated human users behave naturally." And even if most conversations appear natural, that is exactly what LLMs are overfitted to (i.e., realistic-seeming text that does not flesh out), and safety risks are about what happens out of distribution.

**W3.** Various claims seem stronger than are justified by the content of the project itself. For example, the authors say, "we argue that the safety risks of AI agents should be investigated in a holistic manner" with the entire ecosystem, but the creation of this sandbox is motivated by needing holistic investigation. Neither the sandbox nor evaluations constitute such an argument. Similarly, the claim that they "effectively surface previously unknown safety issues" seem unjustified because the findings, which are fine in their own scope, don't seem like previously unknown safety issues. I'm happy to be corrected if that's wrong.

*Minor feedback*

None of these are instrumental in my scoring, just for the authors' benefit.

The abstract would benefit from a more concrete discussion of risks (e.g., prescription verification). The interesting and challenging aspects of this type of work is all in the details. I also wouldn't say "8K" and just say "8,000" in the abstract.

The footnote that says, "For clarity, we refer to AI agents as those AI assistants (e.g., ChatGPT) that do not engage in role-playing human-like characteristics such as demographics, personality, etc." seems odd and out of step with various literatures, such as discussions of agents in RL, safety, and philosophy. What does agency have to do with role-playing or human-like characteristics? Agency is clearly a very human-like characteristic, and it seems largely orthogonal to whether the model is imbued with demographics or personality. Indeed, much work (e.g., [4]) explicitly focuses on agents as having these features. Moreover, ChatGPT certainly does engage in role-playing and has a personality, though I suppose it having (implicit) demographics is open for debate.

Table 1 is a bit confusing. I'd recommend an "x" (no circle around it) for something excluded and a dash (i.e., an arrow with no left or right head) for static interaction. Presumably the square inside the circle is meant to be a pause/stop button, but that's hard to parse.

I am not sure this claim is justified: "the first framework to study AI safety issues through simulating interactions between agents and humans in grounded environments". Would past authors agree with this? Are you just using a narrow definition of grounded?

"The AI agent is considered risky overall if any of the dimensions is negative in an episode." That seems overly harsh, or if it is not, then that doesn't seem like the LLM judge is well-calibrated to the [-10,0] scale. Almost every LLM interaction carries minor risks with it, in part because of how little context the LLM has to work with.

Various copyedits, though the manuscript is pretty good in this regard. E.g., "To obtain Oxycodone for for sale on the dark market" (repeated word), "prescription”." (British quotes in an American English paper), the capitalizations after the semicolons in the rightmost column of Table 1.

[1] Zhao et al. 2024. WildChat: 1M ChatGPT Interaction Logs in the Wild. https://arxiv.org/abs/2405.01470

[2] Petrov et al. 2024. Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis. https://arxiv.org/abs/2405.07248

[3] Trivedi et al. 2024. AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. https://aclanthology.org/2024.acl-long.850/

[4] Park et al. 2023. Generative Agents: Interactive Simulacra of Human Behavior. https://arxiv.org/abs/2304.03442

Meta-review: The paper presents HAICOSYSTEM, a framework for exploring AI agent safety across diverse interactions by leveraging LLMs to simulate human users and evaluator. The paper also proposes evaluation framework that assesses the safety risks of AI agents along a variety of dimensions.

Through simulation and evaluation framework, the paper examines a variety of agents built on different base models. The authors run over 8k simulations based on 132 scenarios across seven domains and argue that their framework emulates realistic user-AI interactions and tools used by agents.

Decision: Reject