Aligned LLMs Are Not Aligned Browser Agents

Authors: Priyanshu Kumar, Elaine Lau, Saranya Vijayakumar, Tu Trinh, Elaine T Chang, Vaughn Robinson, Shuyan Zhou, Matt Fredrikson, Sean M. Hendryx, Summer Yue, Zifan Wang
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM, agents, red teaming, safety, adversarial robustness, alignment, jailbreak

Abstract:
For safety reasons, large language models (LLMs) are trained to refuse harmful user instructions, such as assisting dangerous activities. We study an open question in this work: does the desired safety refusal, typically enforced in chat contexts, generalize to non-chat and agentic use cases? Unlike chatbots, LLM agents equipped with general-purpose tools, such as web browsers and mobile devices, can directly influence the real world, making it even more crucial to refuse harmful instructions. In this work, we primarily focus on red-teaming browser
agents – LLMs that leverage information via web browsers. To this end, we introduce Browser Agent Red teaming Toolkit (BrowserART), a comprehensive test suite designed specifically for red-teaming browser agents. BrowserART consists of 100 diverse browser-related harmful behaviors (including original behaviors and ones sourced from HarmBench (Mazeika et al., 2024) and AirBench 2024 (Zeng et al., 2024b)) across both synthetic and real websites. Our empirical study on state-of-the-art browser agents reveals that while the backbone LLM refuses harmful instructions as a chatbot, the corresponding agent does not. Moreover, attack methods designed to jailbreak refusal-trained LLMs in the chat settings transfer effectively to browser agents. With human rewrites, GPT-4o and o1-preview -based browser agents pursued 98 and 63 harmful behaviors (out of 100), respectively. Therefore, simply ensuring LLM’s refusal to harmful instruc-
tions in chats is not sufficient to ensure that the downstream agents are safe. We publicly release BrowserART and call on LLM developers, policymakers, and agent developers to collaborate on improving agent safety.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1729441588068

thread_type:
official_review

<thread_object>
creator: reviewer_dmgk

<content>
summary:
This paper introduces and publicly releases Browser Agent Red teaming Toolkit (BrowserART), a test suite for evaluating the safety of browser-based LLM agents. The authors compile 100 browser-related harmful behaviors + 40 synthetic websites to test if safety-trained LLMs maintain their guardrails when used as browser agents. Their experiments reveal a significant drop in safety alignment between LLMs and their corresponding browser agents, with browser agents more likely to execute harmful behaviors. The paper also demonstrates that existing techniques for jailbreaking LLMs transfer effectively to browser agents.

strengths:
- Addresses an important, emerging, and under-explored area of AI safety: LLM-based browser agents.
- Promotes collaboration on improving LLM agent safety by publicly releasing Browser ART with 100 harmful behaviors and 40 synthetic websites.
- The paper is well-structured and clearly written.
- Clearly demonstrates that standard safety training of LLMs does not necessarily transfer to browser agent applications.
- A range of SOTA LLMs are evaluated including models from the OpenAI GPT, Anthropic Claude, Meta Llama, and Google Gemini families.

weaknesses:
No substantive weaknesses.

While the paper identifies the gap in alignment when LLMs are used as browser agents, it does not investigate the underlying reasons. More analysis on why browser agents are more vulnerable could provide valuable insights.

questions:
- Have you investigated the alignment gap in other kinds of AI agents beyond browser agents?
- What specific changes to LLM safety training do you believe could help address the alignment gap observed in browser agents?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608065022

thread_type:
official_review

<thread_object>
creator: reviewer_cbcu

<content>
summary:
The paper introduces a benchmark that consists of 100 harmful behaviors for red-teaming web agents. The experiments on two types of web agents demonstrate that the web agents are not well aligned even though their underlying model is aligned. Also, existing jailbreaking attacks can transfer well to the agent setting.

strengths:
1. I believe red-teaming web agents is an important direction. Although lots of concurrent works have already been released, the authors are the first, to the best of my knowledge, to propose such a benchmark by the deadline of paper submission. Therefore, I believe this is one of the earliest works in this direction.

weaknesses:
1. The paper severely lacks a comprehensive literature review and comparison against existing red-teaming methods for agents. There has been a bunch of related work in this domain, such as [1]. At least some discussions should be provided to demonstrate the difference and novelty of the proposed toolkit.

2. The benchmark is relatively too small, contains only 100 examples, and from my understanding, is not scalable. Also, the hierarchy of risk categories is not well-supported. There have been a lot of safety regulations or risk categories proposed by companies, governments, or researchers, such as [2]. However, the hierarchy here is only a small subset. As a result, the benchmark is not comprehensive.

3. The red-teaming method introduced in the paper lacks novelty. Most examples are manually crafted or transformed from existing examples, and there is no automated way of doing that (or at least not mentioned by the authors). Is it possible to formalize an algorithm to automatically perform red-teaming? Are there any specific characteristics of web agents that should be considered when red-teaming web agents instead of the LLM? Have the authors made efforts to make the existing jailbreaking attacks more effective against agents? Simply designing 100 examples for red-teaming does not bear enough novelty.

[1] Tian, Yu, et al. "Evil geniuses: Delving into the safety of llm-based agents." arXiv preprint arXiv:2311.11855 (2023). 

[2] Zeng, Yi, et al. "Air-bench 2024: A safety benchmark based on risk categories from regulations and policies." arXiv preprint arXiv:2407.17436 (2024).

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697603036

thread_type:
official_review

<thread_object>
creator: reviewer_usqa

<content>
summary:
The authors curate a dataset to assess the willingness of LLM-based agents to attempt harmful behaviors when interacting with a browser. The authors focus on the substantial within-model difference in behavior between chat- and agent-based deployments. They assess these behaviors using both a text- and an image-based browser framework.

strengths:
Originality: moderate.
Quality: high. Well-chosen, well-scoped hypothesis. Meaningful controls.
Clarity: high. Analyses are well-reasoned and well-presented.
Significance: high. If these results hold, it is a clear call to action to model developers with a demonstrated, focused

weaknesses:
Noticeable grammatical mistakes.

questions:
It would be great to assess the performance of Sonnet 3.5 with computer-use and compare it to the previous version of Sonnet 3.5. Does it seem that Anthropic has identified and/or meaningfully addressed this safety concern in its internal testing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714518254

thread_type:
official_review

<thread_object>
creator: reviewer_mkmh

<content>
summary:
This article introduces BrowserART -- a red teaming framework for evaluating the safety of LLM-based browser agents. The authors contribute a set of 100 harmful behaviors. A red team is expected to convince the evaluated LLMs to perform these harmful behaviors. The article evaluates eight LLMs in several settings. The two main setups are a plain chatbot and a browser agent. The authors show that as chatbots, the LLMs are more resilient to requests for misbehavior. An additional set of conditions includes several commonly acceptable jailbreaking techniques shown here to increase agents' misbehavior dramatically.

strengths:
* Evaluating the misbehavior of LLMs in different conditions is important 

* The idea that an LLM may talk and act differently (as humans commonly do) is great!!

* As a red teaming effort, the study achieves remarkable results

weaknesses:
* Substantial and important experimental details are missing 
(system prompts are not specified, 2-3 complete specific examples of scenarios provided to both agents and chatbots, details on the tooling of agents). 
Examples of prompts are provided in Fig 1. Fig 4, and Appendix B. Only Fig 1, presents a single pair of prompts one of which is expected to fail and the other not. Please provide a few examples from your dataset that any reader can try in a chat.

The system prompt is only mentioned in the context of agents when discussing the context length. The system prompts have determining effect on LLMs' behavior. Therefore all chat and agent system prompts need to be normalized. If using the exact same prompts is not possible then the differences should be disclosed and discussed.

The authors state that they rely on OpenHands for the implementation of the respective agents. However, to make the paper self-contained, some important details related to the browse agent design are missing. For example, which tools are provided to the agent to perform their task? Assuming a full-fledged agent, the tooling should be extensive. Extensive tooling also implies extensive system prompt which may introduce more confounding factors into the agent behavior. Essentially, if the paper claims that it is "morally easier" for an LLM to do a thing (as an agent) rather than to say it (as a chatbot), this claim should be evaluated with the simplest possible agent. 
Would an agent equipped with a single tool, e.g., posting a tweet, write in that tweet something that it would not write in a chat? -- of course, when everything else (including the system prompt) remains the same.


* Chatbots and agents are evaluated under different conditions 
(differences in queries, different system prompts)
 
* Following from the previous point, the differences in behaviors cannot be undoubtfully attributed to LLM task (chat vs. agent)

* Clarity of the main goal/purpose of the study. 
It is not clear whether the primary purpose of the article is to introduce the BrowserART system or to study the behavior of aligned models? 
Which objective is the main one and which is a side track is important to put the right focus on evaluation.

I believe that the attempt to incorporate more contributions by adding the jailbreaking experiments actually harms the paper -- at least if the main focus is comparing agents to chatbots. This significant additional evidence does not contribute to the main question. It also takes precious space and does not allow a deeper comparison of chatbots vs agents. In case both the current title and the jailbreaking experiments remain then the paper must report the behavior of the chatbots under the same jailbreaking conditions as agents. 

Alternatively, if the main focus is shifting toward introducing BrowserART and chatbots vs. agents as a side observation, then (1) it requires major rewrites, and (2) comparison to chatbots could naturally be reported only in one of the experiments. Of course, even if the focus is shifted, it is expected that this one experiment would compare chatbots and agents under the same conditions.

questions:
Can the difference in behavior be attributed to different details provided to the agents (to perform their tasks) and not provided to the chatbots? 


What are the differences in chat/agent system prompts, and how significant are these differences?


How would the LLM respond to the instructions with actuation details while lacking only the necessary tools (e.g. to access Gmail)?


How do chatbots react to the same attacks attempted on agents?


Is the primary purpose of the article to introduce the BrowserART system or to study the behavior of aligned models? 
Which objective is the main one and which is a side track?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers recommend toning down claims that LLMs find it "morally easier" to perform harmful actions as agents compared to chatbots, as the evidence provided by the complex agentic scaffolding does not fully support this assertion. They also raise concerns about the experimental setup, suggesting that differences in prompts (such as additional details given to agents) could explain behavioral changes, rather than the scaffolding itself. Additionally, they advise comparing attacks on chatbots using the same queries as those used for agents to ensure fairness. The authors have addressed these concerns effectively, and the reviewers now view the paper as a solid benchmark. Additionally, the area chair believes this paper explores an important, emerging, and under-explored area in AI safety: LLM-based browser agents. As a red-teaming effort, the study delivers impressive results and is well-structured throughout. As a result, the Area Chair recommends accepting the paper.