RiOSWorld: Benchmarking the Risk of Multimodal Compter-Use Agents
Authors: name: Jingyi Yang, name: Shuai Shao, name: Dongrui Liu, name: Jing Shao

With the rapid development of multimodal large language models (MLLMs), they
are increasingly deployed as autonomous computer-use agents capable of
accomplishing complex computer tasks. However, a pressing issue arises: Can the
safety risk principles designed and aligned for general MLLMs in dialogue
scenarios be effectively transferred to real-world computer-use scenarios?
Existing research on evaluating the safety risks of MLLM-based computer-use
agents suffers from several limitations: it either lacks realistic interactive
environments, or narrowly focuses on one or a few specific risk types. These
limitations ignore the complexity, variability, and diversity of real-world
environments, thereby restricting comprehensive risk evaluation for
computer-use agents. To this end, we introduce RiOSWorld, a benchmark
designed to evaluate the potential risks of MLLM-based agents during real-world
computer manipulations. Our benchmark includes 492 risky tasks spanning various
computer applications, involving web, social media, multimedia, os, email, and
office software. We categorize these risks into two major classes based on
their risk source: (i) User-originated risks and (ii) Environmental risks. For
the evaluation, we evaluate safety risks from two perspectives: (i) Risk goal
intention and (ii) Risk goal completion. Extensive experiments with multimodal
agents on RiOSWorld demonstrate that current computer-use agents
confront significant safety risks in real-world scenarios. Our findings
highlight the necessity and urgency of safety alignment for computer-use agents
in real-world computer manipulation, providing valuable insights for developing
trustworthy computer-use agents. Our benchmark is publicly available at
https://yjyddq.github.io/RiOSWorld.github.io/.