ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic
  Scientific Workflows
Authors: name: Qiushi Sun, name: Zhoumianze Liu, name: Chang Ma, name: Zichen Ding, name: Fangzhi Xu, name: Zhangyue Yin, name: Haiteng Zhao, name: Zhenyu Wu, name: Kanzhi Cheng, name: Zhaoyang Liu, name: Jianing Wang, name: Qintong Li, name: Xiangru Tang, name: Tianbao Xie, name: Xiachong Feng, name: Xiang Li, name: Ben Kao, name: Wenhai Wang, name: Biqing Qi, name: Lingpeng Kong, name: Zhiyong Wu

Large Language Models (LLMs) have extended their impact beyond Natural
Language Processing, substantially fostering the development of
interdisciplinary research. Recently, various LLM-based agents have been
developed to assist scientific discovery progress across multiple aspects and
domains. Among these, computer-using agents, capable of interacting with
operating systems as humans do, are paving the way to automated scientific
problem-solving and addressing routines in researchers' workflows. Recognizing
the transformative potential of these agents, we introduce ScienceBoard, which
encompasses two complementary contributions: (i) a realistic, multi-domain
environment featuring dynamic and visually rich scientific workflows with
integrated professional software, where agents can autonomously interact via
different interfaces to accelerate complex research tasks and experiments; and
(ii) a challenging benchmark of 169 high-quality, rigorously validated
real-world tasks curated by humans, spanning scientific-discovery workflows in
domains such as biochemistry, astronomy, and geoinformatics. Extensive
evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude
3.7, UI-TARS) show that, despite some promising results, they still fall short
of reliably assisting scientists in complex workflows, achieving only a 15%
overall success rate. In-depth analysis further provides valuable insights for
addressing current agent limitations and more effective design principles,
paving the way to build more capable agents for scientific discovery. Our code,
environment, and benchmark are at
https://qiushisun.github.io/ScienceBoard-Home/.