ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
Authors: Qiushi Sun
Zhoumianze Liu
Chang Ma
Zichen Ding
Fangzhi Xu
Zhangyue Yin
Haiteng Zhao
Zhenyu Wu
Kanzhi Cheng
Zhaoyang Liu
Jianing Wang
Qintong Li
Xiangru Tang
Tianbao Xie
Xiachong Feng
Xiang Li
Ben Kao
Wenhai Wang
Biqing Qi
Lingpeng Kong
Zhiyong Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Computer-using agents
AI4Research
Multimodal LLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: ScienceBoard provides a realistic environment and a diverse benchmark across six domains, enabling agents to autonomously perform scientific exploration and supporting rigorous evaluation.

Abstract: Large Language Models (LLMs) have extended their impact beyond Natural Language Processing, substantially fostering the development of interdisciplinary research. Recently, various LLM-based agents have been developed to assist scientific discovery progress across multiple aspects and domains. Among these, computer-using agents, capable of interacting with operating systems as humans do, are paving the way to automated scientific problem-solving and addressing routines in researchers' workflows. Recognizing the transformative potential of these agents, we introduce ScienceBoard, which encompasses two complementary contributions: (i) a realistic, multi-domain environment featuring dynamic and visually rich scientific workflows with integrated professional software, where agents can autonomously interact via different interfaces to accelerate complex research tasks and experiments; and (ii) a challenging benchmark of 169 high-quality, rigorously validated real-world tasks curated by humans, spanning scientific-discovery workflows in domains such as biochemistry, astronomy, and geoinformatics. Extensive evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude 3.7, UI-TARS) show that, despite some promising results, they still fall short of reliably assisting scientists in complex workflows, achieving only a 15% overall success rate. In-depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents for scientific discovery. Our code, benchmark, and leaderboard are available at https://qiushisun.github.io/ScienceBoard-Home/.

Published as a conference paper at ICLR 2026
SCIENCEBOARD: EVALUATING MULTIMODAL AUTONOMOUS AGENTS IN REALISTIC SCIENTIFIC WORKFLOWS
Qiushi Sun Zhoumianze Liu 06 Chang Ma 2
Zichen Ding Fangzhi Xu Zhangyue Yin
Haiteng Zhao Zhenyu Wu Kanzhi Cheng & Zhaoyang Liu 9 Qintong Li
Jianing Wang* Xiangru Tang Tianbao Xie Xiachong Feng Xiang Li
Ben Kao R Wenhai Wang Biqing Qi Lingpeng Kong Zhiyong Wu
The University of Hong Kong SSanghai Al Laboratory FFudan University
PPeking University Nanjing University * East China Normal University * Yale University (qiushisun, changma (liuzhoumianz) wangwenhai, qibiqing, wuzhiyong! @pjlab. org.cn
) connect. hku .hk, (kao, 1pk cs. hku.hh hk
ABSTRACT
Large Language Models (LLMs) have extended their impact beyond Natural Lan- guage Processing, substantially fostering the development of interdisciplinary research. Recently, various LLM-based agents have been developed to assist scientific discovery progress across multiple aspects and domains. Among these, computer-using agents, capable of interacting with operating systems as humans do, are paving the way automated scientific goblem--olvving and addressing routines in researchers' workflows. Recognizing the transformative potential of these agents, we introduce SCIENCEBOARD, which encompasses two complementary contri- butions: (i) a realistic, multi-domain environment featuring dynamic and visually rich scientific workflows with integrated professional software, where agents can autonomously interact via different interfaces to accelerate complex research tasks and experiments; and (ii) challenging benchmark of 169 high-quality rigorously validated real-world tasks curated by humans, spanning scientific- scientifie-discovry work- flows in domains such as biochemistry, astronomy, and geoinformatics. Extensive evaluations of agents with oftee-of-the-at backbones (e.g., GPT-5, Gemini-2.5-Po UI-TARS) show that, despite some promising results, they still fall short of reliably assisting scientists in complex workflows, achieving only 20% overall success rate. In-depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents for scientific discovery. Our code, benchmark, and leaderboard are available Scienceboard Homepage.
INTRODUCTION
In the pursuit of scientific advances, researchers combine ingenuity and expertise to perform novel research grounded in experimental explorations. In the modern era, scientific discovery increas- ingly driven by specialized software and tools that empower scientists to engage deeply with the experimental world (Hacking, 1983). Tools like simulation engines (Hollingsworth & Dror, 2018), data analysis software (The MathWorks Inc., 2022), and visualization platforms (Goddard et al., 2018) are essential for formulating hypotheses, validating results, and advancing scientific understanding.
However. as scientific software grows more sophisticated and workflows become more demanding. the learning curve and operational burden on human researchers intensify (Sänger et al., 2024). These challenges motivate the vision of autonomous agents to play a central role in automating research pipelines and assisting human researchers as "AI co-scientists" (Luo et al., 2025; Schmidgall et al., 2025; Gottweis et al., 2025). For example, while human scientist may take weeks to master protein analysis tool (Meng et al.. 2023) and spend hours making sufficient observations, an autonomous agent could perform the same tasks within minutes. By enabling fully autonomous workflows from tool usage to making novel discoveries (Lu et al., 2024a) such agents promise to accelerate science and empower researchers with unprecedented capabilities.

---

Published as a conference paper at ICLR 2026
Instruction: Predict the protein structure for the amino acid sequence of MGND....
IiADD...vvvia via AlphaFold in ChimeraX.
Step1: Toggle widget AlphaHold.
Step2: Input the given sequencee and call out AlphaFold for structure prediction.
Step3: Wait until the prediction finished.
Instruction: Show planets' orbits of Solar System in Celestia.
Step1: contect menu.
click 'Goto' in
Stup2: Slide wheel the camera away from Sol.
Step3: Click to show odbits of planets.
Figure I: SCIENCEBOARD is pioneering computer environment for scientific discovery agents, integrated with professional software. It enables agents to autonomously follow instructions and complete realistic scientific tasks by interacting with the system via GUI or CLI.
Recently emerging computer-using agents (Wu al., 2024; OpenAL, 2025), capable of operating digital devices in human-like manner, present a promising approach toward achieving these visions. These agents can interact with operating systems through Command-L.nee Interfaces (CLI; Sun et al., 2024a; Wang et al., 2024d) or perform mouse and keyboard actions via Graphical User Interfaces (GUI; Cheng al., 2024: Wu al., 2025b), mimicking the user experience flexibly automate complex workflows (Xie et al., 2024; Rawles et al., 2025; Hu et al., 2024). As illustrated in Figure 1, to predict the protein structure of amino acid sequence, the agent launches ChimeraX, selects the AlphaFold widget, and inputs the sequence for prediction. In this way, scientific tasks could be performed through step-by-step autonomous interaction with software.
To initiate the use of computer-using agents to assist human scientists with daily tasks, we intro- duce SCIENCHBOARD, a novel realistic environment designed for developing Al-powered research assistants. Our infrastructure comprises a scalable framework for scientific exploration that in- tegrates: (1) a flexible ecosystem comprising scientific software across multiple domains, and (2) standardized evaluation pipelines for rigorous assessment. It supports dual-mode interaction, allowing LLM/VLM-based computer agents to operate through either CLI or GUI.
Building upon SCIENCEBOARD, scientific experiment workflows drawn from six scientific domains, including algebra, biochemistry. theorem proving, geographic information systems, astronomy, and scientific documentation. These high-quality and challenging tasks are meticulously designed by annotators with disciplinary back- grounds, simulating the daily routines faced by human scientists. Task completion requires agents to interact with the system via CLI and GUI, exercising wide range of capabilities-incluee visual and textual reasoning, tool manipulation, coding, mathematics, spatial understanding, and deep domain- specific knowledge. Unlike widely used desktop applications, scientific software exhibits considerable complexity in I/O formats. Consequently, we reconfigure all software involved to ensure the accuracy and reliability of execution-basee evaluation. We design a suite of evaluation functions that verify task completion by retrieving the internal states of the system.
curate benchmark comprising 169 tasks that encompass
We evaluate widely used LLMs and VLMs as agents on SCIENCEBOARD, incorporating both proprictary models and their open-source counterparts. Across different observation settings, the average success rate of agents ranges between 0% to 15%, with performance peaking 20% in the most favorable subcategories. This demonstrates that current computer-using agents, while promising. remain far from capable of serving as scientific assistants, largely due their limited action capability and domain knowledge. Our analysis further reveals their inherent limitations and explores design principles for developing more agents for science.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Computer- Using Agents. Language agents (Sumers et al., 2024) have recently garnered significant attention due to their interactive capabilities (Li et al., 2023; Sun al., 2024c; Hong et al., 2024; Liu et al., 2024a). Recent studies indicate their potential to interact with operating systems and automate computer tasks as humans do, leading to the proliferation of computer-using agents (OpenAI, 2025). One line of research utilizes Command Line Interface (CLI), where agents generate executable scripts (e.g., Python or Shell scripts) interact with systems programmatically (Wang et al., 2024b). In this process, agents perform code synthesis (Sun et al., 2024a) or invoke APIs (Wu et al., 2024; Zhang et al., 2024). Another line of research focuses on Graphical User Interface (GUT) agents (Cheng et al., 2024; Wu et al., 2025b; Lin et al., 2024) that interact with digital devices through human-like mouse and keyboard actions (Niu et al., 2024; Zheng et al., 2024; Gou et al., 2025). These agents transform user instructions into executable actions within the operating system (e.g.. clicking icon scrolling through a page). Powered by VLMs, GUI agents have been applied to automate desktop (Xie et al., 2024) and mobile (Rawles al., 2025) tasks, as well specialized engineering workflows (Cao et al., 2024), showing promising paths toward digital automation. This work innovatively initiates the use of computer agents in scientific workflows, taking step closer autonomous research assistants.
AI for Scientific Discovery. The rapid advancement of LLMs has reshaped the landscape of scientific discovery (Microsoft, 2023), boosting multiple stages of the research cycle (Luo et al., 2025). With the rise of L.MM/V.M-based agents, there is growing demand for these game- changers with college-level knowledge (Wang et al., 2024a) to transcend traditional tasks like question answering (Lu et al., 2022; Krithara al., 2023; Lu et al., 2024b). Recent efforts have been directed towards harnessing such power to assist with diverse components of the research cycle, including idea and hypothesis generation (Si et al., 2024; Liu et al., 2024b), data analysis (Chen et al., 2025; Gu et al., 2024; Majumder et al., 2024), scientific programming (Tian et al., 2024; Novikov et al., 2025), paper writing (Wang et al., 2024c), and peer-reviewing (Yu et al., 2024). Mcanwhile, incorporating domain knowledge even constructing foundation models (Microsoft, 2025) can endow these agents with the capability to solve domain domain-specific problems, such as theorem proving (Song et al., 2025), chemical reasoning (Ouyang al., 2024; Tang et al., 2025) and biological discovery (Wang et al., 2025; Zhao et al., 2025; Wang et al., 2025; Frey et al., 2025). With the vision of constructing autonomous research assistants (Schmidgall et al., 2025), work represents the first to support agents in executing end-to-end scientific exploration workflows, thereby laying a cornerstone for advancing Al-powered scientific discovery.
SCIENCEBOARD ENVIRONMENT
In this part, we introduce SCIENCEBOARD environment, which encompasses real-world science software that could be manipulated through GUT and CLI interfaces. The interface is developed based on an Ubuntu virtual machine (VM), serving as the underlying infrastructure. The dynamic and visually intensive environments distinguish ScienceBoard from all previous works that evaluate the scientific capabilities of models or agents.
3.1 PRELIMINARIES AND TASK DEFINITION
A computer-using agent receives task instructions, selects actions to manipulate software, and receives feedback reflecting changes in the environment (tabletop). This interaction is modeled as a Partially Observable Markov Decision Process (POMDP). defined by the tuple A,A, where g is the goal, S is the state space, .A is the action space, O is the observation space (including environment feedback), and T S A + S is the state transition function. Given policy म, the agent predicts actions at each time step t based on the goal g and memory my oj. (f,0 (ft1.......... (0 S j t), which records the sequence of past actions and observations. The trajectory [so, ag. $1, St is determined by the policy and environment dynamics:
(T( = p(so) II
< l, 8t, m.) t=
Observation and Memory. We evaluate computer agents using three types of observation spaces: text-only, visual-only, and combined text-visual observations. For text-based observations,
3

---

Published as a conference paper at ICLR 2026
Software
Environment (Virtual Machine)
Screershot
Ally Iree
VMManager
Install
GrassGil
New
BoX
ChimenaX
APP
MONITOR
STATRS
Evaluator
OpS
Action
init
true
Taak Complete
Files
Config
init ()
predict ()
oval ()
Files
Setup
K Incomplete
Ens.
Downloads
Interact
Agent
Figure 2: Overview of the SCIENCEBOARD infrastructure. The scalable environment is built upon VM pre-installed with scientific discovery software. It supports both CLI and GUI interfaces to enable autonomous agent interaction. For each task designed to evaluate the agent's capability as research assistant, initialization script, configs, and related files provided. Agents perceive the environment through visual or textual modalities, and are expected to plan and act accordingly. After the interaction, an evaluation function determines completion based on the VM internal states.
use accessibility trees (allytree') to generate structured textual representations of sereenshots. For visual observations, we capture high-resolution screenshots directly. The specific observation combinations used in our experiments Appendix B.5. Our POMDP agent requires memory to retain historical information. Following previous work (Yao al., 2023; Ma et al., 2024), we construct this memory by concatenating the agent's 's most recent observations.
detailed in Section 5.1, with further information in
Goal and Unified Action Space. Each task specified by natural language (NL) instruction, such as Di splay atoms in sphere style, describing the user's 's intended goal. The policy model decomposes a complex goal instruction into a sequence of actions. We specially design a unified action space A in SCIENCE integrating diverse interaction modalities crucial for scientific tasks. For GUT actions, agents can perform the full range of human-computer interactions, including mouse movements, clicks, keystrokes, and other typical input behaviors in prior work (Xie et al., 2024; Zhou et al., 2024) (e.g., CLICK [991, 019]). For CLI actions, agents can interact at two levels: (a) invoking system-level commands within the Ubuntu terminal, and (b) utilizing application- specific CLI or scripting mechanisms. Moreover, A comprises an answer action, enabling agents to provide specific answers for QA tasks, and call pall_api action, allowing agents to leverage predefined external APIs to broaden their capabilities. A comprehensive list of supported action types available in Appendix B.4.
LLM/VLM-based Policy Model. An LLM / VLM model acts as the policy model to drive the agent's 's behavior. The policy model receives the current observation and generates the next action accordingly. For pure-text observation, adopt LLMs as the policy. Otherwise, we leverage VLMs.
3.2 SCIENTIFIC DISCOVERY EVALUATION FRAMEWORK
Unlike prior work that primarily focuses on static QA, coding, or single-step tasks, we aim to provide agents with realistic and visually grounded environment to support autonomous exploration, which in turn introduces greater challenges for plamning and action. In SCIENCEBOARD environment, as shown in Figure we (1) simulate scenarios where scientific software is used to solve domain-specific problems, (2) enable agents to interact with the environment through diverse observations, and (3) ensure that agent behaviors be rigorously evaluated.
"allytaa lytree: Accessibility (al (ally) trees hierarchical structures representing UT elements the screen.
4

---

Published as a conference paper at ICLR 2026
Scientific Software Installation and Adaptation. For each domain, we select open-source application that supports both visual and textual observations as the agent's playground. To enable access to the internal state of each application within the VM, we adapt the software accordingly. Given the complexity and limited completeness of scientific applications, we inject a lightweight server that launches alongside the application's main UI process expose internal states via HTTP requests. This server is capable of querying the application's 's runtime internal states, which serve as the basis for downstream evaluation. For applications that do not natively support remote control via RESTful APIs, we modify and recompile their source code to ensure that both UI elements and internal states can be accessed. In addition, the server supports partial state control of the software, allowing us to initialize with specific configurations to simulate contextualized task environments. More about the software selected and further implementation details are provided Appendix B.3,
Agent Interactions with the Environment. The LLM/VLM agent interacts with the environment as described in Section 3.1, receiving observations and executing actions accordingly. Scientific software processes these actions and returns updated states. The agent operates autonomously, continuing this loop until it outputs a signal (DONE or FAIL) or reaches the predefined attempt limit.
Table 1:
Typical evaluation cases of SCIENCEBOARD include exact matching, range-based assess- ment, and numerical tasks with tolerance. We have tailored appropriate evaluation methods for each task. Additional evaluation strategies are detailed in Appendix D.4.
Initial
Instruction
Evaluation Script (Simplified)
Select all water malecules and drow their centrolds with vadius ChimeraX.
yae' fin"",, "key"=ffelll* ( (fatio id #11/4:2018a ypp
"type" "etates",
"eidd: *"aanbda W.w.tk.nnddiit (f __naaee 17)
"ey 1 ""anbaaa k."..._.ttoo ".
vallom""" (C100.00012 1.7766 21.3672 1.11*
Display and ONLY display the layer beamdary region" Grass GIS.
"tyge" ""into,
"eenbbaa cumpilen (durg) Layers" "yalus"
"Kay " "lanbda dump:dano Layezs"; [0] ( ["nane" 1",
"boonddryy or (0
Set the Julian date to 2400000 in Celestia.
"Lyge iinfo, "Kay 1 tainniinn"
Yos low :2400000,
"gowd" #landdd left, right (lefs rights
Evaluation Pipeline. Given the complexity of scientific tasks, conventional answer-matching metrics and even execution-bar based evaluations (Xie et al., 2024; Zhou et al., 2024), often lack the granularity required to assess workflows accurately. For instance, as shown in Table 1, the rotation of a protein does not affect the correctness of visualization, whereas computational tasks in astronomy usually influenced by the current clock state. Therefore, we propose fine-gre grained evaluation based on both the correctness of key I/O during the workflow and the final state of the VM.
To handle the diverse criteria for determining task correctness (e.g.. exact matching, range- based assessment, numerical tolerance, file comparison), we design a set of evaluation templates. For each specific task, the relevant template is then instantiated with the appropriate parameters and expected gold standard values. This ensures both consistent validation and scalability for future extension. More evaluation details are in Appendix B.2.
4 SCIENCEBOARD BENCHMARK
In this section, we present the covered domains, the annotation pipeline, and statistics of the bench- mark constructed based on the SCIENCEBOARD environment.
5

---

Published as a conference paper at ICLR 2026
4.1 DOMAIN AND TASK COVERAGE
As pioneering benchmark for scientific exploration, SCIENCEBOARD spans six domains selected for their relevance to key stages of the scientific workflow, such as simulation, modeling, prediction, and knowledge (Microsoft, 2023). selecting software for each domain, we consider not only its representativeness, but also practical criteria for evaluation: open-source availability, allytree compatibility, and no requirement for user authentication.
(1) Biochemistry. We employ UCSF ChimeraX (Goddard et al., 2018; Meng et al., 2023), molecular analysis tool that supports structural modeling (e.g., AlphaFold (Jumper et al., 2021)). )). The tasks assess the agent's 's ability to manipulate biomolecular structures, as well as to reason over spatial conformations and biochem annotations.
(2) Algebra. KAlgebra a is employed to evaluate the agent's 's potential in symbolic mathematics. Tasks involve executing algebraic expressions, interpreting plots, and manipulating symbolic functions. These scenarios require the agent to exhibit strong mathematical symbolic reasoning and visual grounding capability.
(3) Theorem Proving. We use Lean (Moura Ullrich, 2021) as a proof assistant to assess agents' abilities in formal logic and deductive reasoning. The ATP tasks in this category emphasize syntactic precision and logical coherence, evaluating the agent's capability to generate semantically valid formal proofs.
(4) Geographic Information System. GrassGIS, a computational engine for raster, vector, and geospatial processing, is included examine the agent's skills in understanding terrain, hydrology, and handling ppaporltemporal data, with support for functions such as ecosystem modeling. (5) Astronomy. We integrate Ce Celestia ia, a planetarium software simulating real-world astronomi- cal scenarios. Agents must demonstrate temporal-sper -spatallll awareness and knowledge of the cosmos and celestial objects by tracking planetary systems, simulating orbital events, and querying object metadata across time and space.
(6) Scientific Documentation. To simulate research documentation workflows, we adapt and incorporate TeXstudio o to assess the agent's 's technical writing capabilities. In standalone tasks, agents are expected to compose well-structured abstracts, generate plots, and produce formal reports based on provided instructions. In cross- apossiapplio scenarios, TeXstudia is coupled with the aforementioned software to evaluate whether agents can extract meaningful insights from experiments and synthesize them into coherent narratives.
These domains enable evaluating a science agent's 's capabilities across multiple dimensions, including visual/ / textual reasoning, math, coding, tool use, spatial understanding, domain-specific knowledge, and more. Additionally, to explore the potential for end-to-end scientific automation, documentation tasks are integrated with other domains to support cross- apossapplicntion workflows- such as automatically generating an experimental report based on completed upstream tasks. More details about the software platforms used instantiate and convey the tasks SCIENCEBOARD are provided in Appendix B.3.
Agestic epet Bedaeraary....
Diffcallt:
Agesti Paoege
Leura
Agesti Pounge
BilGooly
Talk: Elaplas alan ophare dtya and
Bintadly
Task
Step 1: Learn Tutorials and Handbooks
Step 2: Task Curation
Step 3: Task Formalization and Verification
Chack & Validation
Execution
App Install
Write Code
File Download
Step Write Evaluation Function
Step 4: Task Configuration
Figure 3: The annotation pipeline of the tasks in SCIENCEBOARD benchmark.

---

Published as a conference paper at ICLR 2026
4.2 TASK ANNOTATION PIPELINE
To effectively construct tasks that are appropriately challenging, diverse, and aligned with the features of scientific software, we leverage pipeline that spans from training annotators with tutorials and handbooks conducting execution-based validation, as shown in Figure 3.
(1) Tutorial Learning. Five annotators initially collect and learn from tutorials and handbooks related to the software. After that, each annotator studies and explores a software's 's basic unit operations, e.g., plotting the Bernoulli lemniscate in KAlgebra. Details are in Appendix D.I. (2) Task Curation. Each annotator selects scientific software, installs it within SCIENCEBOARD, and begins drafting task instructions based on its functionalities. Task types include but are not limited to: configuration, simulation, QA, and domain -peeif- expertise. Each task is tentatively assigned difficulty. Thereafter, agentic prompts aligned with the drafted tasks will be curated. (3) Formalization and Selection. Different annotators exhibit varying linguistic habits, we employ ChatGPT to standardize the task format. Annotators then conduct cross-check, excluding those lacking diversity, poor executability, non-unique answers, finalize the set of tasks for use. (4) Configuration Function Writing, The purpose of this step is to initialize the software and pro- vide specific contexts, egg. supplying map for GIS tasks or a protein sequence for biochemistry tasks. Annotators will write a set of functions for each software to modify the VM status, i.e., the internal state of the software, along with general configuration functions (e.g. downloading required files). Tasks commence only after all initialization have been successfully executed. (5) Evaluation Function Writing and Validation. Evaluation functions are developed to assess task outcomes rigorously. As described in Section 3.2, evaluations are state-based, with functions derived from base evaluator template. Annotators retrieve the task state from the VM and assess based on criteria such as I/O matching and predefined ranges. The function returns either "task complete" or "task fail." Cross-validation is performed for consistency, with each task executed by two randomly selected annotators on separate VMs. The results are analyzed to ensure the evaluator's is correctness, even under intentional attempts by annotators to deceive the system.
4.3
TASK STATISTICS
The task statistics of SCIENCEBOARD benchmark are presented in Table 2. Specifically, it comprises 169 unique tasks across domains, with task difficulty categorized into three levels. We curate balanced number of tasks that are representative enough while keeping the evaluation cost manageable. During annotation, we define multiple task types to evaluate agents' ability to perform diverse operation flows and leverage domain-specift knowledge.
Table 2: Statistics of SCIENCEBOARD.
Raric Upe.
Bivee Heareroun
Dan Frediass
The Heiernaai
Sallwan Cundit
Task Type
Statistics
Asabon
Total Tasks
169 (100 %
Strect Acaipua
15sS wapebres
Gometty
GUI
38 (22.5%)
+UI
CL.I
98 (58.0%)
33 (19.5%)
tnterp.
() (17.3%)
Loan L.S. sau
Aloore
Fere
Difficulty
spunt Prodiction
Tick soudho fe 9..%%)
Bailin
Easy
91 (53.8%)
Buntc Settage
Celontia (10.00)
Lag
Hard
Medium
28 (16.6%)
48 (28.4%)
Speiil Rozeentgg
E20 290
Data Eaiting
Reae taego
Open Problems
2 (1.2%)
lmsan Aundrade
Data Denter
Instructions
Avg. Lenggth of Task Instructions
20.0
Diepker
Avg. Length of Agentic Prompt
374.9
Execution
Avg. Steps
9.0
Avg. Time Consumption
124(s)
Figure 4: Distribution of tasks in SCIENCE- BOARD benchmark.
The distribution of task types is shown Figure 4. Beyond the innovation of a realistic environment, SCIRNCEBOARD benchmark also improves upon prior work in terms of task design and content
7

---

Published as a conference paper at ICLR 2026
diversity. More details about task diversity, stability analysis, and comparison with representative scientific benchmarks are provided in Appendix D.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Backbones. We employ three types of backbones for agents. These include proprietary models:
GPT-4o (Hurst al., 2024), GPT-5 (OpenAI. 2025b), Claude 7-Sonnet (Anthropic AI, 2024), Gemi ni ni-2.00 0-F Fashh (Gemini Team, 2024), 1i.iiii-2.5PP... (Gemini Team, 2025) and o3-miní (OpenAI, 2025a): open-source models: Qwen2.5-VLL-72B-nnss
(Bai et al., 2025), Intern VL3-78B (Chen et al., 2024), QvQ-72B- Pree-rrw (Qwen Team, 2024), and GPT-oss- 120B (Open AI, 2025); and GUI action models: 3S-AAtas-Pro-71 (Wu et al., 2025b), UGround-VI- (1-TARS--1.577) (Qin al., 2025), and GUI-Actor- 7B (Wu et al., 2025a). More details in Appendix E.I.
7B (Gou et al., 2025), UI-TARS-72B-
Observation Space. We follow established observation settings (Xie et al., 2024; Zhou et al., 2024): (I) full desktop screenshots; (2) allytree, structured text-only representation; (3) Screenshots + ree; and (4) Set-of-Marks (Yang et al., 2023), which partitions images into marked regions to aid grounding. Further details are in Appendix B.5.
5.2 RESULTS
We compare the performance of computer-use agents powered by different LLMs and VLMs on SCIENCEBOARD, as presented in Table 3. We summarize our key empirical findings follows:
Performance Hierarchy. Existing agents remain far from being capable of effectively assisting human scientists in completing real-world scientific exploration tasks. Even SOTA models, such as 5PT-5 and Gemini- -2. 5rroo, achieve an average success rate of only 20%. Across various settings, open-source counterparts can partially match proprietary models. However, they still exhibit markedly lower overall performance, with an average success rate of less than 12% and approaching nearly 0% in some task categories. The gap between agent and human performance underscores the limitations the status quo and necessitates further research.
Performance Insights. Across domains, we observe clear performance imbal- ances: models perform moderately well on Algebra and Biochemistry but degrade notably on GIS and Astronomy, We attribute this to: (1) Interfaces: Algebra and Biochemistry tasks often support both CLI and GUI execution, whereas GIS and Astronomy rely mainly on GUI interactions. Agents generally handle CLI commands more reliably than finennrained GUI grounding, which demands precise visual localization. (2) Task emphasis: Geographical and astronomical tasks involve dense visual elements (e.g., maps, star charts), making it difficult for agents to identify and reason over relevant information. This also indicates the limited 3D spatial reasoning ability of current VLMss
Domain-Specific
Different observation modalities have significant impact. Overall, a 1ytree- + screenshots setting yields the best performance. In other settings, Qwen2.5- LL performs exceptionally well under screenshot setting, which we attribute to its advanced GUI ability. Under al llytrees the attribute information of elements allows LLMs to complete certain tasks by relying solely on textual observations. Meanwhile, we observe that the SoM sometimes introduces negative effects. It is likely that although SoM provides bounding boxes to ease grounding, scientific software often contains massive elements on screen (e.g., dense celestial objects and complex cosmic backgrounds), which introduces substantial noise and increases the difficulty of visual reasoning.
Impact of Different Observations.
Impact of Compute Scaling and Native Reasoning. To determine if test-time compute can mitigate these limitations, we further evaluate frontier models with extended generation limits and native reasoning modes in Appendix F.4. Our findings indicate that scaling inference-time compute yields tangible but bounded improvements. For instance, allocating high reasoning effort increases success rates in the Biochemistry and GIS domains. However, this scaling does not fundamentally resolve the overall performance deficit. This suggests that the primary bottleneck for current agents in scientific workflows is not merely the depth of cognitive reasoning, but rather general computer-usin using and agentic capabilities, which require accurately perceiving dense, domain-specific elements and translating high-level scientific plans into precise, executable actions.
8

---

Published as a conference paper at ICLR 2026
Table 3: Success rates on SCIENCEBOARD. We present the performance of each agent back- bone across different scientific domains under various observation settings. Proprietary Models Open-Source VLMs LLMs. and GUI Action Model are distinguished by color
Success Rate (1)
Obs.
Model
Algebra
Biochem
GIS
ATP
Astron
Doc
Overall
GPT-5
6.45%
24.1 14.14%
0.00%
0.00%
12.12%
12.50%
9.20%
GPT-4o
3.23%
0.00%
0.00%
0.00%
0.00%
6.25%
1.58%
Claude-3. 7-Soane:
9.67%
37.93%
2.94%
0.00%
6.06%
6.25%
10.48%
6.45%
31.03%
0.00%
0.00%
0.00%
12.50%
8.33%
Screenshot
Gemini-24
6.45%
3.45%
2.94%
0.00%
0.00%
6.06%
3.15%
Qwen2.5-VL-72B
22.58%
27 .7.5%
5.88%
0.00%
9.09%
12.50%
12.94%
InternVLL3-78B
6.45%
3.45%
0.00%
0.00%
0.00%
6.25%
2.69%
UI-TARS-1.5-7B
12.90%
13.79%
0.00%
0.00%
6.06%
0.00%
2.69%
12.90%
20.69%
2.94%
0.00%
6.06%
0.00%
7.10%
Claude-3 7-Sonne
19.35%
34.48%
2.94%
3.85%
12.12%
0.00%
12.12%
Gem1 -1-200-F Fssh
9.68%
17.24%
0.00%
0.00%
0.00%
0.00%
4.49%
allytree
o3-mini
16.13%
20.69%
2.94%
3.85%
15.15%
6.25%
10.84%
Qwen2.5-VL -BB
9.68%
10.34%
2.94%
0.00%
3.03%
0.00%
4.33%
3.23%
3.45%
0.00%
0.00%
0.00%
0.00%
1.11%
GPT-oss-1200
19.35%
13.79%
0.00%
0.00%
9.09%
0.00%
7.04%
GPT-5
41.93%
62.07%
5.88%
7.69%
15.15%
12.50%
24.20%
GPT-4o
22.58%
37 9.3%%
2.94%
7.69%
3.03%
12.50%
14.45%
Claude-33, 7-8onne:
12.90%
41.37%
8.82%
3.85%
9.09%
18.75%
15.79%
Screenshot
+ #allytree
Gemini 2.5 Pro
16.13%
55.17%
2.94%
0.00%
15.15%
12.50%
16.98%
Gemini-2.00 -0-F2.0--ll
16.13%
24.14%
2.94%
0.00%
18.18%
12.50%
12.32%
Qwen2.5-V1-722
16.13%
20.69%
2.94%
0.00%
18.18%
12.50%
11.74%
VL3-77B
6.45%
3.45%
0.00%
0.00%
3.03%
6.25%
3.20%
GPT-4o
6.45%
3.45%
0.00%
0.00%
3.03%
12.50%
4.24%
Claude 7-87onnn
16.13%
31.03%
5,88%
0.00%
6.06%
12.50%
11.93%
Gemini 2.0 Flash
3.23%
0.00%
0.00%
0.00%
3.03%
6.25%
2.09%
Sct-of-Mark
Qwen2.5-VL-72B
6.45%
6.90%
2.94%
0.00%
3.03%
12.50%
6.36%
QvQ-72B-Preet Prevview
0.00%
0.00%
2.94%
0.00%
3.03%
0.00%
0.49%
InternV1.3-78B
3.23%
6.90%
2.94%
0.00%
0.00%
0.00%
2.18%
Human Performance
74.19%
68.97%
55.88%
42.31%
51.52%
68.75%
60.27%
ANALYSIS
To further investigate the factors influencing agents' capabilities, we conduct additional analysis to understand the underlying causes and the behavioral differences among heterogeneous models.
GUI vs. Hybrid. Some tasks support both GUI and CLI as interchangeable interfaces. For exam- ple, ChimeraX offers nearly full functional coverage
20
GUT + CLI Only
Success Rate (%))
through both modes for biochemistry tasks. To test how computer-using agents handle such hybrid software, we disable ChimeraX's 's CLI, enforcing GUI-only execu- tion (allytree + screenshot). As shown in Figure 5, GPT-4o and Intern VL3 suffer clear drops in perfor- mance, whereas Qwen2.5-VV VL remains largely unaffected, indicating better adaptation GUT execution.
15
GP1-40
Qwen2.5-VL
InternVL3
Figure 5: GUI + CLI v.s. GUI Only.

---

Published as a conference paper at ICLR 2026
These results suggest that future agents should be more adaptable and equipped with stronger GUI capabilities to remain robust across hybrid and vision-only settings. Extended analyses are provided in Appendix F.
Disentangled Planning and Action. Observations from failure cases indicate that some models, such as GPT-4o, can effectively plan tasks but lack sufficient grounding capabilities. Therefore, we explore separating planning and action. Following existing practices (Wu et al., 2025b), we configure GPT-4o as the planner and utilize various VL and GUI action models as the grounding models.
Table Success rates different VLM agent combinations under the planner + grounding model setting on SCIENCEBOARD, The observation setting used in this experiment is screenshot. Colors denote Proprietary Models Open-Source VLMs and GUI Action Models.
Success Rate
Planner
Grounding Model
Algehra
Biochem
GIS
Astron
Overall
OS-Atlas Pro-7B
OS-AtlasPPro-7B
6.25%
10.34%
0.00%
3.03%
4.92%
UGround-V V1-7B
0,00%
3,45%
0.00%
3.03%
1.62%
CPT-4o
Dwen2.55VVL772
12.50%
34.48%
11.76%
9.09%
16.96%
UI-TARS-72B
3.23%
10.34%
5.88%
6.06%
6.38%
GUI-Acto--7B
21.88%
44.83%
2.94%
12. 12%
20.44%
GPT-40
3.23%
0.00%
0.00%
0.00%
0.81%
The results in Table show that modular approaches yield significant improvements and are promising for tackling complex and visually demanding tasks in scientific software workflows.
CONCLUSION
We propose SCIENCEBOARD, a first-of-its-kndd realistic environment designed to empower au- tonomous agents in scientific exploration with rigorous validation. Building upon our infrastructure. we curate highly challenging benchmark of diverse scientific tasks meticulously crafted by human experts. Through extensive experiments and analysis, we found that even state-of-the-art computer- using agents perform significantly below human-level proficiency. Although the realization of autonomous agents for scientific discovery remains distant goal, this work offers actionable insights for future development, and we believe constitutes advancing Al- Al-powered scientific discovery.
ETHICS STATEMENT
Computer-using agents operating in live OS environments could potentially affect the normal func- tioning of the system. This non-negligible in scientific workflows, where poorly controlled agent could potentially misconfigure experiments, corrupt sensitive research data, or even lead to irreversible data loss. However, considering that all settings in this work are conducted within isolated virtual environments, we do not view this as concern.
ACKNOWLEDGEMENT
We would like to express our sincere gratitude to the anonymous reviewers from ICLR and the WCUA @ ICML. workshop for their insightful comments and valuable suggestions, which have significantly helped improve the quality of this work. This research is supported by the WYNG Foundation (HKU 25AG100407). We gratefully acknowledge their support.
REFERENCES
Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional gencralsstsppeciiaist framework for computer use agents, 2025. URL https: / wwwp...rgg/bs//500-e5e 009066.
10

---

Published as a conference paper at ICLR 2026
Angelos Angelopoulos, James F. Cahoon, and Ron Alterovitz. Transforming science labs into automated factories of discovery. Science Robotics, (95)xaddm699), 2024. doi: (tt2///scioooooiic.aamm66699 sci robort iss. adm6 991.
TRLhttp::/ httpps://www... http::/wwwwwween..ee.... www.ssc.eeccc........... org /t///oi/aaaa aaa//b///// 1126/
Anthropic AI. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 1:1, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL https:/ pettps//aaxxte /aaxivv....// org/ als// 2502. 13923.
Benjamin Burger, Phillip M Maffettone, Vladimir V Gusev, Catherine M Aitchison, Yang Bai, Xiaoyan Wang, Xiaobo Li, Ben Alston, Buyi Li, Rob Clowes, et A mobile robotic chemist. Nature, 583(7815):237 241, 2020.
Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xiong Xinzhuang. Hanchong Zhang, Wenjing Hu, Yuchen Mao, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Vic- tor Zhong. Lu Chen, Kai Yu, and Tao Yu. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? In The Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2024. URL https:  //operreview..o net/ wwr.mforumiddiemee www.mmmi2i2ii xS.
Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint wrXiv:2412.05271 2024.
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific dis- covery. In The Thirleenth International Conference on Learning Representations, 2025. URL https: wwwpeeeeeeeeeeeeeeeeeeeeeeeeeeee net forum?ide 6
Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu.
SeeClick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9313-9332, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL long. 505.
https: http///aaaaaataaa/ae/aa/eaaaaaaa www.tntwll.y.ooo//00.1ee wrg//024 2024 acl
Nathan Frey, Isidro Hötzel, Samuel D Stanton, Ryan Kelly, Robert Alberstein, Emily Makowski, Karolis Martinkus, Daniel Berenberg, Jack Bevers III, Tyler Bryson, al. Lab-in-the-looy therapeutic antibody design with deep learning. bioRxiv, pp. 2025 02, 2025.
Gemini Team. Introducing gemini 2.0: our new ai model for the agentic era, 2024.
Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality. long context,
and next generation agentic capabilities. arXiv preprint arXiv:2507.06260 2025.
Alireza Ghafarollahi and Markus J Buchler. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning, arXiv preprint arXiv:2409.05556 2024.
Thomas D. Goddard, Conrad C. Huang, Elaine C. Meng, Eric F. Pettersen, Gregory S. Couch, John H. Morris, and Thomas E. Ferrin. Uesf chimerax: Meeting modern challenges in visualization and analysis. Protein Science, 27(1):14-25, 2018. doi: http:://ooi.rrg/// http://.....o/00...... wiley com/ doi abs 10. 1002/pro......// 3235.
http///.....//.......o 1002/pro.3235. URL
ht https: http:///nnnnnnibarre
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou. Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan,

---

Published as a conference paper at ICLR 2026
Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, José R Penadés, Gary Peltz, Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards an ai co-scientist, 2025. URL https: / arxiv.. org/ wttg/bs/////e/// abs/2502 18864.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for GUT agents. In The Thirteenth International Conference on Learning Representations, 2025. URL net/ forum?iddxxnooeeae
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard- -ohhrrd-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A Mer- rill, Jeffrey Heer, and Tim Althoff. BLADE: Benchmarking language model agents for data- driven science. Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 13936-13971, Miami, Florida, USA. November 2024. Association for Computational Linguistics. doi: 10. 20.11653/v11220 1/2024 www..e/...iiiinggnnn......... URL https htt///aaaaaoolloaoeoeeeoo org/202g findings- emnip. 815/.
Ian Hacking. Representing and intervening: Introductory topics in the philosophy natural science. Cambridge university press. 1983.
Scott A Hollingsworth and Ron O Dror. Molecular dynamics simulation for all. Neuron, 99(6):
1129-1143, 2018.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations. 2024. URL https::////.......t. wwt.rppenrvvieeeeeee net. forum? (at.:/iitVtttmeTTT.
Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A preliminary case study with claude 3.5 computer use. arXiv preprint arXiv:2411. 10323, 2024.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, and Zhiyong Wu. Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant. arXiv preprint arXiv:2410.1880 2024a.
Chengyou Jia, Changliang Xia, Zhuohang Dang, Weijia Wu, Hangwei Qian, and Minnan Luo. Chat- gen: Automatic texi-to-image generation from freestyle chatting. arXiv preprint arXiv:2411.1717. 2024b.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583 589, Aug 2021. ISSN 1476-4687. doi: 10.
URL www.thtts///// ht:./ooioooooo weriiiorg/1000 10. 10.00.1...0 1038/ 4 1 021 138819922
Jing Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po- Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649 2024.
Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. Bioasq- qu: A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170, 2023.
12

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium Operating Systems Principles, pp. 61 1-626, 2023.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative agents for "mind" exploration of large language model society. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
wtt///orummmio//// Woworrmmid-Iyyy220DD //openreview. htwp:r//wweeenteooourieee-----
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Mul- timodal ArXiv: dataset for improving scientific comprehension of large vision-language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14369-14387, Bangkok, Thailand, August 2024. As- sociation for Computational Linguisties. doi: 10.18653/v 11//3//1122222 www.2.2................ //acla aollaanhoololl ooy. org 20222 aec...aclllc. wlng...llon..................... 775/-
URL https:
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhiwei Li, Bao-Long Bi, Ling-Rui Mei, Junfeng Fang, Zhijiang Guo, Le Song, and Cheng-Lin Liu. From system to system 2: A survey of reasoning large language models, 2025. URL https: //arxiv. org/ abs 2502.17419.
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One lasion-language--e action model for gui visual agent, 2024. URL https: / wrxiv. wrw.ii.orrrooooooo.oo abs/ 22ls/ 17465.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang. Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs agents. In The Twelfth International Conference on Learning Representations, 2024a. URL https: /pppneeeeeeeeeeeeeeee/eeeeeereee net forum?id=zAdUUOeer
Zijun Liu, Kaiming Liu, Yiqi Zhu, Xuanyu Lei, Zonghan Yang, Zhenhe Zhang, Peng Li, and Yang Liu. Aigs: Generating science from ai-powered automated falsification, 2024b. URL
https:/ / //trxv/vvv////////////eee/e/wave www.aewwwww...a.a./..eooeee..eee wwtg/bss/////eooooooo /241 11910.
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.0629) 2024a.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Pe- ter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022. URL net wwwaamidd--jw-----
Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao, Hai-Tao Zheng, and Yu Li. MoleculeQA: A dataset evaluate factual accuracy in molecular comprehen- sion. In Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 3769- 3789, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 111865531122222 findings-emnlp... findings-emnipp 216/.
URL https:tt///// httpss///acaatthlogg.. org 022/ 2024.
Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. Llm4sr: A survey large language models for scientific research, 2025. URL https:/ /aaxivivv.... wwt.r/orggab/b//aeo.o/.o /2501 04306.
Jakub Lála, Odhran O'Donoguue, Aleksandar Shtedritski, Sam Cox, Samuel G. Rodriques, and Andrew D. White. Paperqa: preprint arXiv:2312.07559, 2024. URL https: / /w/. orgi 48550/455000 wat.////rri...... 2312. 07559.
generative agent for scientific research. arXiv
13

---

Published as a conference paper at ICLR 2026
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-tura LLM agents. In The Tiiht Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL. https: htpp//ppeeeeeeeeeeeeeeeeeeeeeeee net/ forum? d=4S8agvK
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models, 2024. URL
https://arxtie wtg/abs////ooooooooooooooo.....s 2400 01725.
Elaine C. Meng, Thomas D. Goddard. Eric F. Pettersen, Greg S. Couch, Zach J. Pearson, John H. Morris, and Thomas E. Ferrin. Ucsf chimerax: Tools for structure building and analysis. Protein Science, 32(11))e4792, 2023. doi: http://dii.or//11.000/////// https: htt////nniineee htt////mnnnnibaarrrrrr.eeaaaaaaa wiley. com/ doi/ /as///a/////a// 10. 1002 /pro. 4792.
URL
Microsoft. The impact of large language models scientific discovery: a preliminary study using gpt-4, arXiv preprint arXiv:2311.07366 2023.
Microsoft. Nature language model: Deciphering the language of nature for scientific discovery, 2025.
URL https::pp////c /ttp://aaxe wttpiiiiiii.///.ia//..aae..aatva org/ abs 2502 07527.
Leonardo de Moura and Sebastian Ullrich. The lean theorem prover and programming language. In Automated Deduction CADE 28: 28th International Conference Automated Deduction, Virtual Event, July 12-15, 2021, Proceedings, pp. 625-635, Berlin, Heidelberg, 2021. Springer-Verlag Verlag. ISBN 978-3-030-79875- doi: 10.
URL https: /toii/oi.../ org/
10.1007/900 978 3 030 7 798778878 6 5_37.
Runliang Niu, Jindong Li, Shiqi Wang. Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Chang, and Qi Wang. Screenagent: vision language model-driven computer control agent. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI '24, 2024. URL 24963/ (44963/ijcaa iiiciiicii... 2024 /111.
https: / /oiiiiioo////// org/10 waargg/....996.0
Alexander Novikov, Ngån Vü, Marvin Eisenberger. Emilien Dupont, Po-Sen Huang, Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri. George Holland, Alex Davies, Sebastian Nowozin, Push- meet Kohli, and Matej Balog. Alphaevolve: A coding agent for scientific and algorithmic discovery. https://deepmind. https://ceemmiddggggeddevveeeeee http:///wwweeeooogaapaaeeeeeeeee
agent-
2025.
OpenAl. Computer-using agent: Introducing universal interface for ai to interact with the digital world, 2025. URL https: / openai. com/ index/ http///nndteeteererre///// usmputer-usi using-------------e agent.
Open AI. gpt-oss- 120b gpt-oss-20b model card. gpt-oss model card, 1:1, 2025.
OpenAl. Openai 03-mini system card, 2025a.
OpenAl. Openai gpt-5 system card, 2025b. URL
022..00000.///2000.22.....11 03267.
Siru Ouyang, Zhuosheng Zhang, Bing Yan, Xuan Liu, Yejin Choi, Jiawei Han, and Lianhui Qin. Struc- tured chemistry reasoning with large language models. In Proceedings of the 41 International Conference on Machine Learning, ICML 24. JMLR.org. 2024.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents, arXiv preprint arXiv:2501. 12326. 2025.
Qwen Team. Qvq: To the world with wisdom, December 2024. URL wwwphhttp//cc httppps/////:nnnnm github. wt/bi///lgg/o wtbp///--7-pppiieee
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E Bishop. Wei Li, Folawiyo Campbell-Ajala, Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy P Lillicrap. and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. In The Thirteenth International Conference on Learning Representations, 2025. URL ht https: /openeeviee net forum? id=i 15egg1yUgsrc..
14

---

Published as a conference paper at ICLR 2026
Samuel Schmidgall, Yusheng Su, Ze Wang. Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants, 2025.
URL https: /awaaaiivvvooo.../.a.a org/ abs/ 04227.
Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can ilms generate novel research ideas? large-scale human study with 100+ nlp researchers. arXiv preprint M4Xi:2449.041099 2024.
Peiyang Song, Kaiyu Yang, and Anima Anandkumar. Towards large language models as copilots for theorem proving in lean, arXiv preprint arXiv:2404. 12534, 2025.
Theodore Sumers, Shunyu Yao. Karthik Narasimhan, and Thomas Griffiths. Cogmitive architectures for language agents. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https:: /ttp:://ppenne wwtp//eeeeeeeeeeeee/ee/e. net / forum? (ireummmdd111 62c.vil10 Survey Certification.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403,14734 2024a.
Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412. 19723, 2024b.
Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. First Conference on Language Modeling. 2024c. URL https: openreview, net fore//oruuutt rum 71mim==m7mmTo
Mario Sänger, Ninon DeMecquenem, Katarzyna Ewa Lewinska, Vasilis Bountris, Fabian Lehmann, Ulf Leser, and Thomas Kosch. A qualitative assessment using chatgpt as large language model for scientific workflow development. GigaScience, 13, 2024. ISSN 2047-217X. doi: 1093/
gigascience/giae0000. URL http::tt///. /xx doi. org /1tg///0.100 10 099 htt.:////ggaaaaageaaaeege
Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, Arman Cohan, and Mark Gerstein. Chemagent: Self-updating memories in large language models improves chemical reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttps:///ppeeeeee openrevi... net/ net/forum? forum? i d=
The MathWorks Inc. Statistics and machine learning toolbox documentation, 2022. URL https: //www, wwt.works. com/ help/ sttats/innee html.
Minyang Tian, Luyu Gao, Dylan Zhang. Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, HAO TONG, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du Tianhua Tao, Ofir Press, Jamie Callan, Eliu A Huerta, and Hao Peng. Scicode: A research coding benchmark curated by scientists. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: //openreview.o net forum? id witemiiiddAdddeeae.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 986))2579-2665, 2008. URL http: //mm///m.../////////. arg/papers/v9/
vandermaaten html.
Hanchen Wang, Yichun He, Paula P Coelho, Matthew Bucci, Abbas Nazir, Bob Chen, Linh Trinh, Serena Zhang, Kexin Huang, Vineethkrishna Chandrasekar, et al. Spatialagent: An autonomous ai agent for spatial biology. bioRxiv, pp. 2025-04, 2025.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating college-level scientific problem-solving abilities of large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 00622-00699 PMLR, 21-27 Jul 2024a. URL https: / https://proceedt ngs ml press v235 wang 4 z. html.
15

---

Published as a conference paper at ICLR 2026
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li. Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In Proceedings of the 41st International Conference on Machine Learning, ICML'24 24. JMLR.org, 2024b.
Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. Autosur- vey: Large language models can automatically write surveys. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 115119-115145. Curran Associates, Inc., neurips wwtpaper_fiiiieeeeeeee ferrrfiiee/pppppp pttpes//pppe//22// Paper- pdf.
2024c. URL https::/pproeee
Zhiruo Wang, Zhoujun Cheng, Hao Zhu, Daniel Fried, and Graham Neubig. What are tools anyway? survey from the language model perspective. In First Conference on Language Modeling, 2024d.
URL wttptttt///// /tt:://pppeee openrevieweeeeeeeeeeeeeee /tt/ wiiiddihhiiiiss
Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang. Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. Gui-actor: Coordinate-free visual grounding for gui agents, arXiv preprint arXiv:2506.03/1 03143, 2025a.
Zhiyong Wu, Chengeheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement, 2024. URL https: arxiv. org/ abs / 22002 07456.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng. Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. OS-ATLAS: Foundation action model for generalist GUI agents. In The Thirteenth International Conference Learning Representations, pestps://opente /pppppppeeeieeeeeeeee/eeeeeeeeee net / forum? -
2025b. URL https::////
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng. Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024. URL https: // //openreview net
Yiheng Xu, Hongjin SU, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang. Caiming Xiong, and Tao Yu. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference on Learning Representations, 2024a. URL https: BttLhtt///pppeeeeiiie /opp///eeeeeeee.ee/e
net/forum: forum? ? d=hhhhsSmttRRi
Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction, 2024b.
Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv: 11441, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https: ///pppppeeveeeeee net forum?
id-=E_vluYUI--
Jianxiang Yu, Zichen Ding, Jiaqi Tan, Kangyang Luo, Zhenmin Weng, Chenghua Gong, Long Zeng, RenJing Cui, Chengcheng Han, Qiushi Sun, et al. Automated peer reviewing in paper sea: Standardization, evaluation, and analysis. In Findings of the Association for Computational Linguistics:
EMNLP 2024, pp. 10164-10184, 2024.
Chaoyun Zhang, Liqun Li. Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Ufo: A ui-focused agent for windows os interaction, 2024.
16

---

Published as a conference paper at ICLR 2026
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 3132-3149, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.
findings-acl. 186. URL https:: a aaaalaaanooooooooooooooaaooooooa ogy. org/2024. Findings-ach 186/.
Haiteng Zhao, Chang Ma, Fangzhi Xu, Lingpeng Kong, and Zhi-Hong Deng. Biomaze: Bench- marking and enhancing large language models for biological pathway reasoning. arXiv preprint arXiv:2502.16660 2025.
Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v(ision) is generalist web agent, if grounded. In Forty-first International Conference on Machine Learning, 2024. URL
https: htttps://operrevee ew net / Corum?id=pieckKKD
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL https: wpwp//eeeneeeeeeeeeeeeeeee net forum? id oon
LARGE LANGUAGE MODEL USAGE
In this submission, employed LLMs aid and polish writing, including grammar and typo checking, as well as for identifying related works.
LIMITATIONS AND BROADER IMPACTS
As pioneering effort marking the early stages of integrating computer-using agents into scientific workflows, is important to acknowledge certain limitations. While our current evaluation, based on both VM states and key I/O correctness, provides robust validation, its reliance on a binary success flag may not fully capture process correctness or partial task completion (e.g., an agent succeeding in most steps but failing at final one). Introducing "partial credit" could offer more granular evaluation, but accurately defining and implementing such system for open-ended, OS-level tasks within diverse scientific software presents significant challenges due vast state action spaces. One potential direction for improvement is to introduce VLMSs to serve as judges capable of assigning partial credit and providing richer feedback. We leave this future work.
A DISCUSSION AND FUTURE DIRECTIONS
SCIENCEBOARD represents significant advance in using autonomous agents for scientific workflows. Our findings suggest several key directions for future research:
Harmonized Domain Knowledge and Agentic Capability. Our evaluations suggest that one contributing factor to current agents' limitations in scientific exploration is their insufficient domain knowledge. For instance, the GUI action models we evaluated, while effective at automation, lack the specialized understanding required for complex scientific tasks. Therefore, future advancements may focus on enhancing domain-specific abilities, such as enhancing scientific comprehension (Li et al., 2024), learning from highly relevant resources such manuals and tutorials, and enabling on-demand knowledge retrieval (Lála et al., 2024). A key challenge will be to effectively harmonize this specialized knowledge with general agentic capabilities (Xu et al., 2024a).
Collaborative and Specialized Agents as a Solution. Analysis in Table 4 indicates that even a basic modular approach of separating planning and action to different agents can yield significant performance improvements in complex scientific software workflows. This points toward developing sophisticated multi-agent systems composed of specialized, heterogeneous agents (Jia et al., 2024a; Ghafarollahi & Buchler, 2024; Agashe et al., 2025). For example, responsibilities could be disentan- gled by assigning planning to agents capable of deep reasoning (Li et al., 2025), action execution to specialized GUI action models (Wu et al., 2025b; Xu et al., 2024b), and domain-specinf capability to models in particular disciplines (Microsoft, 2023; 2025). These agents could be -nd---aad-
17

---

Published as a conference paper at ICLR 2026
allowing flexible application across broader aspects of the scientific lifecycle, such as data analy- sis (Chen et al., 2025), scientific plotting (Jia et al., 2024b), and paper revision (Yu et al., 2024). While promising, it also demands more sophisticated multi-agent designs manage and coordinate the intricate and multifaceted nature of scientific tasks.
Extending Digital Agents to Physical Laboratory. Current Al-assisted scientific workflows are primarily at the digital level, focusing tasks such data analysis, simulation, and software control. A natural and impactful next step is to extend the capabilities of such autonomous agents, as fostered and benchmarked in SCIENCEBOARD. into physical laboratory environments. This transition involves interfacing agents with robotic systems (Burger et al., 2020; Angelopoulos et al., 2024), applying principles of embodied AI perceive and interact with the physical world. Agents would manipulate laboratory instruments and samples, carry out experimental protocols, and monitor physical processes in real time, thereby fostering a tab-inntheeloopp (Frey al., 2025) future where experimentation and Al-driven methods are mutually reinforcing.
B DETAILS OF SCIENCEBOARD ENVIRONMENT
B.1 ENVIRONMENT SETUP
Virtual machines can operate their own kernel and system, enabling compatibility with wide variety of operating systems. For experiments covered in this paper, utilize Linux environment (Ubuntu 22.04.1 LTS with kernel 6.8.0-57 6.8.0-57-generic) running on x64 personal computers.
B.2 EVALUATION CRITERIA
As stated in Section 3.2, we employ a fine-grained evaluation methodology based on:
The final state of the VM (Determinant)
1/O states and intermediate steps NNon-Determinant)
While the final state of the VM often provides determinant measure of overall task completion, the diverse nature of 1/O and intermediate steps necessitates a varied set of criteria. The following outlines the primary principles applied for I/O correctness:
Exact Match:
Strict equality: The output or relevant state must be exactly identical to the gold standard (e.g.. for specific textual outputs numerical values).
Set equality of lines: For multi-line textual outputs, the content of all lines must match the gold standard, but their order may not be strictly enforced.
(Question-answeriii The agent's provided answer question is compared against correct answer or set of acceptable answers.
Predicate Satisfaction:
Verifying if specific information and generated outputs satisfy predefined logical conditions or predicates. This includes:
Value Existence: A required value, file, or UI element present as expected.
Value Non-Existence: specified value, file, or UI element is correctly absent.
Range Check: A numerical output or parameter falls within predefined acceptable range (often with specified tolerance).
Correct Task Failure (FAIL): The agent correctly identifies task as infeasible terminates
appropriately when unable to complete the objective, outputting a designated F FAIL signal.
Domain-Speciffe Success Markers: For certain domains, unique success criteria are employed:
- Lean Tasks: Successful compilation the generated Lean proof code is considered a primary indicator.
18

---

Published as a conference paper at ICLR 2026
B.3 SELECTION AND MODIFICATION OF SCIENTIFIC SOFTWARE
To ensure both technical feasibility and representative task diversity, we selected software tools based on the following criteria:
Accessibility. The software must be open-source or freely available, allowing transparent integra- tion and reproducibility of experiments.
2. GUI Compatibility. The software must expose usable accessibility tree (ally tree) to support fine-grained GUI grounding and interaction.
Domain Representativeness. The software should be representative of key scientific and technical domains, enabling meaningful assessment of multimodal agent capabilities across different types of tasks.
Based on these principles, we selected the following software for each target domain:
Lean, A functional programming language and interactive theorem prover grounded in dependent type theory (specifically Martin-Löf Type Theory). Lean enables formal verification mathe- matical theorems and software correctness through rigorous type checking and logical inference, supporting robust development of maintainable and accurate code.
ChimeraX. A next-generation molecular visualization software developed by UCSF, designed for detailed interactive exploration, visualization, and analysis of protein and biomolecular structures. ChimeraX enhances performance and user experience compared to its predecessor, UCSF Chimera, offering improved graphics rendering, extensibility via plugins, and streamlined workflows for structural biology research.
KAlgebra. An educational calculator and graphical plotting application within the KDE Education Project. It supports wide range of numerical, logical, symbolic, and analytical computations, enabling users to visualize mathematical functions interactively in both two-dimensional (2D) and three-dimensional (3D) environments, thus effectively bridging computational mathematics and educational usability.
Celestia. A cross-platform, interactive real-time 3D astronomical simulation software that allows users explore the universe through detailed, dynamic visualizations. Celestia is highly extensible via scripting, empowering educational and professional users to model and visualize celestial phenomena and space missions with precision and customization.
GrassGIS. An advanced Geographic Information System (GIS) supporting both raster and vector geospatial data, along with powerful analytical capabilities for spatial modeling, hydrological analysis, and environmental simulations. GrassGIS includes comprehensive Python API for automation and custom analysis, enabling complex geospatial and temporal analyses tailored to diverse research and application scenarios.
TeXstudio. An integrated ыЧTК editor that provides writing environment tailored specifically for creating and managing complex technical and scientific documents. TeXstudio enhances productivity through features such syntax highlighting, real-time document preview, automatic reference checking, and intuitive assistance tools, greatly simplifying the process of technical writing and document preparation.
B.4 DETAILS OF ACTION SPACE
The action space employed SCIENCEBOARD is shown in Table 5. We combine standard interaction primitives (such as GUI operations) with the flexibility of system-level and application-specife Command- Interfaces (CLIs), and has been further expanded with several augmented actions tailored for scientific workflows.
B.5 DETAILS OF OBSERVATION SPACE
We primarily adhere well-established settings (Xie al., 2024; Zhou et al., 2024) for observation space, encompassing: (1) Screenshots, which consist of a full desktop screenshot as observed by human users; (2) al a ree, structured text-only representation without visual information,
19

---

Published as a conference paper at ICLR 2026
Table 5: Action space of SCIENCEBOARD environment.
Action
Description
noveTo (x, y)
Moves the mouse to the target coordinate.
noveRel (x, y)
Moves mouse by an offsct from current position.
(x, y)
Drags mouse to the target coordinate.
dragRel (x, Y)
Drags the mouse by offset from current position.
click (x, y)
Clicks at the largel coordinule.
rightClick (x, y)
Performs a right click at the target coordinate.
micaleclick tx, y)
Performs middle click at the target coordinate. Performs double clicks the target coordinate.
doubleClick 1x, y)
tripleclick (x, nouseDown (x, button)
Performs triple clicks the target coordinate. Presses a mouse button down.
nouseUp (x, Y, button)
Releuses mouse button up.
DONE
Agent decides the task is finished.
FAIL
Agent decides the task infeasible.
WAIT [n]
Agent decides it should wait, "n' defaults 5(5).
ANS [s]
Agent decides it should submit answer, 's' denotes the answer.
API [name, arqs]
Invokes registered API call with name and arguments.
CODE
Run generated code script (for in-app system-level tasks, custom functions).
applicable for agents that take pure text input; (3) Screenshots + allytree, a hybrid approach that combines and complements both textual and visual modalities; and (4) Set-of-Marks (Yang et al., 2023), a visual prompting method aimed at enhancing the visual grounding capabilities by partitioning an image into marked regions. Details are as follows:
Screenshot. We capture screenshot of the entire computer screen. For screen resolution, set a default value of 1920x1080, and it also offers 16:9 aspect ratio. Following OSWorld (Xie al., 2024), our environment also supports modifying the resolution of virtual machines to avoid potential memorization of absolute pixel values and to assist studies on topics like generalization across different resolutions.
Allytree. An allytree refers an intricate structure generated by the browser OS accessi- bility APIs that renders a representative model of the content, providing a means of interaction for assistive technologies. Each node within the accessibility tree hosts important information about UI element. In SCIENCEBOARD, which utilizes an Ubuntu-based GNOME desktop environment, we employ the Assistive Technology Service Provider Interface 2, Specifically, we adopt pyatspi to programmatically retrieve the accessibility tree on Ubuntu.
To make complex al cllytree tractable, and critically, to ensure they fit within the context length of open-source models, we filter out non-essential elements. This filtering is performed based on element attributes such as their tag, visibility, and availability. For the elements that remain after filtering, only key information-specce subsequently concatenated to form the input representation for the agent.
their tag, name, text, position, and size is retained and
Screenshot allytree. To further enhance the action execution capabilities of computer-using agents, especially for models with weaker grounding abilities, we utilize a combined input of screenshots and allytree.
Set-of-Mark. We follow the official implementation of Set-of-Mark (Yang et al., 2023). We leverage the information from the filtered allytree and mark the elements on the screenshot with numbered bounding box. Following VisualWebArena (Koh et al., 2024) and UFO (Zhang et al., 2024), we further combine the annotated screenshot with the text metadata from a lytre..
htttps://docs..kk.rrr//////
20

---

Published as a conference paper at ICLR 2026
C ACCESSING SCIENCEBOARD ENVIRONMENT
To facilitate broader adoption and reproducibility, provide several methods for accessing SCI- ENCEBOARD environment. Researchers can choose the most suitable option based on their technical requirements and resources:
Direct Deployment. The entire framework, including all scientific software and evaluation scripts, available for direct deployment on native Ubuntu system. Full setup instructions and dependency lists are provided our repository.
Docker Container. We also provide a Docker image that encapsulates the environment, making easy to run SCiENCEBOARD across different machines and operating systems, which is available at http://aaooyyooos..pppocccccccco
Cloud Platforms. For scalability and powerful computational resources, SCIENCEBOARD can be deployed on cloud platforms like Amazon Web Services (AWS). We will provide guidelines upon acceptance.
D DETAILS OF SCIENCEBOARD BENCHMARK
D.1 TASK ANNOTATION
During the task annotation process, we primarily utilize the tutorials and handbooks listed in Table to guide annotators in exploring the relevant domain and corresponding software and tools. All app data collection and task creation are completed by the authors.
D.2 TASK DIVERSITY
To explore the diversity of tasks in SCIENCEBOARD, we perform t-SNE (van der Maaten & Hinton, 2008) visualization, as shown in Figure 6. We obtain embeddings for all task instructions using text embedinngd 3-small and then apply t-SNE to reduce their dimensionality to two for visualization. The semantic distribution of instructions clearly distinguishes tasks across different domains, while also revealing considerable diversity within each individual domain. Furthermore, we can observe some intersections between Scientific Documentation tasks and tasks from other domains, which reflects the presence of cross-application workflows in benchmark.
Alpobra
Alpobra
Alpobra
Astranony
Astrocany
10-
Dimunsion 2
Dimotston
Dimon Ston
-10
-15
-10
10
-15
-20
-15
-10
Dimensson
Dimension
Dimension
Figure 6: t-SNE visualization of task instructions distribution. The seeds of t-SNE are randomly sampled for each plot.
D.3 COMPARISON WITH EXISTING BENCHMARKS
We compare SCIENCEBOARD with existing well-established benchmarks for scientific tasks, as shown in Table 7.
21

---

Published as a conference paper at ICLR 2026
Table 6: Sources of the tutorials and handbooks employed in the task annotation process.
Software
Tutorial & Handbook Sources
kde. kalgebra//ndexx.ttmll
Kalgebra
htmll
ChimeraX
httee://leac.laan.cor///hheeeepp 1ean4/
httpe///eeanppereecccmmntttttttp matranntiloo.hhlleen//eeee..aiin
Lean
http:///baarnaaggger///////ttttt html
Grass GIS
htte:://ellsstapppppeersppppppep
Celestia
httess///elletttooooprerrssppees
orteepppp..cciica////a.aoaaaaaae html
TeXStudio
http://latee-uuoiial.ccouuuuotot
22

---

Published as a conference paper at ICLR 2026
Featurv
SCIENCEBOARD tour work)
SeienceQAA (La al., 2022) SeiCode (Tian al, 2024)
Stience StienceAAentBeecc (Chen al,, 2025)
I/0 Formory
Input Visual Information
x
Question-Anse Answering
Scientific Computing
X
GUI Automation
Table A comparison of SCIENCEBOARD to notable and recent Al4Science benchmarks.
SCIENCEBOARD is the first to offer a realistic environment for evaluating scientific tasks. In terms of V/O, incorporates structured code input and visual information, which are critical for simulating scientific experiment workflows. It also supports GUI automation, making it well-suited for visual agents fulfill tasks like humans do. Additionally, SCIENCEBOARD covers broader range of task types compared to existing works, including but not limited to question-answering and scientific computing. These unique features make SCIeNCEBOARD both versatile playground and an expandable framework for evaluating agents' scientific capabilities.
D.4
MORE EVALUATION SCRIPT EXAMPLES
Beyond the evaluation cases listed Section 3.2, Table showcases a broader variety of evaluation pipelines created using our templates.
D.5 HUMAN PERFORMANCE
In our main experiments, as reflected Table we recruit college-level students to establish normal human performance SCIeNCEBOaRD benchmark. Before attempting the tasks, participants required to familiarize themselves with foundational knowledge the relevant scientific disciplines and study the provided operational manuals. They were then given instructions, shown in Instruc- tion 1, to complete the assigned tasks. Participants were compensated at rate of $10 per hour for their involvement.
The SCIENCEBOARD enviromment and scientific software used do not record any personal infor- mation, and all participants provide informed consent. The experiment does not involve surveys, interviews, or any behavioral tracking.
D.6 STABILITY ANALYSIS
Considering that dynamic environments could potentially lead to experimental instability, we conduct an additional set of experiments focusing on consistency. For these, we utilize GPT-4o under the allytree + screenshot setting, with results and error bars reported in Figure
35
Success Rate (%6
30
15
10
Algebra
Biochem
Figure 7: Stability analysis.
Across three independent runs, performance on Algebra tasks remains stable. However, Biochemistry tasks exhibited minor fluctuations in success rates. Upon closer inspection of individual cases, hypothesize that these variations likely stem from network connectivity issues or transient system lag encountered during task execution.
23

---

Published as a conference paper at ICLR 2026
Table 8: More evaluation cases SCIENCEBOARD include exact matching, range-based assessment, and numerical tasks with tolerance.
Initial State
Instruction
Evaluation Script (Simplified)
Select all ligand(s) and color them mugeviu ChimeraX.
"eyso": "incon,
"key": "sel",
"value": ("aton id /A:98N2 idatn_tyse N3+*,
"eype": "inzo",
"key": "rescoloz /A",
Moalue" (*#1/As1 color #d2b48c*,
There puint located the Mediterranean
"tyae": "db"
"and"; "v.to.db",
delete
"Avarqa":
*clage":
"map":
"typet; "point",
"aption": "coor"
"Key": "lambda out: out, out.otrip()",
"valus": (x/y/AAn. 3499478811774
"prad": *Lambda key, value: key =- vales"
Approach to the Earth and display
"tyge": "info",
solar ectipse Celentia
"key": "lanida (..E"BarLhi" distaann
"salue": 0,
"prod": "Lambda X, V. abotk 450003*
"syso": "into",
"kay"s "lanida ... (
talse
3.1
"syge": "into",
"Koy": "lanbda ... ["Moor"] [""oo"""""""ss"o ["visibier)""
"value": true
"Lyow": "info",
"Kay": "lanbda ...",
"value":
"pred": *Lambda key, value: key value"
theorem IP_3
(TopologicalSpace
"tyge": "placeholder"
[TopologicalSpace Y]
12 : Set X) thi Cont IsConnecced
IsConnested ly
yJ
D.7 EVALUATION COST
We use API keys access proprietary models. On average, single run on all SCIENCE- BOARD tasks costs $64 using GPT-4o, $86 using Claude-3, Seaude-3.7-Sonnes 7-Sonnet, and $45 using Gemini-2.0-Flee.
E DETAILS OF EXPERIMENTS
E. BACKBONE MODELS
We briefly discuss the backbones we used to build our computer- using agents.
Proprietary Models. Proprictary models now demonstrate striking capabilities in complex reason- ing and are increasingly exhibiting agentic potential for dynamic real-world interaction, prompting
24

---

Published as a conference paper at ICLR 2026
closer look their diverse forms. In the experimental section, accessed the following proprietary models via API keys:
GPT 1Pr-4o 4o (Hurst et al., 2024).
1.aude-3.7-Sonme (Anthropic AI, 2024).
Gemini-2. (Gemini Team, 2024).
03-mini (OpenAI, 2025a).
Open-source models are demonstrating remarkable advancements, steadily narrowing the performance gap with proprietary models. Crucially, the open-source community recognized the significance of agentic capabilities early on, fostering development in this direction. This foresight has translated into exceptional performance, particularly within GUI scenarios where these models now excel on various challenging benchmarks. Our evaluation is based on the following open-source models, which characterized by their advanced grounding capabilities:
Open-source Models.
Qwen2.5-VLL VL-72B- I-Istrutt (Bai et al., 2025): The latest evolution in the Qwen dision-langual model family, primarily distinguished by its robust agentic capabilities. It operates directly as a visual agent, proficient in reasoning, dynamically utilizing tools, and executing tasks for computer and phone operation. Complementing its agentic prowess, U.ve2..5VVI-72.. Innstrutt demonstrates advanced proficiency in detailed visual amalysis (including texts, charts, icons, and layouts within images), comprehension of videos exceeding one hour with event pinpointing, precise object localization with structured coordinate output, and the generation of structured data from documents such as invoices and forms. our experiments, this model is deployed using interconnected clusters of A100 80GB GPUs with vLLM (Kwon et al., 2023).
Intern InternVL3-78B VL3-78B (Chen et al., 2024): An advanced MLLM recognized for its superior overall perfor- mance and significantly enhanced multimodal perception and reasoning. key advancement its robust agentic functionality, demonstrated through proficient tool usage and GUI agent operations, alongside extended capabilities in areas like industrial image analysis and 3D vision perception. These comprehensive abilities underpinned by innovations such training approach, supervised fine-tuning with diverse, high-quality data tailored to these advanced tasks, and mixed preference optimization for refined reasoning. In our experiments, this model is deployed using interconnected clusters of 8 x A100 80GB GPUs with vLLM.
native multimodal pre-
E-Q-72B-Prrevion (Qwen Team, 2024): An experimental research model focused on advancing visual reasoning capabilities. It has achieved compelling performance in complex multidisciplinary understanding and problem-solving highlighting its specialized strength in sophisticated visual cognitive tasks. However, exhibits some limitations in instruction following, appearing less adept in agent scenarios that require precise action outputs. In our experiments, this model is deployed using interconnected clusters of 8 A100 80GB GPUs with vLLMM
GUI Action Models. While foundational models provide impressive general-purge intelligence, their intrinsic agentic capabilities for nuanced GUI manipulation are still under active exploration, often requiring further specialization. Consequently, prominent line of research involves adapting open-source VLMs by fine-tuning them on extensive, GUI-specific datasets. This targeted training methodology yields dedicated action models equipped with significantly enhanced proficiencies for understanding and interacting with GUIs. The GUI action models adopted in this paper are as follows:
OS-Atlas-Pro- (Wu al., 2025b): A foundational GUI action model that significantly advances open-source VLMs for agentic tasks, excelling in GUI grounding and out-of-diittriuutin scenarios through innovations in modeling and the creation of the largest open-source, pross-platform GUI grounding corpus with over 13 million elements. It demonstrates state-of-the-an performance across six diverse benchmarks (mobile, desktop, web) and verifies the existence of model scaling laws in GUI scenarios. In our experiments, this model is deployed using a single A100 80GB GPU with vLLM (Kwon al., 2023).
UGround- V1-7B (Gou et al., 2025): A universal visual grounding model that identifies GUI action elements by pixel coordinates. It powers the SeeAct-V V framework (Zheng et al., 2024), which enables purely visual GUI perception and pixel-level operations. Agents using SeeAct-V V with
25

---

Published as a conference paper at ICLR 2026
UGround have achieved SOTA results across five distinct benchmarks spanning web, mobile, and desktop evaluations. In our experiments, this model is deployed on single A100 80GB GPU with vLLM.
-TTAR--72B-DPO (Qin et al.. 2025): An end-to-end native GUI agent that uniquely perceives screenshots its sole input perform human-like keyboard and mouse interactions, outperforming prevailing agent frameworks that depend on heavily wrapped commercial models with expert- crafted prompts. has established state-of-the-ar performance across more than ten GUI agent benchmarks. This advanced capability stems from key innovations including enhanced perception, unified action modeling. System-2 reasoning, iterative training with reflective online traces, and a final Direct Preference Optimization (DPO) phase, which refines its ability to make precise, context-aware decisions. In our experiments, UI-TARS-72B-DE utilizes vLLM for inference and is deployed on interconnected clusters of A100 80GB GPUs.
GUT-Actor-7B (Wu et al., 2025a): A recently proposed GUI grounding model that introduces a novel coordinate-fres visual grounding approach. utilizes an action head to direct the special token <ACTOR> to the target screenshot patches for localization. It claims to surpass the text-based coordinate prediction baseline and demonstrates better generalization in out-of-distribution (OOD) scenarios. In our experiments, we used the 7B version of GUI-Actor based on the Dwen2.5-VL. backbone.
E.2 EVALUATION SETTINGS MAIN EXPERIMENTS
We adhered to common prompt engineering strategies from previous works (Sun et al., 2024b; Zhou et al., 2024; Zhang & Zhang, 2024) for the agents under evaluation. For each domain, the agent interacts with the environment under the guidance of a meta-prompt, which includes information about the software being operated, executable special actions, and related details. When taking actions, the agent generates outputs in the ReAct style (Yao et al., 2023), with its step-by-step thoughts recorded in the interaction history.
Throughout the evaluation, set the temperature parameter to 0.5, top_E 0.9, and max__ fex_tokens tokens to 1500. We list some prompt examples Prompt 14, Prompt 15, Prompt 16 and Prompt 17.
E.3 EVALUATION SETTINGS ANALYSIS
In experiments with interleaved planning and action, we first address inconsistencies coordinate outputs from different GUI action models. While InternVLL3-788 (Chen et al., 2024) outputs coordinates on a [0, 1] scale, models such as OS-Atlas, UI-TARS, and UGround use [0, 1000] scale. To ensure uniformity, we normalized all coordinate outputs to a [0, 1] scale prior to execution.
This part of the experiments employs a two-stage process: First, the planner model receives the current observation (obs) and task instruction to generate high-level plan or a specific action. If the planner outputted directly executable primitive action (e.g.. non-GUI system-level command or special control token like DONE), that action will be performed immediately, and the action model was not invoked for that step. Otherwise, the grounding model received the current observation and the plan (or sub-task) from the planner. Its role to output low-level executable instructions. If actions directly, these commands were exccuted. For
the grounding model generate pyaut models outputting in their specific native formats, implement custom parsers translate these into pyautoqui actions: for UGround and UI-TARS, all coordinate- based outputs were interpreted click, whereas for OS-Atlas, its outputs parsed to differentiate between click, type, and scroll 1 based on its defined schema.
We list prompt examples in Prompt 18, Prompt 19, Prompt 20 and Prompt 21.
26

---

Published as a conference paper at ICLR 2026
F EXTENDED ANALYSIS
F.1 INTERFACES
In Section 6, analyze the performance difference between Vision-Only and Hybrid Interface settings under the a 1 ree + screenshot. Here, we present empirical results under the other three observation settings.
allytree
Screenshol
SuM
30
GUT CLI
Succoss Rate (%)
25-
GUT Only
20
15-
10-
5-
4o
Qwea2.5.VL
InternVL3
GPT4o
Qwen2.5 2.en2.5VVL. VLL
InternV13
G7T4o
Owen2.5
InterVL33
Figure 8: Extended analysis of Vision-Only vs. Hybrid Interface.
As shown in Figure 8, the hybrid GUI CLI setting consistently achieves performance that is comparable to or better than the GUI-Only setting across scenarios. Interestingly, while Go achieves state-of-the-art performance under other observation settings, it exhibits very weak action capabilities when using screenshot setting, indicating the reliance on structured observations for effective reasoning and planning.
F.2 INTERACTIVE ENVIRONMENTS
ATP represents one of the most logic-intensive tasks for agents and has been traditionally studied in textual settings in prior works (e.g., plain text or bash terminal).
Textual
Interactive
15
Success
GFT-4o
Claude-3.5
Qwen2.5-VL
Figure 9: Textual v.s. Interactive
We extend ATP to live OS in SCIENCEBOARD and further compare agents' performance under textual and interactive settings. The latter, similar to environments commonly used by humans, provides a live VSCode interface with features such as syntax highlighting, autocompletion, type inference, and other functionalities. As shown in Figure 9, in the textual setting, the agent applies heuristic strategies (e.g., Monte Carlo search) to make predictions over the proof tree without interacting with the environment. In contrast, in the interactive setting, the agent must autonomously decide which PRoOFSTATE to proceed with. Moreover, the agent is also required to localize the relevant code segments within the interface. Completing formal methods tasks becomes substantially more challenging in realistic environments, which significantly increases the cognitive complexity.
27

---

Published as a conference paper at ICLR 2026
F.3 DIFFICULTY ANALYSIS
We further analyze the success rates of computer-using agents on the SCIENCEBOARD benchmark across different task difficulty levels. We employ Claude-3 7-Sonnet, GPT-4o, and Qwen2.5- VL, with results presented in Figure 10.
Claude-3.7 3.7-Sonnet
GPT-4o
Owen2.5-V1.-7B
100
Number of Problems
Streembot+
Screenshot
Screenshot+ lytree
80
Screenshot
60
40
20
0-
Easy
Medium
Hard
Easy
Medium
Hard
Easy
Medium
Figure 10: Comparative analysis of task difficulty solve rates.
The findings indicate that solvable tasks are primarily concentrated among a subset of "Easy" problems and few "Medium" tasks. All "hard" tasks, which involve complex computations, cross-application workflows, or long-horizon planning, could not be completed by any of the evaluated agents.
F.4 COMPUTE SCALING
During the evaluation of frontier models, a natural question arises: can scaling inference-tinge compute (test-time scaling) resolve the performance bottlenecks observed in complex scientific workflows? To investigate whether extended generation limits and native reasoning modes enable agents to better navigate these dynamic environments, we conducted an extended analysis focusing on three domains.
Table 9: Success rates of varying max_tokent under the Screenshot observation setting.
max_ max_tokens
Algebra Biochemistry
GIS
1500
41.90%
62.10%
11.80%
2000
41.90%
58.62%
11.80%
2500
41.90%
65.52%
14.70%
Further, we evaluated the performance of GPT-5 utilizing its native reasoning mode with varying levels test-time reasoning effort (medium um and high). As detailed Table 10, elevating the reasoning effort improves performance across several settings. Notably, under the Screenshot + allytree setting, shifting from medium high reasoning effort slightly increases the success rate in Biochemistry from 68.96% to 72.41%, ,, and in GIS from 14.70% to 17.64%.
Table 10: Performance of GPT-5 under different native reasoning efforts across two observation settings.
Observation
Reasoning Effort
Algebra
Biochemistry
GIS
Medium
41.90%
65.52%
14.70%
Screenshot
High
45.16%
65.52%
14.70%
Medium
48.39%
68.96%
14.70%
Screenshot + allytree
High
51.61% %
72.41%
17.64%
These results empirically demonstrate that while scaling test-time compute and leveraging native rea- soning models provide measurable benefits, they do not fundamentally overcome the core challenges of our benchmark. The bottlenecks likely stem from the agent's 's general agentic capabilities, i.e., the ability to accurately perceive dense, Nomain--specife UI elements and translate high-level scientific plans into precise, executable actions, rather than solely deficiency in deep cognitive reasoning.
28

---

Published as a conference paper at ICLR 2026
F.5 FAILURE ANALYSIS
To further investigate the reasons why computer-using agents fail when planning taking actions on scientific tasks, here we include and discuss several typical examples of such errors.
Opening the Wrong File. This error is frequently caused by grounding issues. The agent initially clicks on an incorrect file and then attempts to perform subsequent actions, such as inputting data, within that wrong file. This often leads to the agent repeatedly making the same mistake getting stuck in an unproductive loop. typical case shown in Figure 11.
Figure 11: Use wrong file.
Inability Invoke the Correct Function. In some instances, agents need to identify and use specific function within a software application but attempt to do so by directly typing assumed function name into a search bar or command input. If the exact function name is unknown or guessed incorrectly, a more robust strategy would be to browse available menus or function lists. Instead, agents may incorrectly assume knowledge of the function name and attempt to look up its usage, leading to failure. A typical example of this behavior is presented in Figure 12.
Figure 12: Function invocation error.
Incorrect CLI Code. Failures also occur when agents formulate CLI commands incorrectly. This can involve syntax errors, wrong command names, or incorrect parameters. Notably, some of these failed CLI attempts, the intended task could have been accomplished more straightforwardly by interacting with a corresponding button or element in the GUI. typical example shown in Figure 13.
29

---

Published as a conference paper at ICLR 2026
Figure 13: CLI code error.
G PROMPTS
The prompt examples we used in SCIENCEBOARD are listed below.
30

---

Published as a conference paper at ICLR 2026
Agentic Prompt ChimeraX with screenshot
You are an agent which follow my instruction and perform desktop computer tasks Instructed.
You have good knowledge of Chimerax, molecular visualization software; and assume your code will rur on a computer controlling the mouse and keyboard.
For each step, you will get observation of the desktop by accessibility tree, which based on AI-SPI library, and you will predict actions of the next step based on that.
You required to observation but NOT use the 'pyautogui laateeenterrnneeenn function to locate the element you want to operate with since we have no image of the element you want to operate with. DO NOT USE prautogui.screensee () to make screenshot.
'pyautegui'
perform the action grounded to the
You ONLY need return the code inside code block, like this:
your code here
Return line or multiple 1ines of python code perform the action each time, and be time efficient. When predicting multiple lines of code, make some small sleep like (llee.aeep machine could take breaks. Rach time you need to predict a complete code, and no variables or function can be shared from history.
interval so that the
also allowed to return the following special code: when you think the task is done, return " "'DONE";; when you think the task can net be done, return "'FAIL", Don't easily say "'FAIL""; try your best to do the task; When you think you have wait for n"', in which n defaults to (s);
Specially,
time, return "WWAIT"A "WAIT
When you are asked to submit an answer, return "'ANS s" without and use 'FAIL' if there is no answer to quotation marks surrounding the question.
My computer's password is 'password', feel free to use it when you need sudo rights.
DC NOT introduce any unrelated models easily close existing models, otherwise the task might be evaluated FAILED.
DO NOT close the current ChimeraX session, or every effort you made will be in vain.
NEVER try to zeopen the command line interface in Chimerax if it is hidden, because it has been deactivated and cannot do anything. But you are welcone to use once it presented.
First give the current observation and previous things we did a short reflection, then RETURN ME THE CODE OR SPECIAL CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE. You are asked complete the following task: Fetch 201X from PDB in ChimeraX.
Prompt 14: Prompts for ChimeraX with screenshot
31

---

Published as a conference paper at ICLR 2026
Agentic Prompt Celestia with screenshot
You are an agent which follew my instruction and perform desktop computer tasks Instructed.
You have good knowledge of Celestia, three-dimension space simulator; and assume your code will run on computer controlling the mouse and keyboard.
For each step, you will get an observation of the desktop by screenshot, and you will predict actions of the next step based on that.
You are required to use 'pyautogui' to perform the action grounded to the observation, but DO NOT element you want to operate with since we have no image of the element you want to operate with. DO NOT USE pauttgui.ssreenssie () to make screenshot.
the 'pyautogui, Coateeettrrtnnnneeeeenn function
to locate
You ONLY need to return the code inside code block, like this:
your code here
Return one line or multiple lines of python code to perform the action each time, and be time efficient. When predicting muitiple 1ines of code, make some small sleep like 'time.aleep (0. ; interval that the machine could take breaks. Each time you need to predict a complete code, and no variables or function can be shared from history.
Specially, it is also allowed return the following special code: When you think the task is done, return " "'DONE";; when you think the task can not be done, return "'EAIL"', Don't easily say "'FAIL";; try your bost to do the task; when you think you have to walt for some time, return "'WAIT"' "WAII n"', in which defaults When you are asked to submit an answer, return "AANS s" without quotation narks aurrounding and use 'FAIL" if there is no answer to the question.
(s);
My computer's password is 'password', feel free to use when you need sudo rights.
The criterion for celestial bedy to be displayed on the screen is that the object's center is within the window range and is not blocked by others.
First give the current observation and previous things we did a short reflection, then RETURN ME THE CODE OR SPECIAL CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE. You are asked complete the following task: Set the Julian date 240000C in Celestia.
Prompt 15: Prompts for Celestia with screenshot
32

---

Published as a conference paper at ICLR 2026
Agentic Prompt ChimeraX with set-of-marks
You are an agent which follow my instruction and perform desktop computer tasks as instructed.
You have good knowledge of ChimeraX, molecular visualization software; and assune your code will keyboard.
computer controlling the mouse and
For each step, you will get an observation of the desktop by 1) an accessibility tree, which is based on AT-SPI library; and 2) screenshot with interact-able elements marked with numerical tags, and you will predict actions of the next step based on that.
You are required to use 'pyautogui' to perform the action grounded to the observation, but DO NOT use the 'pyautogui. LoateeenterrnScren function locate the element you want to operate with since have image of the element you want to operate with. DO NOT USE "pyautonu..screenhhoo to make screenshot.
screenshot
You ONLY need to return the code inside a code block, like this:
your code here "
Return one line or multiple lines of python code to perform the action each time, and be time efficient. When predicting muitiple 1ines of code, make some small sleep like 'time. (0.ep((.5) machine could take breaks. Each time you need to predict complete code, and no variables or function can be shared from history.
interval that the
You can replace operate with, such as:
in the code with the tag of elements you want to
pyautogui.movcToo (tag_3) Pyautogus.click pyautogui. (tag_l, button='left')
When you think you can directly cutput precise x and y coordinates or there is no tag on which you want to interact, you can alse use them directly; but should be careful to ensure the correct of coordinates.
also allowed to return the following special code: when you think the task is done, return " "'DOE""; when you think the task can net be done, return "'FAIL" Don't easily say "'FAIL";; try your best to do the task; When you think you have wait for n"', in which n defaults to (s);
Specially,
time, return "'WAIT"A "WAAIT
When you are asked to submit an answer, return "'ANS s" without quotation marks surrounding and use 'FAIL' if there is no answer to the question.
My computer's password is 'password', feel free to use it when you need sudo rights.
DC NOT introduce any unrelated models easily close existing models, otherwise the task might be evaluated as FAILED.
DO NOT close the current ChimeraX session, or every effort you made will be in vain.
NEVER try to zeopen the command line interface in ChimeraX if it is hidden, because it has been deactivated and cannot do anything. But you are wolcone to use once it presented.
First give the current observation and previous things we did a short reflection, then RETURN ME THE CODE OR SPECIAL CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE. You are asked complete the following task: Fetch 201X from PDB in ChimeraX.
Prompt 16: Prompts for ChimeraX with Set-of-Marks
33

---

Published as a conference paper at ICLR 2026
Agentic Prompt Celestia with set-of-marks
You are an agent which follow my instruction and perform desktop computer tasks as instructed.
You have good knowledge of Celestia, three-dimension space simulator; and assume your code will rur on computer controlling the mouse and keyboard.
For each step, you will get an observation of the desktop by 1) an accessibility trec, which is based on AT-SPI library; and 2) a screenshot with interact-able elements marked with numerical tags, and you will predict actions of the next step based on that.
You are required to use 'pyautegui' to perform the action grounded the function
observation, but DO NOT use the to locate the element you want to operate with since we have no image of the element you want to operate with. DO NOT USE "pyautogui to make screenshot.
You ONLY need to return the code inside a code block, like this:
your code here
Return one line or multiple lines of python code to perform the action each time, and be time efficient. When predicting multiple lines of code, make some small sleep like 'time.sleen ; interval so that the machine could take breaks. Each time you need to predict complete code, and variables function
shared from history.
You can replace operate with, such as:
in the code with the tag of elements you want to
pyautogui.nover noveTo (tag 3) syautogui.clln click (tag_2)
cetttoguiddragTin (tag_l, button=/left!
When you think you can directly cutput precise and y coordinates there no tag on which you want to interact, you can also use them directly; but you should be careful to ensure the correct of coordinates.
Specially, it is also allowed return the following special code: When you think the task is done, return " "'DONE"'; When you think the task can not be done, return "MAAII. Don't easily say ""FAlL"M try your best to do the task; when you think you have to wait for some time, return "WAITHW "WAIT n"', in which defaults to (s))
When you are asked to submit an answer, return "ANSS s without
quotation narks surrounding and use 'FAIL' if there is no answer to the question.
My computer's password is 'password', feel free use when you need sudo rights.
The criterion for celestial bedy to be displayed on the screen is that the object's center is within the window range and is not blacked by others.
First give the current observation and previous things we did a short reflection, then RETURN ME THE CODE OR SPECIAL CODK ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE. You are asked complete the following task: the Julian date 2400000 in Celestia.
Prompt 17: Prompts for Celestia with Mer-of-Marks
34

---

Published as a conference paper at ICLR 2026
Human Instructions
You are required to finish the given tasks manually to provide sanple data of human accuracy.
First, please start up the evaluation script with debug option ON and headless option OFF, Then, wait for the environment to be initialized and perforn your actions when you receive corresponcing logs from stdout. Press ENTER after you finish operating and the script will evaluate your result subni tted automatically; Attention:
1. If you need to finish the task with primiti vis other than TIMBOU",
please input directly into stdin;
2. You can search for documents or manuals if you encounter
domsin-specificc knowledge you are not familiar with;
3. Make sure that the number of your steps is less than expected. To be more precise, a popup without possibility to predict its position should be split into different steps.
Instruction I: Instruction for humans.
Agentic Prompt OS-Àtlas
You agent which follow my instruction and perform desktop computer tasks as instructed.
You have good knowledge of Celestia, chree-dimension space simulator; and assune your code will run on a computor controlling the mouse and keyboard.
For each step, you will get an observation of the desktop by sereenshot, together with plan generated by the planner, and you will parse the plan to operate actions of next steps based on that.
You required to your grounding ability to perform the action grounded to the observation and the plan.
You need to return a basic action together with arguments, of which the available ones are listed below:
CLICK; to click at the specified position.
format: CLICK pooint>[[x-xxxs) y-axisll y-axisl1</point>
example usage: CLICK spoint>[[100 8721 | </point>
TYPE: to enter specified text at the designated location.
format:: TYPE [input text]
-example usage: TYPE (Shanghai shopping mall]
SCROLL: to scroll the specified direction.
format: SCROLL Idirection (UP (B//DOM//EE://AGHT]
example usage: SCROLL (UP)
My computer's password 'password', feel free use it when you need sudo rights.
Some plans provided may contains unexpected code blocks or confusing instructions. Be flexible and adaptable according to changing
cireumstances.
First give the current observation and the generated plan, then RETURN ME THE CODE I ASKED FOR, NEVER EVER RETURN ME ANYTHING ELSE. You are asked to complete the following task: Set the Julian date to 2400000 in Celestia.
Prompt 18: Prompts for OS-Atlas
35

---

Published as a conference paper at ICLR 2026
Agentic Prompt UGround
You are an agent which follow my instruction and perform desktop computer tasks as instructed.
You have good knowledge of Celestia, three-dimension space simulator; and assume your code will rur on computer controlling the mouse and keyboard.
For each step, you will get an observation of the desktep by screenshot, together with a plan generated by the planner, and you will parse the plan to operate actions of next steps based on that.
use your grounding ability to perform the action grounded to the observation and the plan. You need to return a 26 coordinate (x, y) indicating the position you want to click.
You are required
My computer's password is 'password', Seel free to use it when you need sudo rights.
Some plans provided may contains unexpected code blocks or confusing instructions. Be flexible and adaptable according to changing circumstances.
First give the current observation and the generated plan, then RETURN ME THE CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE. You are asked complete the following task: Set the Julian date to 2400000 in Celestia.
Prompt 19; Prompts for UGround
36

---

Published as a conference paper at ICLR 2026
Agentic Prompt Qwen
You are an agent which follow my instruction and perform desktop computer tasks as instructed.
You have geod knowledge of Celestia, three-dimension space similator; and assune your code will keyboard.
computer controlling the and
For each step, you will get an observation of the desktop by a screenshot, together with plan generated by the planner, and you will parse the plan to operate actions of next steps based that.
You are required to use 'pyautoqui' to perform the action grounded to the observation and the plan, but DO NOT use the 'pyautogui. LocateCeneerrnnSernnn function to locate the element you want to operate with since have image of the element you want operate with. DO NOT USE You ONLY need to return the code inside a code block, like this:
wcw.eeguhsscreeshhtttoooooooooo. () to make screenshot
your code here
Return one line or multiple lines of python code to perform the action each time, and be time efficient. when predicting multiple 1ines of code, make some small sieep like machine could take breaks. Each time you need to preaict complete code, and no variables or function can shared from history.
interval so that the
Specially, it is also allowed to return the following special code: when you think the task is done, return "'DONE";; When you think the task say "'FAIL"" try your best to do the task; When you think you have to wait for some time, zeturn "WAIIT" or "'NAIT n"', in which n defaults to (s))
not be done, return "VEAIL" Don't easily
When you asked to submit answer, return "'ANS s without quotation marks surrounding s, and use 'FAIL' if there is no answer the question.
My computer's password is 'password', feel free to use it when you need sudo rights.
Some plans provided may contains unexpected code blocks or confusing instructions. Be flexible and adaptable according to changing
circumstances.
First give the current observation and the generated plan, then RETURN ME THE CODE OR SPECIAL CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE.
You are asked to complete the following task: Set the Julian date to 2400000 in Celestia.
Prompt 20: Prompts for Qwen
37

---

Published as a conference paper at ICLR 2026
Agentic Prompt UI-Tars
You are an agent which follow my instructions and performs desktop computer tasks as instructed. You have good knowledge of Celestia, three-dimension space simulator; and assume your code will rur on computer controlling the mouse and keyboard.
For each step, you will get an observation of the desktep by screenshot, together with a plan generated by the planner, and you will parse the plan to operate actions of next steps based on that.
use your grounding ability to perform the action grounded to the observation and the plan. You need to return a 26 coordinate (x, y) indicating the position you want to click.
You are required
My computer's password is 'password', Seel free to use it when you need sudo rights.
Some plans provided may contains unexpected code blocks or confusing instructions. Be flexible and adaptable according to changing circumstances.
First give the current observation and the generated plan, then RETURN ME THE CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE. You asked complete the following task: Set the Julian date to 2400000 in Celestia.
Prompt 21: Prompts for UI-TARS
38


Reviews:
[{"review_id": "Vb1LS3rPnQ", "reviewer": "Reviewer_fF65", "summary": "This work presents ScienceBoard, a multi-modal benchmark for evaluating computer-use agents in automating scientific workflows. The benchmark covers 169 expert-curated tasks from several scientific domains and focuses on the agents' abilities in using domain-specific software to complete scientific tasks. Evaluations results on several popular agents reveal the challenge of existing models to solve the tasks.", "questions": "1. Table 2, what's the unit for length? How is time consumption calculated?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work is a timely addition to the community for evaluation and development of autonomous agents for scientific workflows.\n2. Evaluation results delivers insights into the performance of existing agents, especially under different input modalities.", "weaknesses": "1. I think the tested models are outdated for this conference. Given the rapid development of the field, it will be great to include results from more recent models, such as GPT-5, Gemini-2.5, Qwen3, etc. \n2. Related to 1, the cost of evaluation is unclear from the paper, which may hinder the adoption.\n3. Several key details are missing, which are critical for assessing the quality of the benchmark:\n    1. What are the background of the experts/annotators? Since the tasks are drafted by annotators, are they ecologically valid tasks that correspond to real-world scientific workflows?\n    2. line 325, how is difficulty defined?\n    3. How is human performance measured? Intuitively, average population and professionals should perform quite differently for scientific tasks, it is important to discuss the background of participants in human study.\n4. The scope of this paper seems to be over-claimed.  While the author claims to cover \"end-to-end scientific exploration workflows\", critical research steps like literature review, idea generation, and report/paper writing are missing. The focus is mainly on execution and using scientific software.\n5. Evaluation setup is unclear. How many trials are allowed for each agent? Do you use the same context length limit for all agents? Also, as far as I know, o3-mini supports multimodal input. Why it is only evaluated in the a11ytree only setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qNN5zAdhZX", "reviewer": "Reviewer_CtCn", "summary": "This paper presents ScienceBoard, a new benchmark for testing AI agents on real scientific software. It runs in a virtual machine and includes 169 tasks across six domains (like biochemistry and astronomy) using professional tools like ChimeraX and Celestia. Agents have to use both GUI and CLI to complete tasks. The authors tested top models like GPT-4o and found they fail badly, with only a 15% success rate, showing these agents aren't ready to be \"AI co-scientists\" yet.", "questions": "Your GUI-only vs. GUI+CLI analysis is great. Can you give a clearer breakdown of the 169 tasks? How many must be solved with the GUI (like in Celestia?), and how many are hybrid, where you disabled the CLI for the test? This would help clarify the balance of skills needed.\n\nWhat are the two \"Open Problems\" mentioned in Table 2? They sound really interesting. How are they different from the \"Hard\" tasks, and how do you evaluate them?\n\nI know the pass/fail scoring is tough. Since your tasks seem to have multiple steps (from Fig 3), have you thought about a partial credit or step-based success rate? It could show more granular progress, even if it's not the main metric.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The main strength is that this is a novel and needed benchmark. Testing agents on real scientific software is a great idea. The technical work to get this running is impressive, especially the way they check the internal state of the apps to see if a task is done. The paper is clearly written. The analysis is also good, especially the part where they separate the \"planner\" (like GPT-4o) from the \"actor\" (GUI models), which gives a good hint at how to build better agents.", "weaknesses": "The paper is strong, but a few things could be better. The pass/fail scoring is a bit harsh; an agent that almost finishes a task gets the same zero as one that fails immediately, so it's hard to see partial progress. The number of tasks per domain is a bit uneven. Also, the \"scientific discovery\" framing is a slight overstatement. The tasks are more about using scientific tools correctly, not really about discovering new science, which is fine but good to be clear about. The two \"Open Problems\" sound cool, but it's a shame there are only two.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BMGABCXV4V", "reviewer": "Reviewer_cnsQ", "summary": "The paper presents \"yet another benchmark\" for evaluating current LLM agents' capabilities in navigating typical scientific workflows across different domains. These workflows encompasses interacting with multiple tools, parsing feedback, multi-modal interpretation and reasoning, coding and domain-specific knowledge. The environment provided to an agent is VM with the necessary tools and the action space is unified through GUI and CLI. The conclusion is that current LLMs in the standard ReAct framework severely under-performs human across all domains (with overall success rates of ~15% vs ~60%). One interesting empirical finding is that separating planner and action executor improves the performance.", "questions": "Check above weaknesses.\n1. I wonder if the models were run in native-reasoning mode (gpt-oss even supports different level of reasoning). \n2. Did the authors run ablations or sweeps on sampling parameters, especially max_tokens?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The paper presents a well-engineered agentic evaluation and a benchmark on realistic and complex scientific workflows across multiple domains.\n2. The experimental analysis is quite comprehensive, with ablations on the operators of ReAct and tools.", "weaknesses": "1. There are no experiments with test-time scaling infra -- these tasks are too complex to be done in one-shot with simple prompting. Current experiments do not fully and fairly demonstrate the LLM abilities to reason and interact with environment since the provided compute infra is quite limited. \n2. The sampling params like max_tokens = 1500 seem too restrictive for the frontier LLMs, also it seems that they were not run in reasoning mode. Even then, these LLMs typically would require a lot of think tokens (much more than 1.5k tokens) on these complex tasks.\n3. Several technical details are unclear. For example, how is the memory implemented -- is it simply the concatenation of observations, actions or did it involve some summarization? With more complex tasks, the prompt can get super long and naturally LLMs will run into all sort of long-context issues. Also. please provide ReAct traces for a few tasks.\n3. The presentation can be improved with better formatting on tables, so that one can easily get some insights on what models perform well on certain domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CSpy04TMZc", "reviewer": "Reviewer_N2NY", "summary": "This paper introduces SciBoard, a benchmark designed to evaluate LLM/VLM-based agents on scientific-discovery workflows in the following domains: biochemistry, algebra, theorem proving, geoinformatics, astronomy, and scientific documentation. The authors created six environments as virtual machines that can be interacted with via command lines, APIs, or the GUI. Across all environments, SciBoard regroup set of 169 tasks designed by domain experts. Each task provides specific instructions and a mean to automatically evaluate its completion. The authors empirically show how challenging SciBoard is for current LLM/VLM-based agents achieving only a 15% overall success rate.", "questions": "- Is there anything that prevents the agent from cheating given that its has access to the full machine? Could it for instance read some log files or inspect memory to get extra information that a human user wouldn't have access to?\n- In table 2, what does the two open problems correspond to exactly? How do you evaluate for this?\n- In table 2, in the execution section, what does steps and time correspond to exactly? Are those obtained from human trajectories? Are those human expert or non-expert?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- A lot of work went into designing and implementing the six environments for SciBoard. Then, designing the tasks within these environments required careful consideration of the unique challenges and diversity present in each scientific domain.\n- The authors show the impact of different type of observations (e.g., images, text, images+text). This provides useful insights into how different modalities contribute to agent capabilities in those particular scientific workflows.", "weaknesses": "- I don't see a clear mention of train/validation/test splits for the tasks in SciBoard. Are all 169 tasks used for evaluation only? If so, how do you envision researchers using SciBoard for training their agents? You can bet people will want to use SciBoard for RL fine-tuning their agents on it as well.\n- For figure 5, it is mentioned that \"GPT-4o and InternVL3 suffer clear drops in performance, whereas Qwen2.5-VL remains largely unaffected, indicating better adaptation to GUI execution\". To me, it could also indicate that Qwen2.5-VL is not really leveraging the CLI and only relies on GUI. It would be interesting to see the CLI-only performance for that model as well to confirm that hypothesis.\n\n#### Minor\n- How was the human baseline obtained? Was it through domain experts using the same interface as the agents?\n- line 376: Gemini-2.0-Flash (Team, 2024) -> (Google, 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)