InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction
Authors: Bin Lei
Weitai Kang
Zijian Zhang
Winson Chen
Xi Xie
Shan Zuo
Mimi Xie
Ali Payani
Mingyi Hong
Yan Yan
Caiwen Ding
Venue: NeurIPS 2025 poster ()
Keywords: LLM based Agent
Multimodal Generalist Agent
Automated Computer Interaction
Infrastructure
Primary area: infrastructure
TL;DR: We have built a highly modular, multimodal general-purpose agent that can interact with a computer via text, images, audio, and video.

Abstract: This paper introduces \textsc{InfantAgent-Next}, a generalist agent capable of interacting with computers in a multimodal manner, encompassing text, images, audio, and video.
Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within a highly modular architecture, enabling different models to collaboratively solve decoupled tasks in a step-by-step manner. 
Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e., OSWorld), but also more general or tool-intensive benchmarks (e.g., GAIA and SWE-Bench).
Specifically,
we
achieve a $\mathbf{7.27\\%}$ accuracy gain over Claude-Computer-Use on OSWorld.  
Codes and evaluation scripts are included in the supplementary material and will be released as open-source.

GUI click p JInfantAgent-Nextt A Multimodal Generalist Agent for Automated Computer Interaction
Bin Lei Weitai Kang²* Zijian Zhang' Winson Chen' Xi Xie Shan Zuo
Mimi Xie Ali Payani Mingyi Hong' Yan Yan Caiwen Ding'
University of Minnesota University of Illinois Chicago
University of Connecticut "The University of Texas at San Antonio "Cisco Research 1e1001260umn, edu, wkang1260uic.
Abstract
This paper introduces INFANTAGENT-NENT a generalist agent capable of inter- acting with computers multimodal manner, encompassing text, images, audio, and video. Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within highly modular architecture, enabling different models to collaboratively solve decoupled tasks step-by-step manner. Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e.. OS OSWorld), but also more general or tool-intensive benchmarks (e.g.. GAIA and SWE-Bench curacy gain over Claude-Computer-Uss on OSWorld. Codes and evaluation scripts are open-sourced at https: // githtb..h..... com www.o///nnnnaapll..ooaama wwtt://aappleeeaaaaaaeeeee
Specifically, we achieve a 7.27% ac-
1 Introduction
Automated AI agents [28, 41, 35, 47, 29, 22] are vital in today's digital era. By integrating large language models (LLMs) [2. 19] and visual large language models (vLLMs) [36, 6, 32, 46, 10], they can multimodally understand user intents-including text, images, voice, and video-and, with minimal human intervention, convert these intents into precise sequences of interface actions. Through LLM vLLM-driven planning, advanced perception of UI elements (such as buttons, text fields, and images), and modular execution framework, they significantly reduce manual effort and enable the efficient execution of complex workflows across diverse software environments.
Current automated AI agents can be broadly classified into two categories, and they have not made sufficiently fine-grained distinctions at the agent's 's tool selection and execution levels, which leads to significant limitations in practice.
The first category consists of tool-based agents, such as OpenHands [47], OWL [29]. and AutoGPT [18], that equip LLMs with a suite of predefined tools (e.g., code generation tools, web search tools) boost task-specific accuracy. However, because these agents usually rely on single model to decide when and how to use each tool, they require the manual definition and integration of tools for every possible desktop scenario-an infeasible and brittle approach that limits generality. 2 The second category comprises pure dision-based agents, such as UI-TARS [40], and Aguvis [54], which use vLLMS control computers via GUL, This design allows broader applicability, since it bypasses the need for tool integration. However, high- high-resolution reasoning with a single model hurts accuracy on tasks that could be easily handled by simple tool calls, such as document editing or code manipulation where tool-basee agents excel. 0 Although advanced models (e.g.. Claude-3.7-Sond 7onnett [6]. o3 [38]) can decompose complex problems into precise, step-by-step plans, they frequently fail at execution-mislocae
EEqual Contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

coordinates [12] (GPT-4o [25] attains only 0.8% accuracy on the ScreenSpot- benchmark [30]) or selecting incorrect line numbers during file edits. Conversely, specialized vision modules (such as Asual-grounding models [31, 17]) exhibit limited reasoning capacity and restricted context windows, preventing reliable inference of subsequent actions. Ensuring both high task-level accuracy and broad generality requires a hybrid agent paradigm that unifies tool-based and pure-vision approaches.
which undertakes detailed modu- larization of agent workflows, tool selection, and tool execution, in favor of a modular architecture with unified dialogue context. Each subtask is routed to the most appropriate specialist- reasoning models handle logical inference, and their outputs are seamlessly merged back into the conversation
Inspired by these challenges, we present INPANTAGENT-NES
models localize UI elements, audio-analysis mod-
els interpret sound, and so on on-and history. This design enables true multimodal interaction with computer interfaces, rather than being confined to formatted HTMI. or Accessibility Tree manipulations or purely vision-based control. Figure presents several real-world task examples addressed by AFFANTGGENT-NEX
TEXT
open Flee_ de25 .Lox)
Use tool oper_ file
python codl analysia data
pythen
the rerult
many wheels da stean loconotives have total in the attached .xlsx file?
Locsuction dy 5rie
Commrctee DA
weels
Teut worth
sirw becer
Image
eouar, click( 'ALL Bookeurks
nousE, click( 'Donr') )
can you save this webpage I'n looking at booknarks bar so I can cone back to it later?
Audio
tool cudto to the assto avelyats model queations about the aveta
can you listen to the recording and tell ne the page nunbers I'n
> parse avtia/endis
Professor lostmick recsoranded
b53.m question 'Whtt pape nacter recomesdded for ald- ')
Fulliwing paps subbers for Caleutio mnd- Lern:
supposed to go over for nid-tern?
niffeeent fal equat lene on page 14s, qpeltically probiives slai lan to proc lows 32 39, and 44.
the last page In the integration section, page 191, with Focus or arobicn pages 132, 133, 134,
I've attached the Honework.np3.
Figure 1: Three real-world task examples addressed by INFANTAGENT-NEN different modality capabilities. Input requests are shown in the block, actions taken by the agent are shown in the block, and execution results appear directly below the action block.
each requiring
Our contributions are as follows: (i) We introduce INFANTAGENT-NERT an open-source, multimodal generalist agent framework that interacts with its host computer via text, images, and audio. (ii) We have modularized the agent's 's workflow, tool selection, and tool execution in the tool-based and pure-vision agent paradigms to achieve both high task-level accuracy and broad generality. (iii) We optimize suite of commonly used agent tools and release them open-source which supports isolated execution environment, enabling future research in the community.
2 Related Work Comparison
In Table 1, we compare INFANTAGENT-NEXT with existing agents across eight key dimensions.
Generalist agents are capable of supporting various computer tasks, while specialist agents are typically designed to address a specific class of problems. For example, OpenHands [47] and MetaGPT [22] primarily focus on software engineering tasks.
Generalist:
For common operations such as web search or file editing, integrating built-in tools into agents can greatly streamline workflows and enable structured output generation. For instance, AutoGPT [18]'s search_wer tool retrieves search results by issuing HTTP requests and parsing the returned JSON/HTML structure. In contrast, agents like UI-TARS [40] lack such tools and rely solely on pure vision- based capabilities for parsing.
Built-in Tools:
Visual Grounding: Built-in tools are often insufficient for diverse computer poftwares-partionn with professional graphics software or niche PC games. Agents such as ShowUl [31], which are equipped with visual grounding capabilities, are better suited for handling these scenarios.

---

Multi-Model Support: Different models exhibit different strengths: reasoning models are effective at complex analytical tasks, non-reasoning models offer faster response times, audio analysis models handle auditory inputs, and visual grounding models enable precise spatial localization. Assigning each model to tasks that match its strengths can significantly enhance overall agent performance. MetaGPT [22] exemplifies this multi-model integration strategy.
Memory Retrieval:
As dialogue histories grow, model inference overhead increases. This overhead can be mitigated by retrieving relevant memory segments and injecting them into the context on demand. LangChain [27] demonstrates this approach by offering multiple memory classes such as memory.
summary memory and retrieval-beae
Dynamic Toolset:
To extend agent's functionality, developers often include a broad range of tools. Tool usage instructions are typically prepended to the conversation history to guide the L.M.. However, large tool inventory lead to increased inference cost and challenging tool selection. Frameworks like LangChain [27] address this by dynamically selecting a relevant subset of tools from the full library, thereby reducing the cognitive and computational burden.
User Interaction during Execution: User's initial prompts are sometimes underspecified (e.g., Please write short story.). In such cases, it is crucial for the agent to interactively elicit additional details rather than completing the task blindly. For example, OpenHands [47] includes a MessageAction class that enables the agent request clarification from the user.
Dedicated Computer: We design a fully isolated execution environment supporting CLI, Python scripting, and GUI interaction; implementation details are provided in the Appendix D.
Table 1: Comparison of INFANTAGENT- NEXT with Different Agent Frameworks
Generalist
Buil-in
Visual
Muli-Model
Memory
Dynamic
User Interaction
Dedicated
Name
Agent
Tools
Grounding
Support
Retrieval
Toolset
during Execution
Computer
AutoGPT [18]
X
X
X
AabyAG1 [34]
LangChain [27]
/
d
X
MetaGiT [22]
X
X
X
X
OpenHands [47]
X
/
X
V
X
UI-TARS [40]
X
X
V
X
X
X
AutoGen [49]
X
/
AGUVIS [54]
X
X
V
X
X
ShowUl [31]
X
OS-Atlas [51]
X
X
AutoAgents [11]
X
InlantAgent-Neex
Compared prior agent frameworks, INPANTAGENT-NEXT uniquely integrates all of the above features. See Section 3 for further details.
InfantAgent-Next
In this section, we first introduce the overall architecture of INFANTAGENT-NEXT to illustrate its operational logic Sec. 3.1. We then describe some crucial components in Sec. 3.2, including the memory extraction mechanism employed during execution, toolkit configured for the system, etc. Finally, we propose our novel improvement on important actions of the agent in Sec. 3.3, including Mouse Click and File Edit. We provide detailed case study in Appendix A.
3.1 Architecture
Overview:
The architecture of INFANTAGENT-NEXT is illustrated in Figure 2. INFANTAGENT-NEXT meticulously modularizes the workflow where different models complete different steps. The process begins when the user submits a request. We first allow the user to configure the arguments for Workflow models and Tool models. After the agent is initialized, it analyzes the user's 's request to generate tasks. It then performs tool selection and invokes the selected tool to complete the tasks. Finally, the agent checks whether the current task has been successfully completed.
Argument Configuration (Figure 2-upper left): In this stage, users can customize and assign different models to different tasks. These include three workflow models responsible for planning, tool selection, and task execution, respectively, and three modality-specific tool models for handling images, audio, and video.

---

Workflow models
planning
Planning Prompts
Optional Argument:
Dar defauit if None
Assign
For Tool Selection
Tool selection Prompts
model
Mundatory Argument
dutics
Execution Prompts
For Executioit
Visual grounding prompt
Package
Audio parsing prompt
multimodal
toolkit
Tool modelt
Video parsing prompt
Vision/Auddlg/
Argument
Agent Initialization
User request
Configuration
User
Toolkit Selection
Select
Tools
request
Register the selected toolkit
Toolsets
Task
Observation
Uppytr
parse & store
Interactive
No
User request is finished?
tarrs
Environment
Agent Workflow
Action
Yes
Finish
Execution
Figure 2: INFANTAGENT-NENT architecture overview. & User input argument and request. Envi- ronment related icons: C Agent Interaction Environment. = Terminal interface. C GNOME desktop. S: Jupyter. Models related icons: C: Load Workflow models. @::
Planning model to: Tool Selection model :: Execution model @: Planner. X: Tool Selector. & Executer. Tools related icon:
C Load Toolsets. @ Tool models argument: Vision_model_name
c Tool models argument:
Audio seciio_moee_nae name. C: Tool models argument: Video sotoo_mmoee,____ name. C: Multimodal toolkit. @
Other Toolkits. Memory related icons: Load agent memory. @: Request from the T
Agent Memory Cache, used for storing all memories.
II. Agent Initialization (Figure 2-upper right): During initialization, we assign specific roles each model with tailored prompt templates. For example, some tools are associated with specific tool models and their prompt templates, and are then packaged into unified multimodal toolkit. At this stage, the agent's 's memory cache is initialized, with the user's request stored as the first entry.
III. Agent Workflow (Figure 2-bottom): The initialized multimodal toolkits, together with suite of predefined utility functions, are organized into a comprchensive toolset. This set of tools, along with the interactive environment, all models, and the agent's memory cache, is integrated into the agent's workflow. The user's s initial request serves as the input and triggers the following iterative loop: O The Planner model analyzes the request and current state produce task, which parsed and stored in memory. @ The agent reconstructs the memory using prompt template for toolkit selection. The Tool Selection model then selects the most appropriate tool from the toolset. The selected tool is registered and activated for this iteration, while others remain idle. O The Executor model invokes the tools and gathers feedback from the environment to complete the task. 4 we successfully resolve the task, the loop terminates; otherwise, the process proceeds to the next iteration.
3.2 Component Design
Memory: Since the agent must perform three distinct types of tasks masks-plannngg tool selection, and execution- it is insufficient to simply log the raw dialogue history. Instead, each model-

---

generated response is parsed using special tags and sequentially stored in the agent's 's memory cache. The agent's reasoning enclosed within <thought>. ... </thought> tags; task assignments are marked by <task>.. </task>; selected toolkits are indicated by <toolkit>. </toolkit>; and tool executions are represented by either When the agent transitions to a new task, sub-
Saccutteebsh___eeee ... </execute__eaeeeeeeeeeeeeeeeeeee bbash> or
<execute caatttte_pthoo..... < ceepxeeuee__ythhoeeeee set of the memory cache is extracted and reconstructed into the dialogue by concatenating relevant memory attributes. Task-specific tags are activated to prevent the LLM from generating irrelevant content. Finally, the reconstructed dialogue is combined with predefined prompt templates to form the conversation history for that task. The restoration process follows these steps:
I. Planning: During the planning phase, the agent retrieves all memories except those associated with tool selection. All tags are removed during reconstruction, except for the <task>. </task> tags. These reconstructed memories are inserted between the planning system prompt and the end prompt for planning, both of which are detailed in Appendix B.
II. Tool selection; For tool selection, only the most recent task memory retrieved from the cache, and all associated tags are removed. This content is then appended directly after the tool selection system prompt, as provided in Appendix B.2.
During execution, all memories except those related to tool selection are retained. When reconstructing them into dialogue, only the serecuteebasht ... </w.ecuteeee........e bash> and <execute_ eexecute_python>... </ eexecute_pythooee tags are preserved. The resulting dialogue inserted between the execution system prompt and execution end prompt, shown in Appendix B.3.
III. Execution:
Tools: To invoke a tool, the agent prepends its documentation and usage examples to the dialogue history. However, as the toolset expands, this strategy introduces two challenges: (1) inflated inference overhead due to the volume of text, and (2) increased difficulty for the model to identify the correct tool. To mitigate these issues, we categorize tools and dynamically select a relevant subset during each tool selection step. Specifically, we define seven toolkits: File Reading, File Searching. File Editing, Web Browsing, Computer Use, Code Execution, and Advanced Tools. The Computer Use toolkit includes all keyboard and mouse operations as well as multimodal tools, while the Advanced Tools toolkit provides various composite commands. Each toolkit is accompanied by usage example. Detailed descriptions of tool functions are available in Appendix C.
3.3 Mouse Click & File Edit
As noted earlier, SOTA models, such as Claude-3.7 3.7-ooneet or GPT-4o, remain limited in mouse-click action (visual grounding) accuracy and robust file editing. We incorporate dedicated mechanisms to mitigate these limitations.
Mouse Click: For mouse_click oper- ations, the model is required to out- put two parameters: (1) the name of the target element (e.g.. Google Chrome VS Code icon), and (2) detailed de- scription of its posi- tion and shape. As illustrated in Algo-
Algorithm Iterative Region Cropping and Mouse Click Logic
Require: agent
1: if Last(agent.memory) .type # mmouse__ciiee then
2: return
3: end if
4: description <-
5: region screenshot
6: for i 1 to n do
coord, VISIONGROUNDDIN region) 8: region + CROPSCREEN((entter = coord,, size = CROP_SIZE
end for
10: final coord + VisSoNGRounDDnN(dsrripiinn region)
rithm we first ver- ify whether the agent's 's most recent action is mouse_ _ooss__kkkkkk if not, the procedure terminates early. Otherwise, we retrieve the associated description from memory and treat full-screen screenshot as the initial search region. We then enter an iterative loop of length n, where each iteration passes the current region and the description to the visual grounding model to obtain candidate coordinate coord, We subsequently call CropScreen centered at coord; to extract a smaller region for the next step. After completing n iterations, the final region is passed to the visual grounding model once more to predict the refined click position final_ccoree coord, which is then used by PerformClis
11: (ERFORMCLICXX(mal coord)
5

---

to click on the screen. By progressively narrowing the search region, this method enhances visual grounding precision and reduces distraction from irrelevant pixels.
File Editing: There are two main formats for the sile_edis com- mand. (1) The three-parameter for- mat used by SWE-Agent [44] spec- ifies the file path, the exact string to replace, and the replacement. often fails when the target string long or occurs multiple times. (2) The pour-parameter format used by OpenHands specifies the file path, start and end line numbers, and the replacement string, but prone to errors such as incorrect overlapping line ranges. To ensure that the agent can perform file edits accurately, we have encapsulated the logic from Algorithm 2 as fol- lows: First, we verify whether
Algorithm File Editing Logic
Require: agent
1: if Last(egent.emorr) + "rile_edit' then
2: return
end if
4: request + Last Lass(ggem.eemorr)cdi. dedttas
for t- to MAX MAX_ITER do
# Generate edit plan
(s, Scunt: C, Grom)) CENERATEEDTPPAAN(yqeeeet
# Verify plan boundaries
if FileLine(s) = Scont A FileLine(e) = ccnm then APPLYEDIT(s. e, request) return end if
8:
9:
10:
# Fallback: locate best matching lines
11:
a, FileLine
12:
de + FileLine (c)
the agent's 's most recent memory en- try is of type file_edit, If not, the procedure terminates immedi-
13:
m, FUZZIFY (Sceat
14:
Me <- FUZZIFY (Ceent
15:
(',,e) FINDBESTMATCH(TI, me)
ately. @ If it is, the agent extracts
16:
request +- (8', é', 0ss Oe, requesl)
the edit request from memory and 17: end for
invokes CenerateEditPlas (request), which returns the proposed starting and ending line num- bers and e, along with the expected contents of those lines, s_cond and e_cont. @ The agent then checks whether the actual content of line s matches s_cont, and whether line matches e e_cont. If both checks pass, it directly applies the edit using ApplyEdit e, request) and termi- nates. ① If the boundary checks fail, the agent switches to fallback strategy: it first saves the original contents at lines s and e as a_s and a_e, then fuzzifies the expected snippets via m_s Fuzzify(6 _s_cottt, the most similar span (s', e') in the file. The edit request then updated with the new span and original content (s', e', a_s,a____ a_e, request), and the process repeats for up to MAX_ ITER iterations.
= Fuzzify (e_ _ccont) and calls FindBestMatch (m_s, m_e) to locate
4 Experiment
4.1 Vision Ability
Benchmark. We assess the visual reasoning capability of INFANTAGENT- NEXT using the OSWorld benchmark [53]. OSWorld provides a scalable, real-world computer environment comprising 369 open-ended desktop tasks. Each task is initialized with fully specified machine state, including high-resolution screenshot, active application windows, and file system context, paired with a natural language instruction and executable evaluation script. The tasks span diverse domains, such as web browsing, file operations, code editing, image manipulation, and multi-application workflows, challenging the agent's s ability to ground natural language instructions in GUI elements.
Experiment Setup. The evaluation is conducted using the official OSWorld codebase [53], executed within a VMware virtual machine. Mouse click events are simulated using PyAutOGUI. For reasoning, we use Claude 3. Sonneet, while visual grounding is performed using UI-TARS-11 5-BB, with max_steps set to 50. For fair comparison with baseline methods, all non-visual toolkits are disabled.
Results. As presented Table 2, INFANTAGENT-NEXT achieves superior accuracy at 50 steps compared to OpenAl CUA and Claude Computer Use. When utilizing Claude-3. 7-Sonnet as the planner model, INFANTAGENT-NEXT outperforms Agent-S2 [3] and all other open-source frameworks, demonstrating its enhanced visual reasoning capabilities.

---

Table Performance INFANTAGENT-NES on OSWorld.
Close source. v: Open source.
Framework
Model
Steps
Open Source
Accuracy
UI-TARS [40]
U--TARS-L.5-2BB
42.5
OpenAl CUA [37]
GPT-4o
200
38.1
UI-TARS [40]
UI-TARS-1.5774
50
38.0
InfantAgent-Ne
Claude-3.7 Munde-3.7-Sonett UI-TARS-
35.3
Agent S2 [3]
Saaude3.7--omie
34.5
OpenAl CUA [37]
GPT-4o
50
32.6
Claude Computer Use [7]
100
28.0
UI-TARS [40]
UI-TARS-1.5 7B
26.9
Claude Computer Use [7]
Claude-3 7.7-onnct
50
26.0
UI-TARS [40]
UI-TARS-72B 7-B--P222PPP
50
24.6
AGUVIS [54]
AGUVIS 72B GPT-4o
17.0
Aria-UI [55]
Aria-UI GPT--4o
15.1
OS-Attls [51]
A--AAlls-Ba---a + GPT-4o
V
14.6
SceClick [12]
SceClick GPT-4o
V
9.21
Qwen2.5 [10]
Qwen2.5-v1-72B
8.8
Table 3: Performance on SWE-Bench- Be---enn----rfiied and SWE-Bench-Lite.
Open/Close Source
Method
Accuracy (%)
WVy-Bench-Verifiee (50 cases)
SWE-agent Claude 3.7 Sonnet Review Heavy [44]
@ppehaans_04_111 [47]
68
InfantAgeent-Next Cla
Open Source
[9]
52
SWE-agent + -W [44]
AppMap Navie v2 [8]
12
Agentless Lite + O3 Mini (20250214) [52]
CodeStory Midwit Agent swe-search
70
AgentScope [15]
66
Close Source
CORTEXA [1]
62
Amazon Q Developer Agent_2024_____________ 02_4_12_022011 [5]
54
Dovlo_2024__00 08 [13]
48
SWE-Bench-Lite (using GPT-4o)
Agentless- 1.5 [52]
32.00
InfantAgent-Neex
31.67
Open Source
Agentless + RepoGrraph [52]
29.67 67.67
OpenHands CodeAct v1.8 [47]
22.00
CodeShellTester [48]
31.33
Close Source
SIMA [43]
27.67
4.2
Logic Reasoning Ability
Benchmark. To rigorously evaluate the logical reasoning capabilities of INFANTAGENT-NEXT
employ two complementary SWE-Bench [26] datasets. First, SWE-Bench-Lite, a 300-issue subset, challenges the agent to interpret real GitHub bug reports, identify defects in Python codebase, propose patches, and verify fixes by exccuting the provided test suites. This setup emphasizes multi-step logical planning rather than GUI proficiency. Second, assess performance the full SWE-Bench- WWE-Bench-Verified suite (500 cases); to manage API-related costs, we uniformly sample 50 cases for evaluation (their instance IDs are listed in the Appendix E). Together, these benchmarks provide thorough yet cost-effective assessment of our agent's 's reasoning capabilities.
Experiment Setup. For SWE-Bench-Lite, our inference is conducted using GPT-4o for plan- ning. tool selection, and execution, consistent with the configurations adopted by most exist- ing agents to ensure fair comparisons. For the 50-case -WF--eench-eerffiedd subset, we employ Claude-3. 3.aude-3.7-Sommae -S for each step, providing direct evaluation against state-of-the-art systems. In both settings, DeepSeek-V3- V3-0324 is used as the dedicated file-editing model.

---

Table 4: Performance of INFANTAGENT on GAIA. The data compared this table are current as of the completion date of our GAIA experiments on April 17. 2025, For results without citations, please refer the official GAIA leaderboard [14].
Open/Close Source
Agent Name
Main Model
Average
Level
Level 2
Level 3
Langtun Agent v2.1]T6]
Cla 3.7-sonet
71.52
83.02
68 60
57.69
Trase Agent v0.3 [45]
ol
70.30
83.02
69.77
46.15
h2oGPTe Agent v1.6.8 [20]
Claude-3.5-Sonee
63.64
67.92
67.44
42.31
Anges
60.00
66.04
65.12
30.77
desearch
GPT-4o
56.97
71.70
58.14
23.08
Ormind v0.1 [39]
55.15
69.81
54.65
26.92
Close Source
Barcetona v0.1
Claude-3, Claude-3.5-sonnct 5-sonnet
50.30
62.26
50.00
26.92
DRP-val-v.LO
46.06
56.60
48.84
15.38
onune
ol-preview
46,06
60.38
44.19
23.08
gbnlp
o1
44.24
52.83
45.35
23.08
LRC-Huuwei
40.61
52.83
43.02
7.69
QwQ-32B
40.61
52.83
43.02
7.69
AgentIM v1.1
gpt-4-turho
40.00
50.94
40.70
15.38
OWL-Roleplayinng
GPT-4o + 03-mimi
58.18
81.13
54.65
23.08
InfantAgent-Ne
Claude-3.7 3laude-37-Soonee
56.97
62.26
62.79
26.92
TapeAgents [42]
Claude-3 75.-entntt
55.76
71.70
53.49
30.77
Belo--eee-Researchh [21]
Claude-3.7 3.nne--.7-Sonnet
55.15
71.70
53.49
26.92
AutoAgent
55.15
71.70
53.49
30.77
Open Source
Open Deep Research [24]
ol ol
55.15
67.92
53.49
34.62
Magnetic-1
46.06
56.60
46.51
23.08
HuggingFace Agents [23]
gpt-4o
44.24
58.49
43.02
19.23
Multi-Agent Exp vO.1
gpt-4-turbo
39.39
54.72
38.37
11.54
FRIDAY [50]
gpt-4-turbo
34.55
45.28
34.88
11.54
Results. On the SWE-Bench-Verifee benchmark (50 cases), INFANTAGENT-NE achieves leading accuracy of 66 %, outperforming many proprietary agents. Notably, several closed-source agents such Amazon Q Developer Agent and Emergent El perform worse, highlighting the effectiveness of our agent's architecture and integration. On the SWE-Bench-Lite benchmark, where we ensure fair comparison by restricting all agents to use GPT-4o, INFANTAGENT-NEXT maintains competitive performance with an accuracy of 31.67%. This places our system among the top-performing open-source agents, demonstrating robustness and adaptability.
4.3 General Task Performance
Benchmark. We evaluate INFANTAGENT-NEN the GAIA benchmark [33], designed to assess general AI assistants. GAIA comprises open-ended, real-world questions across three difficulty Level (basic), Level 2 (intermediate), and Level (advanced). These tasks require integrating core capabilities, including reasoning, multimodal understanding, web navigation, and tool usage, to produce single, verifiable answer.
Experiment Setup. The evaluation conducted on the GAIA benchmark validation set. We em- play Claude-3. 7-Sonnet for reasoning and task execution, Deepseek-V3-033 for tool selection, 2od--oo-uuio--1 o-eeeviewww for auditory processing, and UI EEI-TARS-T 1. 5-7B [40] for visual understand- ing. System prompts are sourced directly from the official GAIA leaderboard to ensure consistency with standard evaluation protocols.
The performance on the GAIA benchmark is presented in Table 4. While closed-source agents maintain a performance edge, INFANTAGENT-NER NEXT ranks second among open-source agents, trailing only OWL, and achieves fftte-of-the--ee art results on Level 2 difficulty questions.
Results.
4.4 Ablation on Visual Grounding design
Benchmark. ScreenSpot-Pro [30] provides high-resolution professional scenarios to stress-test visual grounding. It contains 1,581 annotated sereenshots drawn from 23 applications spanning five industry domains and three operating systems; each sample pairs natural-language language prompt with the exact on on-screen target element, enabling precise click- cicck-cc evaluation. We therefore use ScreenSpot-Pre quantify the performance our newly designed Iterative Region Cropping.
Experiment Setup. We performed our experiment using 2x A100 80G GPU. We first conduct an ablation study the Iterative Region Cropping setup. Specifically, for CROP_SIZE, test three

---

04.0
12-
Accuracy (9)
54.0
05.5
47.8-
20
Vidth
Height (Pixe3)
Teans
(a)
(b)
(c)
(d)
Figure 3: We conduct an ablation study on the Iterative Region Cropping setup from four perspectives. (a) Vary the region width.
(b) Vary the width and height ratios of the cropped region (relative to the full image) (c) Vary the height. (d) Vary the number of iteration
different settings with a fixed iteration length of n = 2 to evaluate its impact: (Fig. 3 (a)) we fix the cropped region area to 150K pixels and vary the region width. (Fig. 3 (b)) we examine how varying the width and height ratios of the cropped region (relative to the full image) affects performance. (Fig. (c)) we fix the width at 700 pixels and vary the height. (Fig. 3 (d)) we further ablate the number of iterations, fixing the region width and height ratios to 0.3 and 0.25, respectively.
Results. Among all strategies, the setting in Fig.3 (c)-fxxing the width while varying the height-yields the highest accuracy and is preferred for determining CROP SIZE. For Fig. 3 (d), accuracy converges at the third iteration, and using two iterations offers a favorable trade-off between performance and inference cost.
4.5 Evaluation on File Editing
Benchmark. We evaluate the file editing capabilities of INFANTAGENT-NEN using the SWE-Bench- Verified dataset [26]. a curated subset of 500 real-world GitHub Jssue-Pull Request pairs from popular Python projects. Each task is human- validated to ensure sufficient context and is verifiable through the project's unit tests. Tasks involve diverse code modification instructions, such as deleting or updating code across multiple files, testing the agent's s ability to interpret and implement complex changes. We randomly sampled 10% of the tasks from the SWE-Bench- Verified dataset across repository categories to form our evaluation set. The IDs of the selected test cases are in the Appendix E.
Experiment Setup. We evaluate each task solution and categorize them into three sets: Crotal, the set of all task solutions; Cfail. solutions failing due to execution-model errors; Crepaired, solutions success- fully repaired. We define the following metries: RepairSuccessRate (RSR) Crepsired // CCaall
and OverallRepairRate (ORR) = [Crepaired /|ttaal
Results. INFANTAGENT-NENT
smaber of-ly-on emme (Find)
Pythun scutas senor Food)
exhibits strong performance on file editing tasks in the SWE- Bench- Verified dataset, achiev- ing a total success rate of 90.4% with 84.3% RSR and 51.4% ORR. The agent excels fix- ing line number off-by-one er- rors (30% repaired), content un- matched (13% repaired), and Python syntax errors (6% re- paired), demonstrating robust code interpretation and repair ca- pabilities. Overall, these results highlight INFANTAGENT-NEXX as highly capable agent for au- tomated file editing, with minor limitations in edge cases.
70
7
Lise zusber off cruse (Nit Finol;
Line mnder zoi Sound (Fized)
unastched (Fised)
ouder Flasd)
Concest mmmariued (Not Phowe)
Connet commands
60
Pyliso syntex erco (Mood)
Couns
40
30%
34
38%
30
16
5k
13%
0.856
6%
2.
0%
Figure 4: Evaluation on a subset of Verified.

---

Conclusion
We presented INFANTAGENT-NER tool-based and pure rision-based paradigms through modular, context-aware architecture. By routing subtasks specialized models and maintaining unified dialogue context, INFANTAGENT- NEXT avoids the limitations of single-model systems and delivers both high task accuracy and broad applicability across diverse interfaces. Empirical results on OSWorld. GAIA, and SWE-Bench confirm its effectiveness, including a 7.27% accuracy gain over Claude- Mauude-oompute--US Use.. All code, models, and evaluation tools will be released to support future research in multimodal agent design.
multimodal generalist agent that bridges the strengths of
Acknowledgment
This research was partially supported by Cisco Research. We thank the open-source community for their valuable suggestions, issues, and pull requests.
References
[1] Cortexa: Enhancing Ilm agents for software engincering tasks via improved localization and solution diversity, https://researe: nvidia. com/labs/atl hataaaaaaaalccotteeeaa///
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2303:03033007774 2023.
[3] Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialis framework for computer agents. arXiv preprint 2rXh::250.000000 2025.
[4] AgentFarm. Agentfarm. https://aide.. awazaa www.a/awwww....oo//////aoaaaaaa/ www.:oo..cc//////eellopr../
[5] Amazon. Amazon q developer. petps://akke
[6] Anthropic. Claude 3.7 sonnet. Available at https:///wwwt httppp///ww.aaaaapppccc.oooaoooa
[7] Anthropic. Claude computer use.
Available at https://www.anttrop..ccco
3 3-5-mode 4-5-modelss-ae sand-comuter--uee
[8] AppMap. Appmap navic v2. http:://aappppptp/ppphhtttpppppp html.
[9] AutoCodeRover. Autocoderover. http:s///ww..utooooorreeee..
[10] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 23Xv:22502.13923 2025.
[1I] Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288 2023,
[12] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint 1009:32401.10935. 2024.
[13] devlo. https://devlo.ail ail. /ttps://deevo.aa//
[14] Hugging Face. GAIA benchmark leaderboard. http:://uggiiggaccccoo/////// http://.a.......appc//////t/aaaa benchmark/ leaderboard, 2025. Accessed: 2025-05-15.
[15] Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma, Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, et al. Agentscope: flexible yet robust multi-agent platform. arXiv preprint arXiv:2402. 14034, 2024.
[16] Google. Langfun. Gitluu repository, 2025. http://ggttuuu..oogooooo.o hamm//oogle/langttee
[17] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024.
10

---

[18] Significant Gravitas. Autogpt. https:// http://hwwaaeoeoootaaaaaaaattaaa 2025. Ae- cessed: 2025-05-15.
[19] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1294 2025.
[20] H2O.ai. H2o.ai. ittps://id.pulic. public w2w.hh...... ai/.
[21] HKUDS.
Auto-deep-researde
GitHub repository.
https:///github.comm/
Bato-Deep-Bee Research.
[22] Sirui Hong. Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang. Zili Wang. Steven Shing Yau, Zijuan Lin, Liyang Zhou, al. Metagpt: Meta programming for multi agett collaborative framework. arXiv preprint arXiv:2308.00352, 3(4):6, 2023.
[23]
huggingface. agents.
Huggingface agents.
[24] huggingface. open deep research. http:////uuguiiggaeeee..eobaeeee
[25] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
[26] Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofrr Press, and Karthik Narasimhan. SWE-hench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.
[27] LangChain Al. Langchain. GitHub repository, 2025, Accessed: 2025-05-14.
[28] Bin Lei, Yuchen Li, Yiming Zeng. Tao Ren. Yi Luo, Tianyu Shi, Zitian Gao, Zeyu Hu, Weitai Kang, and Qiuwu Chen. Infant agent: A tool-integrated, logic-driven agent with cost-effective api usage. arXiv preprint 241.::2411.011111 14, 2024.
[29] Guohao Li, Hasan Hammoud. Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Commu- nicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-575 -20008, 2023.
[30] Kaixin Li, Ziyang Meng. Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma. Zhiyong Huang, and Tat-Seng Chua. Screenspot-pre. Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.0798/ 2025.
[31] Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang. Shiwei Wu, Zechen Bai, Weixian Lei. Lijuan Wang, and Mike Zheng Shou. Showui: One lision-laguaage-action model for gui visual agent. arXiv preprint 2xx::2411.17466 2024.
[32] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16296-26306. 2024.
[33] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants, 2023.
[34] Yohei Nakajima. Babyagi. GitHub repository, 2024. Accessed: 2025-05-14.
[35] Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip Yu, et al. A survey of webagents: Towards rext-generation ai agents for web automation with large foundation models. arXiv preprint arXiv:2503.23350 2025.
[36] OpenAl. Chatgpt. https: // openai. com/ gpt-4.
[37] OpenAL. Computer-using agent.
Available at
hecps://openai..co//nnee
computer-using-agenn/
[38] OpenAL Introducing openai o3 and o4-mini. Available at httpe://openaa..com/ii
introducing- o3- and-o--mmn//
[39] Ormind. Ormind. https : /ttp://///mmm..i//..

---

[40] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501. 12326, 2025.
[41] Pascal Sager, Benjamin Meyer, Peng Yan. Rebekka Wartburg Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin Grewe, and Thilo Stadelmann. Ai agents computer use: A review of instruction-bart based computer control, gui automation, and operator assistants. arXiv preprint arXiv:2501. 16150, 2025.
[42] ServiceNow. Tapeagents. GitHub http://www.aa..oaa//a//////////a
ui_ [__demm/eexamplle gaia_agent.
[43] SIMA. Sima. https:/ httpa//ggag.aaaaaaaaaa.aaaaaa/an 1224776_s1maa_pii
[44] SWE-agent. Swe-agent. https: /tops://giioe htwp:/uwwwcccmm/eeeegget//e///ae agent.
[45] TRASE. Trase. https://wwwt tww.esystems.ccommm com/
[46] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Kegin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-languae model's perception the world any resolution. arXiv preprint arXiv:2409. 2024.
[47] Xingyao Wang. Boxuan Li, Yufan Song, Frank Xu, Xiangru Tang. Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. In The Thirteenth International Conference Learning Representations, 2024.
[48] WisdomShell. codeshell. attps://githinn http:://ggttab...cm///adaasseae ccm//WisdooSSteeloooeeeeellll
[49] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint 2308 08155, 2023.
[50] Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456 2024.
[51] Zhiyong Wu. Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, al. Os-atlas: foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218. 2024.
[52] Chunqiu Steven Xia, Yinlin Deng. Soren Dunn, and Lingming Zhang. Agentless: Demystifying lim-based software engineering agents. arXiv preprint, 2024.
[53] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ender tasks in real computer en environments, 2024.
[54] Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong, Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454 2024.
[55] Yuhao Yang. Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16225 2024.
12

---

A Case Analysis
Figure 5 illustrates the step-by-step process by which INFANTAGENT-NEN solves a real-world query: "According to the World Bank, which countries had gross savings of over 35% of GDP for every year in the period 2001-2010?"
To answer this question, INFANTAGENT-NEXT performs three sequential cycles of Planning + Tool Selection Execution:
The agent selects the rer_rrreer toolkit, conducts a web search, performs mouse-click operations, and captures desktop screenshots execution outputs. This enables it to successfully locate and download the relevant CSV file.
First loop:
Second loop: It then selects the File_Seell Search and File_ __iew toolkits, using Bash commands to inspect the directory structure and examine the contents of the CSV file.
Third loop: Finally, the agent selects the Lodd__Excctiin toolkit and executes Python code to parse and aggregate the data in order to identify qualifying countries.
Through these modular and interpretable steps, INFANTAGENT- NEXT arrives at the correct answer with minimal human intervention.
Turn 1
Turn 2
Turn 3
According the world Bant, which for
craoughtst can ane the fite tas been fownloaded. ctloooddc///tougeee c/ltought>
choughoset''
enalyse data and countries with grose savingg over IS of ape for yeafron 2001-201110/ houghty chusii wille pylhion script to enaoysis the cay Tilesc/Takk
Inpur Quention
2005 20107
Tntel
analyne rilae:/Tasio Tank
Bee wir
<thoughtz find countrics
Clloughtste should file
gress suvings over or aoe
file view toolkits
Coeer draie BARK
for every year fhon 2081-2010
to flarsh this taske Thoughts De vileee/eeeeeeeeeeeeeeeeeeeee
world
<hoougttWW should use code execution trolec/Thoughts
poogle, cxsrcs for "Herld Bank cara grase
cTaskoGpan uaz brouser and cerfors
Tool Selection
Boach
<TocliitsCode Farcort anc/tool
country 2008"</Tai </Taako
clloughtst rotice that the downloaded file actuelly
Tool Selection
Srwe Dade
Tank
Coope Talk
(desptte coov actenston gare 1t).
cHoougtiee Loe should use the web Broase tookkat to Fintsh this Theughts
Lee's untip it to access fucads Seat
<lhoughtstet use following script check The results caeefully by. <TThughh
chthugght>
Executennnip worldbark, erocosse/bacaeee
<erecute>
Tool Selection
Execution Result:
teport
Fanuar python code
(evit coden))
countries.oom over
<Thoughtstet's stare the sTauth//
ing: Netadata Indicutor.
coreutereeon (wser() dibiecates beggoogi
eso a 483. 'e's
in/Loting: AP,_NY GKS, ICTA
encodlage uuf----s- as
25_052_es er_ewy v2 483.cxv
infiatiog:
Netodeto country
for country in sortet/coartrilnn over 31): pelet c/
API NY. SCT.SIIS.TT.. csv
fata files, chhest/
Execution Result
foo
(Exessteshuad -n 4PL wro.. 2010.
code:a)
2or.33 22 /evec///eeeteee sEvecutohest Netadats Inticcttr
Bruwei, Chine, Morocco, Singopare
Execution
AY1 985 2011 bevt/besttte> ctencutesbedd Patadata, Country.
<Thoughts? see the rearch results for Herld
<Theught= the snalysts the world Bank data gross saviings countries that had gross savingss YoE of for in the pertod 2001-2010. asswer Brunei, China, Porocco, Stngapore</hhoghts crintato Finiato
Bank data grees savings of Thuughs
citeesste
Exccution Result:
dberstensileee CSV't /aaacate
(exit codc-a)
"Dota Source", 'sorld Developeent
Indicutors',
... [To Long truncated?
"Last Updated Date 77825-84-15',
... ..oo Long truncated]
"Country Rlas",, "Country Code",
"Indicator
"Indieator Code".
"1960", ; "1961", "1962", "1963", "1964"
[Top truncnted) Exercmmon
Execution
Flaisk
Figure 5: Cases analysis. Zoom in to view the detailed content in the screenshot.
B Workflow Prompt Templates
B.1 Planning
13

---

Listing I: Planning system prompt. It is inserted at the beginning of each planning.
In our interaction with the **User (requester) our goal is to gradually resolve their request or answer their questions. The process involves three roles: **You (reasoner)**, **Me (executor) **, and **User (requester)*s.
In each reaponse, you must execute only one step, choosing to either assign task to **Me (executor) **, or request information from the User (requester)
*+I (executor)**
help **You (reasoner) ** complete tasks that **You (reasoner) cannot perform directly, including:
[task_ category
Our goal is to resolve the **User (requester) ss's request step by step
If **You (reasoner) ** want assign task ** (executor) **, please use the bask>...<<tastte tag to describe your task, don 't give the detail execution commands for now.
If .+You (reasoner) want to request information from the User requester)**, ask then directly without using any tags.
If you believe the user's request has already been resolved, please answer the **User (requester) **'s request based on the entire conversation history and at the end of your ansver add <finish> exit</finish>. Here is example:
USER: <user_request>
Can you help me to dovnload a PDF file called example pdf from the internet and convert its Table 1 to csv file?
</user_request request>
ASSISTANT: <task>Download the example pdf from the internet </tak>> USER: [Downloads output]
ASSISTANT: <task>Convert Table 1 of example. pdf to cav file. </ task>>
USER: [Converts output]
ASSISTANT: Verit> if the cav file exist </task>
USER: [Terminal output]
ASSISTANT: Now the task is finished and we can provide the csv file to the user. siinsshexit<<fiiiee
Listing 2: Planning end prompt. It appended the end of each planning.
If you believe the user's 's request has already been resolved, please answer the **User (requester)*a ** request based the entire conversation history and at the end your answer <finish> exit</finish>,
Otherwise, provide assign a task to ** (executor) ** within the appropriate execution tag: Use the <task>... </task> tag for the task that you would like to assign to me.
Please do NOT repeat the similar analysis that you have already provided.
Listing 3: Planning avoid repetition prompt. appended to the end of the planning dialogue whenever the planning process is needlessly repeated.
If you believe the user's request has already been resolved, please answer the **User (requester)*** s request based on the entire conversation history and at the end of your exit</finish>>
add <finish>
Othervise, please assign a task based on your analysis. Please do NOT repeat the similar analysis/Tast again!
B.2 Tool Selection
Listing 4: Tool Selection System Prompt. it is inserted at the beginning of each tool selection action.
14

---

would like finish task, please help choose the suitable commands complete this task.
1, File editing related commands:
This set of commands can be used to view file content, well as perform additions, deletions, searches, and modifications files
If you want to select this set of commands, please return: <toolkit> file_edit</toolle 2. Code execution related commands:
This set of commands If you want to select this set commands, please return: <toolkit> code_exec</tooe
be used execute code snippets.
3. Computer interaction commands:
These commands can be used to interact with the computer via the keyboard and mouse.
If you want to select this set of commands, please return: <toolkit> hat.ieeriiineratiooo///oooo
4. Web browsing related commands:
This set of commands
be used interact with web pages.
If you want select this of commands, please return: <toolkit> web bobbbrorse<ttookii 5. File understanding related commands:
This set of commands
be used to understand the content of files.
Such reading files, view images, listen audios, watch videos etc.
If you want to select this set of commands, please return: <toolkit> file_ antouunndrsttnd serrrstad</toolkkee </tookiit> If you want to select multiple with commas.
of commands, please separate them
For example, if you think we not only need to edit some files but also execute some code, you should return: <toolkit>file_edi code_exec<<toolkke
B.3 Execution
Listing 5: Execution system prompt. it is inserted at the beginning of execution. It includes usage instructions and examples for the selected tool functions.
{Selected_Toce Tool_ Brcitedd__ooo_Guidee__e_eeee (uode_Book)) {Selected Tool, Examples)
Listing 6: Execution end prompt. it is appended to the end of the execution.
you think the current task: (task) already solved, please respond with your conclusion and include the folloving tag at the end:
<iaik_finish>
exit
</task finish>.
Othervise, provide the next command within the appropriate execution tag:
Use <execute_bash>.. </execute_ for Bash commands.
Use Cexecute_python> fereuteeprrios...eeeeeeeeeeeee e/etecute.python>> for my other customized commands, as mentioned in the beginning.
Please don 't take any steps your own that aren NOT related to completing the current task: (task); will guide you through the steps.
You only need to ensure that the current task is completed.
15

---

C Toolkits
C.1
File Edit Toolkit
Listing File edit tool functions
Please the following file editing functions add, delete, search and modify files.
str, content: str I None): Creates and opens a new file with the given name. Add the content to the file if content is not None.
feeate_fieeffleeaaa
replace.content (file_path, cod_enttent, nex_content): Replaces the old content with the new content in the specified file. For the notttetnt argument, please focus only on the parts that actually need to be changed. edit_file(file_am)
start_line: int, start_str: str, end_limg int, end_str: str, content: str): Edits the specified file by replacing the content between start and content. file_name: Name of the file. start_line: Starting line number. start_str: String content in Starting line. end_line: Ending line number. end_str: String content in Ending line. content: New content to replace.
lines with the new
sopend_file (file_name, content, start_line) Appends given content to a file. file_nad : Name of the file. content: Content to append start_line: Line number to start appending from (default is the end of the file).
search_function (file_path, amctiion_sigature)) Search and show function in the file. For the function sigtiiue, you should only input the function name.
C.2 File View Toolkit
Listing 8: File view tool functions
Please use the following functions to understand the content of files. Such reading files, view images, listen to audios, watch videos, etc.
open_file (path: str, Lime_n:mber: int None 1, context_lines: 100) : Opens a file (txt, csv, word, code file, etc.) and optionally moves to a specific line. path: Path to the file. Line_number, Line number to lines display.
to. context_lines. Number of
parse_pdf (pdf str, page: int); View the specified page of a PDF file. pdf_path: Path to the PDF file. page: Page number to view.
parse_figure (figueeeigrr.ptth str): View the specified figure. Higure_path Path the figure file.
parse_audio (auuooaopp str, question: str): Ask a question about the audio file. audio_path: Path to the audio file. question: Question to ask. zoom_pdf
str, page: int, region: tuple): Zoom in specific region of PDF file. pdf_path: Path the PDF file. page: Page number to viev. region: Tuple specifying the region to zoom in on (x0, y0, x1, y1).
C.3 File Search Toolkit
Listing 9: File search tool functions
Please use the following functions to search files.
search dir (search_ term, dir_path=',/") Searches for term in all files in the specified directory.
16

---

find_file (file_name, dir_path='./") ./);: Finds all files with the given in the specified directory.
C.4 Web Browser Toolkit
Listing 10: Web browser tool functions
You can use the following functions to interact with the browser. open_browser (): Open the browser.
navigate_ (url: str) Navigate to the specified URL.
refresh, refresh_page
paageshppaee00 (): Refresh the current page.
go_back(): Go back
the previous page.
go_forwwrdd); forward the next page.
Close the current tab.
oxecute_javsccrpttsestt (script:: str): Execute the specified JavaScript code.
switch_to_ tab tapgg____d int): Switch the tab the specified index.
ezeate__ew_ttabl (url: str): Open new tab with the specified URL.
s Save the current cookies.
dllectddropdown, optioootstlseetttooo (selector indexr int, option: int): Select the specified option from the dropdown menu. selector_index selector index. option: Index of the option to select.
google_search (contentt (content: str): Perform Google search for the specified content. content: The content to search for.
(): Close the browser.
C,5 Computer Use Toolkit
Listing 11: Computer use functions
You can use the following functions to perform various mouse keyboard operations.
clear_text (): Clear the text in the current input field. please make the input field is selected before using this command.
take_screent take_sscrenshot(()
you want use this command to take a screenshot of the current screen.
check the current screen, you can
mouse_lefs leeeleff__clll click (item: str, description: str): Left mouse click at lenseeleft_clickk ('search bar 'It located near the top center of the Google Chrome brouser window. It
the specified position. For example:
long, rectangular input field with rounded edges. The search bar spans almost the entire width of the browser window and sits directly below the browser's 's tab row. It has placeholder text that reads "Search Google search bar centrally aligned, making easy to spot above the main content area of the browser.") couule_dobble_ilee specified position. For example: icon', 'It is located in the sidebar (Launcher) the screen. It the first icon from the top in the vertical arrangement. The icon has a blue background with a white folded "V "-shaped design in the center. The sidebar is aligned along the leftmost edge of the screen, adjacent to the desktop background right side.')
type URL." The
str, description: str): Double-click at the catiseeddubleeeeee click The VSCode
the left side
mouse_right, wouse_right.cciic(iieer click str, description: str): Right mouse click at the specified position. For example: mouse wwiccigttiiick"" refresh button', 'It is located at the top-left corner of the Google Chrome browser window, inside the address bar. It is right navigation
circular arrow icon situated next to the left arrows (back and forward buttons), The refresh button is just to the left of the search bar. Click it to refresh the current page ")

---

mouse_scroll (direction: str, amount: int): Scroll mouse scroll up down. direction: Direction scroll ("up" "down"). amount: Number times to scroll.
str): Type the given text. text: The text to type. dovnload(ur)) str, save_dir: str): If you know the url of the file, you
this command download the file from the specified URL to the specified directory.
press_key (key: str): Presses the specified key. key: The key or key combination to press (e.g., "Return", "Ctrl+c").
open_application (app__amm: str): Opens specific application using the system application launcher. app_name: The name of the application open (e.g., "chrome").
mouse_drag dx__arrttttat int, y_start: int, x_end: int, y_end: int): Drag the mouse from one position to another. x_start: Starting x- coordinate. y_start: Starting y-coordinate. x_end: Ending coordinate. y_end: Ending y-coordinate.
mouse_box_select (x_start: int, y_start: int, x_end: int, y_end: int) Selects a box by dragging the mouse from one position to another x_start: Starting x-coordinate. y_start: Starting y-coordinate. x_end: Ending x-coordinate. y_end: Ending y-coordinate.
C.6 Code Execution Toolkit
Listing 12: Code execution tool
If no suitable command is available, you can also use bash commands to interact with the terminal.
Do not use bash commands to edit/ create the files. Instead, the file editing functions provided.
C.7 Advanced Toolkit
Listing 13: Advanced tools
You can use the following functions to perform advanced operations. compound commands. When available, please
These commands prefer using these commands rather than attempting to perform the operations by yourself.
search_ arxiv (keyword: str, start_dater str, end_date, str, subject: field: str) Searches for papers arliv based the given keyword, date range, subject (options: cs, math, physica, q-bio, q-fin, stat), and keyword field (options: title, abstract, commenta, author, all). Returns the search results.
dounload_arxiv_ (arxiv_id: str) Downloads the specified arliv paper based its ID (eg: (608.06816) and show the first page of the PDF.
scroll_pdf_pegg (direction: str, pages: int): When you're viewing
PDF document in the web interface, you can use this command to scroll the specified page of the PDF file up doun direction: scroll. str); Please use this command to
Direction to scroll ("up" or "down"). page: Page number watch_video (video_path.or__us Watch a video file or YouTube URL. Especially when there is a sign -in prompt for Youtube. sideo_path.or.ur]. Local path or YouTube URL
count_string..n..dee (pdf_path: str, search.string: str): Count the number of occurrences of a specific string in a **Loaa** PDF file pdf oathp Path to the PDF file. search_string, The string to search for.
18

---

D Details of Dedicated Computer
fully isolated execution environment enhances security and enables full-spectrum command execution. Many existing agents (e.g., AutoGPT [18]) run directly on the host machine or run CLI operations within Docker containers. In contrast, our INFANTAGENT- -ENNGEETTT-E provides dedicated, standalone computing environment that supports command-line interaction via Bash, Python scripts via Jupyter, and direct GUI operations through the GNOME desktop. Specifically, to ensure safety and control, the agent operates within isolated virtual machine (VM), preventing direct access to the user's computer and thus eliminating the risk of modifying or deleting critical files. At same time, the user still interact with the VM. The setup consists of three stages: (1) Build base Docker image; (2) Enable GPU-accelerated rendering to support high frame rates and image quality; (3) Expose the VM's GNOME desktop via remote desktop interface. When the agent requires visual feedback (e.g.. to perform a mouse click), the VM captures screenshot of the current display and sends it back the agent. The agent then analyzes the image before issuing the next command.
E
Test Cases of SWE-Bench-Verifted
We detail the test cases of SWE-Bench-Verrine in the following:
astropy-.eetropy-1297
astropy_.astropy--te
astropy_.
astropy-.astropy-
djaggo--..janoo-1455
anagoo__django-111600
django__ dangoo_.da
django_- j 1450-
django__de ango 15930
16032
matplocl1b__natplo1111-2228
matplolllb..nattpot111--22227
matplotiibb.metplccl11-2222
attploollbb..metplool111--12112
mvaskom__seabornn-0000
mwaskon__seaborn-31111
pallets__flask-5001
ref__reqqusts-1112
psf__requests-1766
psf
ruFF_rrequests_1e 1921
ref__request--5414
pydata__.arrryy4007
pydata_..array-6599
pydata_.xarray-6744
pydata__.array-69988
pylint-der--p21in--661
19

---

yy1mtt-or--py11t--499
pylmntdeev_p72in--668
pyeest-ddv--p7est-00001
pyeest-dev.ppyres--5661
pyeest--de...pytest--8899
aciaitllerm__sckkkt--eeear-- 13135
spiinx-doc_.sphnnx-14445
spiinx-doc...pptixx77555
Aphinx--occ..spiixx--0000
ophinx--oc___shinx---50
sympy___8m
sympy__aympy 14531
yympy_.smpy--
syapy_..ympy-1763.
F Limitations and Further Work
Our current work is confined to the reasoning phase. To mitigate the impact of excessive prompt engineering, we will next further train the model automatically invoke the appropriate tools rather than relying on manually added prompts.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract clearly state the claims made, including the contributions made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitation is included in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
This paper is an experiment-driven study.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the code and test scripts are included in the Supplemental Material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
All the code and test scripts are included in the Supplemental Material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips..
oeiiiiiniiii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececccccccccct http:///ggiiieeeeeeeuuuuiiiiii ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have the experiment setup section for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have the error bar for Visual Grounding experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
We mentioned related information in our Visual Grounding experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http///eerrpp.eeeee//ee cc cc/ httppppllcccccccsteeeeeeee
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the related work,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
25

---

Answer: [Yes]
Justification: Our code have the README.nd part.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
26

---

Justification:
We described the usage of LLM.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: : http///rpps.cccooooeereeeeeeeeee for what should or should not be described.
27


Reviews:
[{"review_id": "X3qz9iMLRw", "reviewer": "Reviewer_c5md", "summary": "The paper introduces InfantAgent-Next, a modular multimodal agent that performs real-world computer tasks by combining tool-based and vision-based approaches. It separates the agent workflow into three stages—planning, tool selection, and execution—each handled by specialized models. The system supports text, image, audio, and video inputs and can interact directly with GUIs. Key contributions include a unified framework for multimodal interaction, an iterative visual grounding method for accurate mouse clicks, and a robust file editing strategy.", "questions": "See weakness.", "limitations": "Yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. Experiments cover a wide range of tasks and benchmarks, showing the agent's effectiveness.\n2. Practical improvements such as iterative visual grounding and robust file editing effectively mitigate common failure modes in current agent systems.\n\n**Weaknesses:**\n1. The core ideas seems mostly integration of known components rather than novel contributions.\n2. The iterative visual grounding method for mouse clicking is intuitive and effective in narrowing down regions. However, it assumes that the initial grounding is at least roughly correct. If the initial prediction is wrong, later iterations may focus on irrelevant regions. The paper does not discuss whether the agent can detect failure (e.g., no object found) and roll back or restart from a broader view.", "was_revised": true, "final_justification": "I’m not familiar with agent design, so I can not provide meaningful feedback. But after reviewing the other reviewers’ comments and the authors’ rebuttal, I recognize the value of the paper’s contribution.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. Below, we provide point‑by‑point responses to the comments.\n***********\n**For weakness 1**\n> The core ideas seem mostly integration of known components rather than novel contributions.\n\n**1. Our contributions:**\n\nBased on the NeurIPS 2025 Call for Papers taxonomy, we position InfantAgent-Next in the **Infrastructure category** (e.g., libraries, improvements in implementation and scalability, and distributed solutions). We have two main contributions:\n\n- Infrastructure for computer-use agents. **We present an infrastructure for building multimodal, general-purpose agents with integrated computer-use capability**—i.e., agents that can operate their own computers.\n\n\n- Implementation improvements via unification and interfaces. **We unify tool-based and purely vision-based approaches into a single stack and expose unified interfaces that broaden agents’ computer-use capabilities**. We hope this serves as a versatile reference for the community.\n\n**2. Our novel methods for new challenges**\n\nWhen integrating pure-vision agents with tool‑based agents to build multimodal, general‑purpose agents, we encounter several categories of challenges not present in prior single‑paradigm agents. Below, we describe each challenge in turn and present our novel solutions.\n***********\n***Challenge 1*** — Growth in tool count inflates inference cost.\n\nTraditional tools‑based and pure‑vision agents each expose their own action space. For example, OpenHands (tool‑based) offers file‑editing commands such as `file_edit` and `file_open`, whereas UI‑TARS (pure‑vision) provides mouse/keyboard primitives like `mouse_click` and `press_key`. When these paradigms are fused, we must unify all action spaces and, to handle additional modalities (e.g., audio/video), introduce extra tools. All of these tools—together with usage examples—need to be registered at the start of the dialogue (e.g., in the tool manifest/system prompt), which substantially increases context length and token usage, thereby raising inference cost and latency relative to single‑paradigm agents.\n\n***Our novel method for challenge 1***:\nWe introduce a tool‑selection step  (**Fig. 2** and Lines 109–118): after the planner produces a subtask plan, the agent performs task‑conditioned tool selection, activating only the minimal set of tools needed for the upcoming subtask.\n\nOriginal workflow:\n```\nagent.register_tools()        # registers/activates everything up front\nawait agent.planning()\nawait agent.execution()\n```\nOur workflow:\n```\nawait self.planning() \nawait self.tool_selection() # selects a minimal tool set for this subtask\nawait self.execution()\n```\n\n***Example***\nAs illustrated in **Fig. 5**, the three tool‑selection phases (blue shaded regions) choose different categories of tools conditioned on the current subtask, rather than registering the entire tool inventory to the model. This selective exposure reduces inference cost.\n***********\n***Challenge 2*** — Rising task complexity vs. limited model capacity.\n\nTraditional tool‑based agents typically rely on a single backbone model to handle an entire task—AutoGPT and BabyAGI, for instance, route planning and execution through one (or at most a two‑slot fast/slow) LLM—whereas some vision‑based agents augment the main model with a dedicated GUI visual‑grounding component for pointer‑level localization, as in OS‑Atlas. However, real‑world multimodal agents must operate across text, images, audio, and video. Given this breadth and cross‑domain complexity, a single model—or a small fixed set of models—is often insufficient to deliver accurate and reliable performance.\n\n***Our novel method for challenge 2***:\nWe leverage the agent’s modular design: the end-to-end workflow is decomposed into well-defined modules, and subtasks are routed to specialized models. Difficult, cross-domain steps are assigned to large, high-capability backbones—e.g., we rely on Claude-3.7-Sonnet for global task planning. We assign latency-sensitive operations to lower-latency models; e.g., Qwen2.5-72B-Instruct is used for precise file-editing commands. Vision subtasks go to vision models, and audio subtasks to audio models. This task‑conditioned model routing preserves accuracy while reducing cost and latency. Implementation details are provided in **Fig. 2** and Lines 101–108.\n\n***Example***. In **Fig. 5**, the execution step beneath Turn 1 (purple region) corresponds to image analysis performed by a vision model, whereas the execution steps under Turns 2–3 are carried out by a code‑specialized model.\nThis setup exploits complementary strengths and reduces failures due to modality limits or capability shortfalls.\n\n***********\n***Challenge 3*** — Tool‑use hallucination\n\nEven SOTA models can hallucinate when invoking tools that require precise numeric arguments. Typical failure modes include providing an incorrect line number to the file‑editing tool or wrong coordinates to the visual‑grounding tool. For example, in our tests reported in **Sec. 4.5**, DeepSeek‑V3‑0324 shows a $61\\\\%$ tool‑use hallucination rate on file‑editing commands without any mitigation; in long‑horizon multimodal tasks, such small errors compound and can ultimately lead to end‑to‑end failure.\n\n***Our novel method for challenge 3***: We design two algorithms tailored to these error modes that substantially reduce tool‑use hallucinations.\n\n**Alg. 1** (Iterative Region Cropping & Mouse Click): Repeatedly ground → crop → re-ground to progressively narrow the region before the final click, reducing misclicks without retraining and working with any GUI grounder.\n\n**Alg. 2** (File Editing: Plan-Verify-Repair): Have the model propose edit boundaries with content anchors, verify them against the file, and on mismatched fuzzy re-localize and repair before applying the edit, curbing line-number hallucinations.\n\nsee **Alg. 1** and **Alg. 2** for details.\n\n***Effectiveness of method 3***. In our ablation studies (**Sec. 4.4** and **Sec. 4.5**), **Alg. 1** increases visual‑grounding accuracy on high‑resolution images by $11.6$ percentage points, while **Alg. 2** corrects $84.3\\\\%$ of file‑editing tool sequencing errors.\n\n**For weakness 2**\n\n> The iterative visual grounding method for mouse clicking is intuitive and effective in narrowing down regions. However, it assumes that the initial grounding is at least roughly correct. If the initial prediction is wrong, later iterations may focus on irrelevant regions. The paper does not discuss whether the agent can detect failure (e.g., no object found) and roll back or restart from a broader view.\n\n\nIn practice, initial mis-grounding that could steer later iterations to irrelevant regions is rare: on ScreenSpot‑V2 (a standard‑resolution benchmark), our default GUI‑grounding model UI‑TARS‑1.5‑7B attains $88.7\\\\%$ accuracy; adding our iterative narrowing raises it to $94.6\\\\%$. Beyond accuracy, InfantAgent‑Next integrates tool‑based and vision‑only capabilities, so when GUI grounding is uncertain the agent can fall back to non‑visual action paths (e.g., terminal commands or Python scripts) to complete the task. \n\nHere is an example from OSWorld benchmark (id: 3aaa4e37-dc91-482e-99af-132a612d40f3):\n\nQuestion: *Could you help me to export the current sheet to a csv file? Export the contents just as they are shown on the screen. Just keep the other options untouched. A default csv format is ok. The csv should share the file name with the original xlsx.*\n\nAgent's actions: First, the agent searches for the `Save As` button and the `CSV` format. Finding no `CSV` option on screen, it switches strategy and executes an one‑line Bash command, which resolves the question.\n***********\nWe once again thank the reviewer for the thoughtful comments. We hope our responses address your concerns, and we would be glad to continue the discussion should any further questions arise."}, {"review_id": "4fJOXXBK6s", "reviewer": "Reviewer_31d8", "summary": "This paper proposes a modular GUI‐agent framework, InfantAgent-Next, that uses vision and tools (e.g., computer operations) to interact with GUI. The method uses UI-TARS-1.5-7B for visual grounding and Claude-3.7-Sonnet for planning. Its action space includes several predefined tools plus mouse‐click actions commonly used by pure‐vision agents. Results show that InfantAgent-Next was the best open‐source agent framework on OSWorld at the time.", "questions": "1. In terms of framework design, what are the main differences between your proposed framework and existing modular GUI agents such as MobileExperts, InfiGUIAgent, and AGENT S?\n2. Have you tried using different backbone models?\n3. I would like to see more examples like those in Figure 5, including both successful and failed cases, to better understand how the system works.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n* Achieves competitive performance on the OSWorld, SWE, and GAIA leaderboards.\n* Provides a detailed description of the proposed method.\n* Conducts experiments across multiple GUI‐agent benchmarks.\n\nWeakness\n* The modular framework has little significant difference from existing modular GUI‐agent frameworks.", "was_revised": true, "final_justification": "This paper proposes a new GUI-agent framework and, although it does not achieve state-of-the-art performance, has promising results. The framework integrates existing approaches without introducing significant novelty, but the experiments and case studies are comprehensive and convincing.", "rebuttal": "Thank you for your positive assessment of our work! Below, we provide point‑by‑point responses to the comments.\n***********\n**For weakness 1 and question 1**\n> In terms of framework design, what are the main differences between your proposed framework and existing modular GUI agents such as MobileExperts, InfiGUIAgent, and AGENT S?\n\nInfantAgent‑Next is a **hybrid framework that unifies tool‑based and pure‑vision agents**. Rather than relying solely on mouse/keyboard GUI manipulation, the agent exposes a broader action space under a unified interface—issuing shell commands, executing Python code, and performing mouse/keyboard actions. A planner chooses among these paths and can fall back to non‑visual execution when GUI grounding is uncertain. This design makes the system applicable to more complex workflows.\n\nFor example, when the agent is asked to process a CSV file, a pure GUI agent would require a long sequence of clicks and drags, where a single grounding error can derail the task; by contrast, InfantAgent‑Next can simply write and run a short Python script to transform and process the CSV file, substantially improving robustness and end‑to‑end success.\n\n**For weakness 2**\n> Have you tried using different backbone models?\n\nWe evaluated three backbone models: Claude‑3.7‑Sonnet, GPT‑4o, and DeepSeek‑V3‑0324. Because DeepSeek‑V3‑0324 lacks multimodal ability, we used it exclusively on the SWE‑Bench test. Since agent performance is strongly coupled with the backbone model’s capability, in our main comparisons against SOTA agents we report results using the strongest backbone per setting. \n\nBelow we provide our measured results for GPT‑4o and DeepSeek‑V3‑0324:\n\nTable 1: SWE‑bench‑Verified test results.\n| Agent            | Model            | Accuracy |\n|:----------------:|:----------------:|:-------:|\n| OpenHands        | DeepSeek-V3-0324 |    $38.8\\\\%$ |\n| Agentless        | DeepSeek-V3-0324 |    $35.8\\\\%$ |\n| AutoCodeRover    | DeepSeek-V3-0324 |    $34.8\\\\%$ |\n| **InfantAgent-Next** | DeepSeek-V3-0324 |    $37.2\\\\%$ |\n| OpenHands        | GPT-4o           |   $41.20\\\\%$ |\n| Agentless        | GPT-4o           |   $38.80\\\\%$ |\n| AutoCodeRover    | GPT-4o           |   $39.20\\\\%$ |\n| **InfantAgent-Next** | GPT-4o           |   $40.20\\\\%$ |\n\nTable 2: OSWorld test results using GPT‑4o as the backbone model (maximum 50 steps per task).\n\n| Agent            | Accuracy |\n|:----------------:|:-------:|\n| Agent S          |   $20.58\\\\%$ |\n| OpenAI CUA       |    $32.6\\\\%$ |\n| Agent S2         |   $27.91\\\\%$ |\n| Aguvis           |   $17.04\\\\%$ |\n| **InfantAgent-Next** |   $30.35\\\\%$ |\n\n**For weakness 3**\n> I would like to see more examples like those in **Fig. 5**, including both successful and failed cases, to better understand how the system works.\n\nHere, we present two additional examples to detail the problem‑solving workflow of InfantAgent‑Next: \n\nLegend: italic text: agent‑generated output; ⚠️:annotation for ease of understanding.\n\n************\n\n***1. Positive example from GAIA (task ID: e1fc63a2-da7a-432f-be78-7c4a95598703).***\n\nQuestion: If Eliud Kipchoge could maintain his record-making marathon pace indefinitely, how many thousand hours would it take him to run the distance between the Earth and the Moon its closest approach? Please use the minimum perigee value on the Wikipedia page for the Moon when carrying out your calculation. Round your result to the nearest 1000 hours and do not use any comma separators if necessary.\n\n****** **Agent Start to Work** ******\n\n**1. Assign task 1:**\n\n*Thought: First, I need to find the minimum perigee distance between Earth and Moon from Wikipedia.*\n\n*Task: Search on Wikipedia for the Moon page and find the minimum perigee (closest approach) distance between Earth and Moon.*\n\n**2. Start to finish task 1**\n\n*Tool Selection: browser* ⚠️ **select the suitable tools from toolkits.**\n\n*Thought: I'll navigate to the Wikipedia page for the Moon to find information.*\n\n*Action: open_browser(), navigate_to(en.wikipedia.org/wiki/Moon)*\n\n*Observation: Screenshot saved at /workspace/screenshots/1744924507.png*  ⚠️ **Agent can see this screenshot.**\n\n**3.  Assign task 2:**\n\n*Thought: Now I need to find Eliud Kipchoge's record marathon pace and calculate how long it would take him to run the distance to the Moon at that pace.*\n\n*Task: Search related information*\n\n**4. Start to finish task 2**\n\n*Tool Selection: browser*\n\n*Thought: I’ll search for Eliud Kipchoge's marathon world record time and pace.*\n\n*Action: google_search(\\\"Eliud Kipchoge marathon world record pace\\\")*\n\n*Observation: Screenshot saved at /workspace/screenshots/1744924600.png*\n\n**5. Assign task 3:**\n\n*Task: Calculate the final answer*\n\n**6. Start to finish task 3:**\n\n*Tool Selection: python* \n\n*Thought:  Now I have all the information, Let me perform the calculation*\n\n*Action: # python code to calculate the final answer, too long, omitted here.*\n\n*Observation: Final Answer: 17* ⚠️ **python code results.**\n\n**7. Provide final answer:**\n\n*Based on our calculations, I can provide the final answer in the requested format. FINAL ANSWER: 17*\n\n************\n\n***2. Negative example from SWE‑bench (task ID: astropy_astropy-7606).***\n\nQuestion: \nUnit equality comparison with None raises TypeError for UnrecognizedUnit\n```\nIn [12]: x = u.Unit('asdf', parse_strict='silent')\n\nIn [13]: x == None  # Should be False\n---------------------------------------------------------------------------\nTypeError                                 Traceback (most recent call last)\n<ipython-input-13-2486f2ccf928> in <module>()\n----> 1 x == None  # Should be False\n\n/Users/aldcroft/anaconda3/lib/python3.5/site-packages/astropy/units/core.py in __eq__(self, other)\n  1699\n  1700     def __eq__(self, other):\n-> 1701         other = Unit(other, parse_strict='silent')\n  1702         return isinstance(other, UnrecognizedUnit) and self.name == other.name\n  1703\n\n/Users/aldcroft/anaconda3/lib/python3.5/site-packages/astropy/units/core.py in __call__(self, s, represents, format, namespace, doc, parse_strict)\n  1808\n  1809         elif s is None:\n-> 1810             raise TypeError(\"None is not a valid Unit\")\n  1811\n  1812         else:\n\nTypeError: None is not a valid Unit\n```\n\n****** **Agent Start to Work** ******\n\n**1. Assign task 1:  ✔️**\n\n*Thought: It appears that the file \"core.py\" used in the traceback was found under a directory path that includes \"units\"*\n\n*Task: Locate the core.py file mentioned in the error traceback to understand the issue*\n\n**2. Start to finish task 1: ✔️**\n\n*Tool Selection: Bash*\n\n*Thought: Let's execute the Bash commands to display the content around \"None is not a valid Unit\" in core.py:*\n\n*Action: # Bash commands.*\n\n*Observation: # Bash commands results*\n\n**3. Assign task 2: ✔️**\n\n*Thought: We should first reproduce the issue.*\n\n*Task: Create a test script to reproduce the issue with UnrecognizedUnit equality comparison with None.*\n\n**4. Start to finish task 2: ✔️**\n\n*Tool Selection: Bash*\n\n*Thought: Based on the repository exploration, I'll create a minimal script that demonstrates the problem.*\n\n*Action: # Bash commands to write the code into the file and run the code.*\n\n*Observation: # Test script results.*\n\n**5. Assign task 3: ✔️**\n\n*Thought: We should implement the fix for the issue where UnrecognizedUnit equality comparison with None raises TypeError.*\n\n*Task: We need to modify the `__eq__` method in the UnrecognizedUnit class to handle None as a special case.* \n\n**6. Start to finish task 3: ❌**\n\n*Tool Selection: Bash* \n\n*Thought: Let's apply the sed command to insert the special case for None into the `__eq__` method:*\n\n*Action: # Bash commands to modify the code.*  ⚠️ **The modification does not handle certain edge cases.**\n\n*Observation: # Modified code*\n\n**7. Assign task 4: ✔️**\n\n*Thought: We need to verify that the code we modified is correct.*\n\n*Task: Let’s run the test script again.*\n\n**8. Start to finish task 4: ✔️**\n\n*Tool Selection: Bash* \n\n*Action: # Bash commands to run the code*\n\n*Observation: #  Test script results.*\n\n**9. Provide final answer: ✔️**\n\n*ANSWER: All tests have passed successfully, confirming that our fix resolves the issue without affecting any existing functionality in the test suite. Please check `/workspace` to see the patch.*\n\n***********\nWe once again thank the reviewer for the thoughtful comments. We hope our responses address your concerns, and we would be glad to continue the discussion should any further questions arise."}, {"review_id": "pmU7W0WPsn", "reviewer": "Reviewer_LrsF", "summary": "This paper introduced a generalist agent framework named InfantAgent-Next. The framework adopts a multi-model collaboration architecture, where models with different modalities and specializations are assigned to various tasks (such as planning, tool selection, and visual grounding) to achieve optimization of overall capabilities. Experiments demonstrate that InfantAgent-Next achieves promising results across multiple types of tasks, including computer interaction (OSWorld), real-world question answering (GAIA), and coding (SWE-Bench).", "questions": "1. Could the author elaborate more on the methodological innovation and contributions? While engineering optimization is crucial for building such an agent system, as an academic paper, we would value an analysis of the underlying principles and methodological advancements.\n\n2. In Table 1, the author compares the proposed method with existing works from the perspective of modular design. Could the author provide an ablation analysis of all these modules in InfantAgent-Next?\n\n3. InfantAgent-Next is designed to be model-agnostic and flexible for use with various LLMs/vLLMs. The experiments only tested Claude-3.7-Sonnet and UI-TARS-1.5-7B—how significantly would performance change with different models?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\n1. The proposed agent system is evaluated on multiple benchmarks and achieves promising results on tasks such as computer interaction and real-world problem-solving. This demonstrates the versatility of the proposed method. And I believe it can be readily extended to a broader range of scenarios and tasks, considering its flexibility in integrating diverse models and tools.\n\n# Weaknesses\n\n1. This work is more of an integration of existing technologies. The design of the agent system's architecture and workflows does not exhibit significant innovations compared to existing methods; the design of core modules, such as planning, visual grounding, tool use, and context management, is also not original.\n\n2. I also have concerns about the fairness of the experiment results. Considering that the contribution of this work lies in the agent architecture rather than the model itself, the comparison should focus on the agent design. However, the baseline methods in table 2-4 are based on different models, some of which have significant gaps in parameter size and model capability.", "was_revised": true, "final_justification": "This paper introduced InfantAgent-Next, a generalist computer-use agent that achieves strong performance on multiple benchmarks. Although this work lacks remarkable innovation in methodology, it offers engineering insights for building efficient computer-use agents by integrating multimodal models and technologies like tool selection and memory retrieval. I believe this work aligns with the infrastructure category of the call-for-papers and will benifit future research.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. Below, we provide point‑by‑point responses to the comments.\n***********\n**For weakness 1 and Question 1:**\n> Could the author elaborate more on the methodological innovation and contributions? \n\n**1. Our contributions:**\n\nBased on the NeurIPS 2025 Call for Papers taxonomy, we position InfantAgent-Next in the **Infrastructure category** (e.g., libraries, improvements in implementation and scalability, and distributed solutions). We have two main contributions:\n\n- Infrastructure for computer-use agents. **We present an infrastructure for building multimodal, general-purpose agents with integrated computer-use capability**—i.e., agents that can operate their own computers.\n\n\n- Implementation improvements via unification and interfaces. **We unify tool-based and purely vision-based approaches into a single stack and expose unified interfaces that broaden agents’ computer-use capabilities**. We hope this serves as a versatile reference for the community.\n\n**2. Our novel methods for new challenges**\n\nWhen integrating pure-vision agents with tool‑based agents to build multimodal, general‑purpose agents, we encounter several categories of challenges not present in prior single‑paradigm agents. Below, we describe each challenge in turn and present our novel solutions.\n***********\n***Challenge 1*** — Growth in tool count inflates inference cost.\n\nTraditional tools‑based and pure‑vision agents each expose their own action space. For example, OpenHands (tool‑based) offers file‑editing commands such as `file_edit` and `file_open`, whereas UI‑TARS (pure‑vision) provides mouse/keyboard primitives like `mouse_click`. When these paradigms are fused, we must unify all action spaces and, to handle additional modalities (e.g., audio/video), introduce extra tools. All of these tools—together with usage examples—need to be registered at the start of the dialogue (e.g., in the tool manifest/system prompt), which substantially increases context length and token usage, thereby raising inference cost and latency relative to single‑paradigm agents.\n\n***Our novel method for challenge 1***:\nWe introduce a tool‑selection step  (**Fig. 2** and Lines 109–118): after the planner produces a subtask plan, the agent performs task‑conditioned tool selection, activating only the minimal set of tools needed for the upcoming subtask.\n\nOriginal workflow:\n```\nagent.register_tools()        # registers/activates everything up front\nawait agent.planning()\nawait agent.execution()\n```\nOur workflow:\n```\nawait self.planning() \nawait self.tool_selection() # selects a minimal tool set for this subtask\nawait self.execution()\n```\n\n***Example***\nAs illustrated in **Fig. 5**, the three tool‑selection phases (blue shaded regions) choose different categories of tools conditioned on the current subtask, rather than registering the entire tool inventory to the model. This selective exposure reduces inference cost.\n***********\n***Challenge 2*** — Rising task complexity vs. limited model capacity.\n\nMost tool-based agents run end-to-end on a single LLM (e.g., AutoGPT, BabyAGI), and some vision-based agents add a GUI visual-grounding module for pointer actions (e.g., OS-Atlas). However, real-world multimodal tasks span text, images, audio, and video, so a single—or small fixed set of—models rarely delivers accurate, reliable performance.\n\n***Our novel method for challenge 2***:\nWe leverage the agent’s modular design: the end-to-end workflow is decomposed into well-defined modules, and subtasks are routed to specialized models. Difficult, cross-domain steps are assigned to large, high-capability backbones. We assign latency-sensitive operations to lower-latency models. Vision subtasks go to vision models, and audio subtasks to audio models. This task‑conditioned model routing preserves accuracy while reducing cost and latency. Implementation details are provided in **Fig. 2** and Lines 101–108.\n\n***Example***. In **Fig. 5**, the execution step beneath Turn 1 (purple region) corresponds to image analysis performed by a vision model, whereas the execution steps under Turns 2–3 are carried out by a code‑specialized model.\nThis setup exploits complementary strengths and reduces failures due to modality limits or capability shortfalls.\n\n***********\n***Challenge 3*** — Tool‑use hallucination\n\nEven SOTA models can hallucinate when invoking tools that require precise numeric arguments. Typical failure modes include providing an incorrect line number to the file‑editing tool or wrong coordinates to the visual‑grounding tool. For example, in our tests reported in **Sec. 4.5**, DeepSeek‑V3‑0324 shows a $61\\\\%$ tool‑use hallucination rate on file‑editing commands without any mitigation; in long‑horizon multimodal tasks, such small errors compound and can ultimately lead to end‑to‑end failure.\n\n***Our novel method for challenge 3***: We design two algorithms tailored to these error modes that substantially reduce tool‑use hallucinations.\n\n**Alg. 1** (Iterative Region Cropping & Mouse Click): Repeatedly ground → crop → re-ground to progressively narrow the region before the final click, reducing misclicks without retraining and working with any GUI grounder.\n\n**Alg. 2** (File Editing: Plan-Verify-Repair): Have the model propose edit boundaries with content anchors, verify them against the file, and on mismatched fuzzy re-localize and repair before applying the edit, curbing line-number hallucinations.\n\nsee **Alg. 1** and **Alg. 2** for details.\n\n***Effectiveness of method 3***. In our ablation studies (**Sec. 4.4** and **Sec. 4.5**), **Alg. 1** increases visual‑grounding accuracy on high‑resolution images by $11.6$ percentage points, while **Alg. 2** corrects $84.3\\\\%$ of file‑editing tool sequencing errors.\n\n**For weakness 2:** \n> ….some of which have significant gaps in parameter size and model capability.\n\nIn **Table 2–4**, we present comparisons between InfantAgent‑Next and existing agents, with the aim of demonstrating competitive performance across a range of tasks.\n\nFor several baselines, using a single, unified backbone is infeasible. For example, in **Table 2**, UI‑TARS reports results with its GUI‑specialized model (UI‑TARS‑1.5‑72B), rather than a public backbone. In **Table 4**, OWL processes webpages by converting them to HTML, whereas our framework operates directly on screenshots. Consequently, a text‑only backbone without visual capability is incompatible with our setting and would substantially degrade InfantAgent‑Next’s performance.\n\nFor baselines where a unified backbone is feasible, we provide comparisons under the same model. For example, in **Table 2** we match AgentS2 by using the same model and enforcing the same step budget; in **Table 4**, we likewise compare with TapeAgents/Auto‑Deep‑Research under the same model. This controls for backbone effects and isolates the contribution of our framework.\n\n**For Question 2**\n\n> ... Could the author provide an ablation analysis of all these modules in InfantAgent-Next?\n\n**Table 1** contrasts InfantAgent-Next with prior agents across eight dimensions. We do not ablate four of them—Generalist Agent, Multi-Model Support, User Interaction during Execution, and Dedicated Computer—because they are problem-setting assumptions, not pluggable modules; toggling them would alter the evaluation protocol.\n\nAccordingly, we ablate the remaining modules by disabling each in turn within InfantAgent-Next and evaluating 50 GAIA cases, reporting success rate and per-task cost; Claude-3.7-Sonnet serves as the backbone.\n\n| Ablation             | Success rate (%) | Average cost per task ($) |Analysis of Results   |\n|:--------------------:|:----------------:|:--------------------------:|:-----------------------:|\n| Full                 | $72$               | $2.77$                       | baseline                |\n| Visual Grounding off | $44$               | $2.43$                       | pointer tasks drop      |\n| Tool off             | $12$               | $2.15$                       | File operation fail     |\n| Memory Retrieval off | $78$               | $3.89$                       | Inference cost increase |\n| Dynamic Toolset off  | $72$               | $4.54$                       | Inference cost increase |\n\n**For Question 3:**\n> ... how significantly would performance change with different models?\n\nWe evaluated three backbone models: Claude‑3.7‑Sonnet, GPT‑4o, and DeepSeek‑V3‑0324. Because DeepSeek‑V3‑0324 lacks multimodal ability, we used it exclusively on the SWE‑Bench test. Since agent performance is strongly coupled with the backbone model’s capability, in our main comparisons against SOTA agents we report results using the strongest backbone per setting. \n\nBelow we provide our measured results for GPT‑4o and DeepSeek‑V3‑0324:\n\n**SWE‑bench‑Verified test results.**\n| Agent            | Model            | Accuracy |\n|:----------------:|:----------------:|:-------:|\n| OpenHands        | DeepSeek-V3-0324 |    $38.8\\\\%$ |\n| Agentless        | DeepSeek-V3-0324 |    $35.8\\\\%$ |\n| AutoCodeRover    | DeepSeek-V3-0324 |    $34.8\\\\%$ |\n| **InfantAgent-Next** | DeepSeek-V3-0324 |    $37.2\\\\%$ |\n| OpenHands        | GPT-4o           |   $41.20\\\\%$ |\n| Agentless        | GPT-4o           |   $38.80\\\\%$ |\n| AutoCodeRover    | GPT-4o           |   $39.20\\\\%$ |\n| **InfantAgent-Next** | GPT-4o           |   $40.20\\\\%$ |\n\n**OSWorld test results using GPT‑4o as the backbone model (maximum 50 steps per task).**\n\n| Agent            | Accuracy |\n|:----------------:|:-------:|\n| Agent S          |   $20.58\\\\%$ |\n| OpenAI CUA       |    $32.6\\\\%$ |\n| Agent S2         |   $27.91\\\\%$ |\n| Aguvis           |   $17.04\\\\%$ |\n| **InfantAgent-Next** |   $30.35\\\\%$ |\n\n***********\nWe once again thank the reviewer for the thoughtful comments. We hope our responses address your concerns, and we would be glad to continue the discussion should any further questions arise."}, {"review_id": "FDSZ9ggBDH", "reviewer": "Reviewer_x34C", "summary": "The paper introduces INFANTAGENT-NEXT, a multimodal generalist agent designed for automated computer interaction, capable of handling text, images, audio, and video inputs. It proposes a modular architecture that integrates tool-based and vision-based approaches, routing subtasks to specialized models (e.g., reasoning, visual grounding, audio analysis) within a unified dialogue context. The agent achieves strong performance across diverse benchmarks, including OSWorld (7.27% accuracy, surpassing Claude-Computer-Use), SWE-Bench (66% on a verified subset), and GAIA (second among open-source agents on Level 2 tasks). The work includes open-source code and evaluation scripts, emphasizing reproducibility and modularity for tasks like file editing, web browsing, and GUI interaction.", "questions": "1. **Failure Case Analysis**: The paper mentions minor limitations in file editing edge cases (e.g., 9.6% failure rate). Could the authors provide a detailed analysis of these failures, including specific examples and potential improvements? This could strengthen the quality score by demonstrating robustness.\n2. **Scalability and Cost**: The paper uses resource-intensive models (e.g., Claude-3.7-Sonnet, A100 GPUs). How does INFANTAGENT-NEXT scale for real-world deployment, particularly in terms of computational cost and latency? Addressing this could elevate the significance score.\n3. **Novelty of Workflow**: The modular workflow is effective but builds on existing paradigms. Can the authors clarify what aspects of the workflow (beyond integration) are truly novel? A stronger case for originality could improve the originality score.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- **Quality**: The paper presents a robust engineering effort with a well-designed modular workflow that effectively combines tool-based and vision-based paradigms. Extensive evaluations across OSWorld, SWE-Bench, and GAIA demonstrate strong empirical performance, with INFANTAGENT-NEXT outperforming notable baselines like Claude-Computer-Use and achieving competitive results among open-source agents. The inclusion of detailed algorithms (e.g., Iterative Region Cropping, File Editing Logic) and open-source code enhances reproducibility.\n- **Clarity**: The paper is clearly written, with a structured breakdown of the architecture, components, and experimental setups. Figures and case studies (e.g., Figure 2 and 5) effectively illustrate the workflow, and the appendix provides comprehensive details on prompts, toolkits, and test cases.\n- **Significance**: The work is highly significant for the field of automated AI agents, addressing practical challenges in multimodal computer interaction. Its broad applicability across GUI tasks, coding, and general tool-use benchmarks makes it a valuable contribution to real-world agent development.\n\n**Weaknesses:**\n- **Originality**: While the engineering is impressive, the paper lacks novel theoretical insights or fundamentally new paradigms in agent design. The modular architecture, while effective, builds on existing concepts (e.g., tool-based and vision-based agents) without introducing groundbreaking ideas.\n- **Quality (Limitation)**: The evaluation, though comprehensive, could benefit from deeper analysis of failure cases or edge scenarios to better understand the agent’s limitations. For instance, the paper notes minor limitations in file editing edge cases but does not explore these systematically.\n- **Significance (Scope)**: The work focuses heavily on performance metrics but provides limited discussion on scalability or real-world deployment challenges, such as computational costs for large-scale use or robustness in dynamic environments.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our work! Below, we provide point‑by‑point responses to the comments.\n***********\n**For weakness 1 and Question 3:**\n\n> the paper lacks novel theoretical insights or fundamentally new paradigms in agent design.\nWhen integrating pure-vision agents with tool‑based agents to build multimodal, general‑purpose agents, we encounter several categories of challenges not present in prior single‑paradigm agents. Below, we describe each challenge in turn and present our novel solutions.\n***********\n***Challenge 1*** — Growth in tool count inflates inference cost.\n\nTraditional tools‑based and pure‑vision agents each expose their own action space. For example, OpenHands (tool‑based) offers file‑editing commands such as `file_edit` and `file_open`, whereas UI‑TARS (pure‑vision) provides mouse/keyboard primitives like `mouse_click` and `press_key`. When these paradigms are fused, we must unify all action spaces and, to handle additional modalities (e.g., audio/video), introduce extra tools. All of these tools—together with usage examples—need to be registered at the start of the dialogue (e.g., in the tool manifest/system prompt), which substantially increases context length and token usage, thereby raising inference cost and latency relative to single‑paradigm agents.\n\n***Our novel method for challenge 1***:\nWe introduce a tool‑selection step  (**Fig. 2** and Lines 109–118): after the planner produces a subtask plan, the agent performs task‑conditioned tool selection, activating only the minimal set of tools needed for the upcoming subtask.\n\nOriginal workflow:\n```\nagent.register_tools()        # registers/activates everything up front\nawait agent.planning()\nawait agent.execution()\n```\nOur workflow:\n```\nawait self.planning() \nawait self.tool_selection() # selects a minimal tool set for this subtask\nawait self.execution()\n```\n\n***Example***\nAs illustrated in **Fig. 5**, the three tool‑selection phases (blue shaded regions) choose different categories of tools conditioned on the current subtask, rather than registering the entire tool inventory to the model. This selective exposure reduces inference cost.\n***********\n***Challenge 2*** — Rising task complexity vs. limited model capacity.\n\nTraditional tool‑based agents typically rely on a single backbone model to handle an entire task—AutoGPT and BabyAGI, for instance, route planning and execution through one (or at most a two‑slot fast/slow) LLM—whereas some vision‑based agents augment the main model with a dedicated GUI visual‑grounding component for pointer‑level localization, as in OS‑Atlas. However, real‑world multimodal agents must operate across text, images, audio, and video. Given this breadth and cross‑domain complexity, a single model—or a small fixed set of models—is often insufficient to deliver accurate and reliable performance.\n\n***Our novel method for challenge 2***:\nWe leverage the agent’s modular design: the end-to-end workflow is decomposed into well-defined modules, and subtasks are routed to specialized models. Difficult, cross-domain steps are assigned to large, high-capability backbones—e.g., we rely on Claude-3.7-Sonnet for global task planning. We assign latency-sensitive operations to lower-latency models; e.g., Qwen2.5-72B-Instruct is used for precise file-editing commands. Vision subtasks go to vision models, and audio subtasks to audio models. This task‑conditioned model routing preserves accuracy while reducing cost and latency. Implementation details are provided in **Fig. 2** and Lines 101–108.\n\n***Example***. In **Fig. 5**, the execution step beneath Turn 1 (purple region) corresponds to image analysis performed by a vision model, whereas the execution steps under Turns 2–3 are carried out by a code‑specialized model.\nThis setup exploits complementary strengths and reduces failures due to modality limits or capability shortfalls.\n\n***********\n***Challenge 3*** — Tool‑use hallucination\n\nEven SOTA models can hallucinate when invoking tools that require precise numeric arguments. Typical failure modes include providing an incorrect line number to the file‑editing tool or wrong coordinates to the visual‑grounding tool. For example, in our tests reported in **Sec. 4.5**, DeepSeek‑V3‑0324 shows a $61\\\\%$ tool‑use hallucination rate on file‑editing commands without any mitigation; in long‑horizon multimodal tasks, such small errors compound and can ultimately lead to end‑to‑end failure.\n\n***Our novel method for challenge 3***: We design two algorithms tailored to these error modes that substantially reduce tool‑use hallucinations.\n\n**Alg. 1** (Iterative Region Cropping & Mouse Click): Repeatedly ground → crop → re-ground to progressively narrow the region before the final click, reducing misclicks without retraining and working with any GUI grounder.\n\n**Alg. 2** (File Editing: Plan-Verify-Repair): Have the model propose edit boundaries with content anchors, verify them against the file, and on mismatched fuzzy re-localize and repair before applying the edit, curbing line-number hallucinations.\n\nsee **Alg. 1** and **Alg. 2** for details.\n\n***Effectiveness of method 3***. In our ablation studies (**Sec. 4.4** and **Sec. 4.5**), **Alg. 1** increases visual‑grounding accuracy on high‑resolution images by $11.6$ percentage points, while **Alg. 2** corrects $84.3\\\\%$ of file‑editing tool sequencing errors.\n\n**For weakness 2 and Question 1:** \n\n> The paper mentions minor limitations in file editing edge cases (e.g., 9.6% failure rate). Could the authors provide a detailed analysis of these failures, including specific examples and potential improvements? This could strengthen the quality score by demonstrating robustness.\n\nTo mitigate hallucinations that arise when the model uses the file‑editing tool, we introduce **Alg. 2** and analyze its effectiveness in correcting different error types in **Sec. 4.5**; the results are visualized in **Fig. 4**.\nHere we will show a specific example:\n\nTask: Edit on `astropy/wcs/wcs.py`, Agent is trying to add a debug statement between line 1249 and line 1250.\n\n```\n ... (1242 more lines above)\n1246 | if len(args) == 2:\n1247 |     try:\n1248 |         xy, origin = args\n1249 |         xy = np.asarray(xy)\n1250 |         origin = int(origin)\n1251 |         if xy.size == 0:\n1252 |             return np.array([])\n... (2040 more lines below)\n```\n\n********** **Original Incorrect Action** ********** ❌\n```\nedit_file(astropy/wcs/wcs.py, \nstart_line = 1249,\nend_line = 1249,   # ⚠️ This line number is wrong, should be 1250.\ncontent = \"         print(f'xy:{xy}')\")\n```\n********** **Our method** ********** ✔️\n\nStep 1: The model is asked to produce the content corresponding to the line it plans to edit.\n```\nedit_file(astropy/wcs/wcs.py, \nstart_line = 1249, \nstart_line_str = 'xy = np.asarray(xy)' \nend_line = 1249,   # ⚠️ if this line number is not correct\nend_line_str = 'origin = int(origin)' # ⚠️ the content will not match\ncontent = \"         print(f'xy:{xy}')\")\n```\n\nStep 2: Upon detecting a mismatch between the model‑generated content and the actual content at line 1249, we display a diagnostic message:\n\n`The string: origin = int(origin) does not match the end line 1249 The end line: 1249 is: xy = np.asarray(xy)`\n\nStep 3: The model regenerates the command:\n```\nedit_file(astropy/wcs/wcs.py, \nstart_line = 1249, \nstart_line_str = 'xy = np.asarray(xy)' \nend_line = 1250,  # Successfully corrected the line number. ✔️\nend_line_str = 'origin = int(origin)' \ncontent = \"         print(f'xy:{xy}')\")\n```\n\n**For weakness 3 and Question 2:**\n> How does INFANTAGENT-NEXT scale for real-world deployment, particularly in terms of computational cost and latency? Addressing this could elevate the significance score.\n\nWe are actively deploying the system in real settings. Due to space constraints, we summarize two practical tactics we use to reduce computational cost at scale:\n\n1. **Preserve the KV cache.** Keep the prompt/tool schema as stable as possible so cached attention states remain reusable across turns. On hosted APIs (e.g., OpenAI/Claude), “cached tokens” are typically billed at ≈$25$–$50\\\\%$ of the price of new tokens; on self‑hosted models, retaining the KV/attention cache also yields substantial latency speedups. **Concretely, we mask tools that are not needed in a given step rather than deleting them** (and we keep tool order and identifiers fixed), which maintains cache compatibility from one turn to the next.\n\n2. **Offload verbose tool outputs to files (workspace artifacts)**. Tool calls often return long texts. Instead of appending the entire output to the chat history, we write it to a file in the agent’s workspace and refer to it by a short handle (path/hash) and a brief summary. The agent re‑opens the file on demand when needed. This keeps the conversation context short, reducing both token cost and inference latency.\n\n***********\nWe once again thank the reviewer for the thoughtful comments. We hope our responses address your concerns, and we would be glad to continue the discussion should any further questions arise."}]

Author rebuttal:

Meta-review: The paper introduces InfantAgent-Next, a modular multimodal agent integrating tool-based and vision-based approaches for real-world computer use, with strong results on OSWorld, SWE-Bench, and GAIA. Strengths: clear writing, thorough experiments, practical engineering (tool selection, modular routing, hallucination reduction), and open-source release. Weaknesses: limited novelty, integration of known methods, some fairness and scalability concerns. Despite modest originality, the work makes solid infrastructure contributions with strong empirical value. In rebuttal, authors clarified contributions, added ablations and examples, and addressed fairness/cost concerns; most reviewers raised scores, while one kept accept but noted limited novelty.
Decision: Accept (poster)