Title: InfantAgent-Next: A Multimodal Generalist Agent for Automated Computer Interaction
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM based Agent

Multimodal Generalist Agent

Automated Computer Interaction

Infrastructure
Abstract: This paper introduces \textsc{InfantAgent-Next}, a generalist agent capable of interacting with computers in a multimodal manner, encompassing text, images, audio, and video.
Unlike existing approaches that either build intricate workflows around a single large model or only provide workflow modularity, our agent integrates tool-based and pure vision agents within a highly modular architecture, enabling different models to collaboratively solve decoupled tasks in a step-by-step manner. 
Our generality is demonstrated by our ability to evaluate not only pure vision-based real-world benchmarks (i.e., OSWorld), but also more general or tool-intensive benchmarks (e.g., GAIA and SWE-Bench).
Specifically,
we
achieve a $\mathbf{7.27\\%}$ accuracy gain over Claude-Computer-Use on OSWorld.  
Codes and evaluation scripts are included in the supplementary material and will be released as open-source.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces INFANTAGENT-NEXT, a multimodal generalist agent designed for automated computer interaction, capable of handling text, images, audio, and video inputs. It proposes a modular architecture that integrates tool-based and vision-based approaches, routing subtasks to specialized models (e.g., reasoning, visual grounding, audio analysis) within a unified dialogue context. The agent achieves strong performance across diverse benchmarks, including OSWorld (7.27% accuracy, surpassing Claude-Computer-Use), SWE-Bench (66% on a verified subset), and GAIA (second among open-source agents on Level 2 tasks). The work includes open-source code and evaluation scripts, emphasizing reproducibility and modularity for tasks like file editing, web browsing, and GUI interaction.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- **Quality**: The paper presents a robust engineering effort with a well-designed modular workflow that effectively combines tool-based and vision-based paradigms. Extensive evaluations across OSWorld, SWE-Bench, and GAIA demonstrate strong empirical performance, with INFANTAGENT-NEXT outperforming notable baselines like Claude-Computer-Use and achieving competitive results among open-source agents. The inclusion of detailed algorithms (e.g., Iterative Region Cropping, File Editing Logic) and open-source code enhances reproducibility.\n- **Clarity**: The paper is clearly written, with a structured breakdown of the architecture, components, and experimental setups. Figures and case studies (e.g., Figure 2 and 5) effectively illustrate the workflow, and the appendix provides comprehensive details on prompts, toolkits, and test cases.\n- **Significance**: The work is highly significant for the field of automated AI agents, addressing practical challenges in multimodal computer interaction. Its broad applicability across GUI tasks, coding, and general tool-use benchmarks makes it a valuable contribution to real-world agent development.\n\n**Weaknesses:**\n- **Originality**: While the engineering is impressive, the paper lacks novel theoretical insights or fundamentally new paradigms in agent design. The modular architecture, while effective, builds on existing concepts (e.g., tool-based and vision-based agents) without introducing groundbreaking ideas.\n- **Quality (Limitation)**: The evaluation, though comprehensive, could benefit from deeper analysis of failure cases or edge scenarios to better understand the agent’s limitations. For instance, the paper notes minor limitations in file editing edge cases but does not explore these systematically.\n- **Significance (Scope)**: The work focuses heavily on performance metrics but provides limited discussion on scalability or real-world deployment challenges, such as computational costs for large-scale use or robustness in dynamic environments.\n\nQUESTIONS:\n1. **Failure Case Analysis**: The paper mentions minor limitations in file editing edge cases (e.g., 9.6% failure rate). Could the authors provide a detailed analysis of these failures, including specific examples and potential improvements? This could strengthen the quality score by demonstrating robustness.\n2. **Scalability and Cost**: The paper uses resource-intensive models (e.g., Claude-3.7-Sonnet, A100 GPUs). How does INFANTAGENT-NEXT scale for real-world deployment, particularly in terms of computational cost and latency? Addressing this could elevate the significance score.\n3. **Novelty of Workflow**: The modular workflow is effective but builds on existing paradigms. Can the authors clarify what aspects of the workflow (beyond integration) are truly novel? A stronger case for originality could improve the originality score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduced a generalist agent framework named InfantAgent-Next. The framework adopts a multi-model collaboration architecture, where models with different modalities and specializations are assigned to various tasks (such as planning, tool selection, and visual grounding) to achieve optimization of overall capabilities. Experiments demonstrate that InfantAgent-Next achieves promising results across multiple types of tasks, including computer interaction (OSWorld), real-world question answering (GAIA), and coding (SWE-Bench).\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. The proposed agent system is evaluated on multiple benchmarks and achieves promising results on tasks such as computer interaction and real-world problem-solving. This demonstrates the versatility of the proposed method. And I believe it can be readily extended to a broader range of scenarios and tasks, considering its flexibility in integrating diverse models and tools.\n\n# Weaknesses\n\n1. This work is more of an integration of existing technologies. The design of the agent system's architecture and workflows does not exhibit significant innovations compared to existing methods; the design of core modules, such as planning, visual grounding, tool use, and context management, is also not original.\n\n2. I also have concerns about the fairness of the experiment results. Considering that the contribution of this work lies in the agent architecture rather than the model itself, the comparison should focus on the agent design. However, the baseline methods in table 2-4 are based on different models, some of which have significant gaps in parameter size and model capability.\n\nQUESTIONS:\n1. Could the author elaborate more on the methodological innovation and contributions? While engineering optimization is crucial for building such an agent system, as an academic paper, we would value an analysis of the underlying principles and methodological advancements.\n\n2. In Table 1, the author compares the proposed method with existing works from the perspective of modular design. Could the author provide an ablation analysis of all these modules in InfantAgent-Next?\n\n3. InfantAgent-Next is designed to be model-agnostic and flexible for use with various LLMs/vLLMs. The experiments only tested Claude-3.7-Sonnet and UI-TARS-1.5-7B—how significantly would performance change with different models?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper proposes a modular GUI‐agent framework, InfantAgent-Next, that uses vision and tools (e.g., computer operations) to interact with GUI. The method uses UI-TARS-1.5-7B for visual grounding and Claude-3.7-Sonnet for planning. Its action space includes several predefined tools plus mouse‐click actions commonly used by pure‐vision agents. Results show that InfantAgent-Next was the best open‐source agent framework on OSWorld at the time.

STRENGTHS AND WEAKNESSES:
Strengths

* Achieves competitive performance on the OSWorld, SWE, and GAIA leaderboards.
* Provides a detailed description of the proposed method.
* Conducts experiments across multiple GUI‐agent benchmarks.

Weakness
* The modular framework has little significant difference from existing modular GUI‐agent frameworks.

QUESTIONS:
1. In terms of framework design, what are the main differences between your proposed framework and existing modular GUI agents such as MobileExperts, InfiGUIAgent, and AGENT S?
2. Have you tried using different backbone models?
3. I would like to see more examples like those in Figure 5, including both successful and failed cases, to better understand how the system works.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
The paper introduces InfantAgent-Next, a modular multimodal agent that performs real-world computer tasks by combining tool-based and vision-based approaches. It separates the agent workflow into three stages—planning, tool selection, and execution—each handled by specialized models. The system supports text, image, audio, and video inputs and can interact directly with GUIs. Key contributions include a unified framework for multimodal interaction, an iterative visual grounding method for accurate mouse clicks, and a robust file editing strategy.

STRENGTHS AND WEAKNESSES:
**Strengths:**
1. Experiments cover a wide range of tasks and benchmarks, showing the agent's effectiveness.
2. Practical improvements such as iterative visual grounding and robust file editing effectively mitigate common failure modes in current agent systems.

**Weaknesses:**
1. The core ideas seems mostly integration of known components rather than novel contributions.
2. The iterative visual grounding method for mouse clicking is intuitive and effective in narrowing down regions. However, it assumes that the initial grounding is at least roughly correct. If the initial prediction is wrong, later iterations may focus on irrelevant regions. The paper does not discuss whether the agent can detect failure (e.g., no object found) and roll back or restart from a broader view.

QUESTIONS:
See weakness.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03