Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
Authors: Tajamul Ashraf
Amal Saqib
Hanan Gani
Muhra AlMahri
Yuhao Li
Noor Ahsan
Umair Nawaz
Jean Lahoud
Hisham Cholakkal
Mubarak Shah
Philip Torr
Fahad Shahbaz Khan
Rao Muhammad Anwer
Salman Khan
Venue: ICLR 2026 Poster (2026)
Keywords: Vision centric Agents
Deep Reasoning
VLMs
Tool Use Evaluation
Primary area: datasets and benchmarks
TL;DR: Agent-X benchmark (828 multimodal tasks) tests deep, stepwise reasoning in vision agents; SOTA models score <50%, revealing major gaps in multimodal reasoning and tool use.

Abstract: Deep reasoning is fundamental for solving complex tasks, especially in vision-centric scenarios that demand sequential, multimodal understanding. However, existing benchmarks typically evaluate agents with fully synthetic, single-turn queries, limited visual modalities, and lack a framework to assess reasoning quality over multiple steps as required in real-world settings. To address this, we introduce Agent-X, a large-scale benchmark for evaluating vision-centric agents’ multistep and deep reasoning capabilities in real-world, multimodal settings. AgentX features 828 agentic tasks with authentic visual contexts, including images, multi-image comparisons, videos, and instructional text. These tasks span six major agentic environments: general visual reasoning, web browsing, security and surveillance, autonomous driving, sports, and math reasoning. Our benchmark requires agents to integrate tool use with explicit, stepwise decision-making in these diverse settings. In addition, we propose a fine-grained, step-level evaluation framework that assesses the correctness and logical coherence of each reasoning step and the effectiveness of tool usage throughout the task. Our results reveal that even the best-performing models, including GPT, Gemini, and Qwen families, struggle to solve multi-step vision tasks, achieving less than 50% full-chain success. These findings highlight key bottlenecks in current LMM reasoning and tool-use capabilities and identify future research directions in vision-centric agentic reasoning models

Published as a conference paper at ICLR 2026
AGENT-X: EVALUATING DEEP MULTIMODAL REA- SONING IN VISION-CENTRIC AGENTIC TASKS
Tajamul Ashraf Amal Saqib"; Hanan Gani', Muhra AlMahri', Yuhao Li', Noor Ahsan', Umair Nawaz', Jean Lahoud', Hisham Cholakkal', Mubarak Shah?, Philip Torr Fahad Shahbaz Khan', Rao Muhammad Anwer', Salman Khan
'Mohamed bin Zayed University of Artificial Intelligence (MBZUAT), United Arab Emirates University of Central Florida, USA á University of Oxford, United Kingdom
'Equal contribution
A ABSTRACT
Deep reasoning is fundamental for solving complex tasks, especially in vision-centric scenarios that demand sequential, multimodal understanding. However, existing benchmarks typically evaluate agents with fully synthetic, single-turn queries, limited visual modalities, and lack a framework to assess reasoning quality over multiple steps as required in real-world settings. To address this, we introduce Agent-X, a large-scale benchmark for evaluating vision-centric agents' multi-stop and deep reasoning capabilitics real-world, multimodal settings. Agent-X features 828 agentic tasks with authentic visual contexis, including images, multi-image comparisons, videos, and instructional text. These tasks span six major agentic environments: general visual reasoning, web browsing, security and surveillance, autonomous driv- ing, sports, and math reasoning. Our benchmark requires agents to integrate tool use with explicit, stepwise decision-making in these diverse settings. In addition, we propose fine-grained, step-level evaluation framework that assesses the correctness and logical coherence of each reasoning step and the effectiveness of tool usage throughout the task. Our results reveal that even the best-performing models, including GPT, Gemini, and Qwen families, struggle solve multi-step vision tasks, achieving less than 50% full-chain success. These findings highlight key bottlenecks in current LMM reasoning and tool-use capabilities and identify future research directions vision-centric agentic reasoning models. Our data and code² are available.
1 INTRODUCTION
Agentic frameworks enable AT systems to interact with their environment by perceiving inputs, invoking tools, and executing actions. While perception and tool use are core components, solving complex tasks effectively requires reasoning, i.e., the ability to draw logical inferences, make decisions, and adapt over time based on multimodal inputs such as text, images, video, and temporal context (Kumar et al., 2025; Yang al., 2023; Yao et al., 2023; Thawakar et al., 2025). Deep reasoning in agents helps effectively plan, execute, and adapt across evolving scenarios (Liu et al., 2024b; Nathani et. al., 2025). Recent works have explored integrating reasoning and tool use within large multimodal models (LMMs) (Achiam et al., 2023; Bai al., 2023b; Bi et al., 2024; Team al., 2023; Jiang et al., 2023; Grattafiori et al., 2024), where LMMs act as controllers for planning, and callable tools handle external actions( CChaso, 2022; Gravitas, 2023; OpenAI, 2023). This architecture enables agents to combine perception, visual understanding, symbolic reasoning, and generative capabilities, significantly improving task performance in complex environments.
Axaggt///t Agent-x
1

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of Agentic Benchmarks. Columns show key dimensions including scale, realism, modality, reasoning depth, tool intcraction, and annotation quality. Our benchmark Agent-X uniquely supports all criteria with 828 diverse, manually verified agentic tasks.
Brachooot
Ageniic Taoks
Multinudda Yoob
Besi-world Quetee Stuiiindall Enpuls
Deep Besscnlng Esocutable Tools Hybeid Ausotation
Artbeax (1 al, 2003
Skkgentheadh (fliing al, 3011
dask filteee 2380
dra 3231)
caats [bas Mls 33380
Recossakgen (Cher aamiig
100
ziaye (hakaua al, 2233)
Agousel ((ueg)
K24
Input files
Query
Inpur files
Query
rede serferm Vinvll Appisiin and wint the total number of disins redes showe the rodar chart?
place video, many people performing duties?
1 by
oressniig stepse
Tools
Agent Courpe
novernt
Cos the the image is andertand the erinnn and sersed.
don dhe
whlp sock in
scue
winy
staps:
Tools
Verrly wosl parforiig lest Kroweage
Deso pras lui aoard ivriie of the uatiag
THOUSHT
Whail day Gerpes
Celomine the docian of ride scens for not in the insqe. boctin Lortn Hundopp Court
oce
(hirg mevd cen you devetiy mwnnt
peopin
distrct moste reppurred stars
The Oteetto eur toll desett low mony people and prisent who
CIJECT
Compurx sotll moetts
de beggh OCk roslts glvs canker
GT Answer
Lonaln Husicppl Coart
Eiiatiio
Justification
Phert moda"
Tord leaal modil 125
Hunicipll Coart The OkpenDoonerr cod5/red Mires angpls perforing dulee Julys reuuity officar and langee via the ract aapear to
ber DCC scala iidesffeed
Figure 1: Agent-X Snapshot: Example tasks from our benchmark illustrating multimodal queries that require step-by-step reasoning, tool use, and visual understanding across images and video. Each task includes structured thoughts, tool invocations, and a ground-truth answer with justification. The detailed annotations in Agent-X X enable thorough evaluation of existing agentic pipelines.
Existing benchmarks for agentic systems have primarily focused on text-based interactions, with limited support for multimodal inputs such images, vidcos, and multi-image com- parisons (Mialon et al., 2023; Liu et al., 2024b; Wang et al., 2024). While some efforts have extended to multimodal lasks, they are often restricted to static images, synthetic environments, or narrowly scoped domains, offering limited insight into an agent's 's ability to perform complex, tool-driven reasoning. A key limitation is inadequate evaluation of deep reasoning": : current benchmarks either neglect this aspect entirely or lack principled metrics for assessing multi-step logical coherence (Wang et al., 2024; Nathani et al., 2025; Liu et al., 2024b; Team, 2023). Furthermore, most existing benchmarks rely on either fully synthetic tasks or manual annotations, which lack scalability and fail to capture the depth of reasoning or the complexity of video-based multinodal interactions, falling short of modeling real-world agentic tasks (Han et al., 2024; Thawakar et al., 2025), as summarized in Table 1. As agents rapidly evolve in their capabilities, existing benchmarks struggle to keep pace, underscoring the need for more comprehensive, caasoning-centiii evaluations to track planning, adaptation, and tool use in authentic, multimodal interactive scenarios (Kiela et al., 2023).
Agent-X is the first benchmark that combines large-scale, real-world multimodal inputs (images, video, text) with tool-augmented step-wise reasoning evaluation across six environ- ments, providing both breadth and depth missing in GAIA (Mialon et al., 2023), GTA (Wang et al., 2024), and other recent dalasets. To address these limitations, we introduce Agent-X, large-scale evaluation benchmark for vision-centric agents that emphasizes two core principles: multmmodal reasoning and vision-first evaluation. Agent-X rigorously tests agents' ability to process complex visual and textual inputs, execute tool-augmented plans, and perform deep reasoning across real-world tasks. primary feature of our benchmark is its deep reasoning assessment, i.e., judging the ability to perform coherent, multi-step problem-solving, logical
In our context, Doep reasoning refers to coherent, multi-step inference grounded multimodal context.
2

---

Published as a conference paper at ICLR 2026
Table Task comparison of Agent-X with existing benchmarks. Unlike prior benchmarks, the queries in Agent-X avoid explicit tool references and direct instructions, thus encouraging agents to reason and act independently. Blue indicate explicit task guidance; Red highlights denote explicit tool invocation in prior benchmarks.
Method
Queries
Related Tools
I'mn writing blog post... firat retrieve available figlet styles and then genernte ASCII art for the strings using the style.
ToolBench
riglet, 1ist Liglet styles
APIBench
I am an engineer at Uber und... Write a puthon program in 1 to lines to call API in TorchHub.
ObjectDetection
GTA
How ninch should pay for the soda in the picture acording to the price the menu?
Imagelencrrptiinn CountGivenūbject, OCR
I need an illustration for my children's is book. I've imagined a where... Aßtr have the image, also nerd to identify all the objects, them add lobels to them.
ImageGeneration. ObjectDetection, Tagging
m&c's 's
Agent-X
What the the video from and what does the
ScaneDescriber, OCR, RegtonDescriber, WabSearch
(Ours)
person dressing corresponds to in normal circumstanees?
inference, and adaptive planning. To move beyond surface-level evaluation (focused on the correctness of the final result), we introduce fine-grained metries that capture both intermediate reasoning steps and overall task coherence. This approach helps distinguish genuine logical progression from inconsistencies and confabulations (plausible-sounding but disconnected steps) in reasoning chains, as illustrated in Figure 1. Agent-X spans six diverse multimodal environments, incorporating images, videos, and spatiotemporal contexts to evaluate agents in rich, realistic settings that demand generalization beyond text. Our task pipelines are derived from authentic user queries without explicit step instructions, simulating naturalistic agent interaction as highlighted in Table 2. These tasks are paired with executable toolchains via a semi-antomated pipeline that integrates a broad set of real-world tools, enabling scalable tool-augmented decision-making. In Agent-X, these com- ponents create comprehensive framework for evaluating agentic performance in perception, reasoning, and action. Our main contributions are as follows:
We propose a large-scale benchmark (Agent-X) for evaluating vision-centric agents, emphasizing deep reasoning capabilities across diverse multimodal environments.
We introduce fine-grained metrics to evaluate LMM based agentic frameworks, focusing tool use, deep reasoning and planning capabilities in real-world tasks.
We evaluate 10 mainstream LMMs on the Agent-X benchmark, uncovering key limi- tations in real-workd multimodal agents and providing actionable insights to guide future research on agentic development.
2 RELATED WORK
Large Multimodal Agents: The rapid progress in LMMs has catalyzed the development of agentic frameworks capable of autonomous planning, tool use, and decision-making. A growing body of rescarch has explored the extension of LMMs beyond conventional text generation by integrating them with external tools such as APIs (Zhang et al., 2025b), document processing (Musumeci el al., 2024), operating systems (Mei et al., 2024), and web interfaces (Song et al., 2024), allowing them to interact meaningfully with their environments. This evolution has led to the emergence of tool-augmented agents such as Avatar (Wu et al., 2024), LangChain (Chase, 2022), AutoGPT (Gravitas, 2023), and BabyAGI (Nakajima, 2023), which provide frameworks for textual reasoning and execution of external actions. In the context of web browsing, systems such as WebShop (Yao et al., 2022), WebGPT (Nakano et al., 2021), and WebCPM (Qin et al., 2023) enhance LMMs with capabilities for browsing, searching, and information retrieval. Beyond textual modalities, integrations such as RestGPT (Song et al., 2023) and AppAgent (Zhang et al., 2025c) enable interaction with REST APIs and emulate touchscreen operations, respectively. In the multimodal domains, viston-centric agents like MLLMTool (Wang et al., 2025) and LLaVA (Liu et al., 2024a) equip LMMs with the ability reason about visual content. through pre-trained vision models. Neverth there
3

---

Published as a conference paper at ICLR 2026
is no standardized evaluation protocol to rigorously assess the reasoning capabilities of these agents, particularly in tasks that require tool invocation and decision-making.
Multimodal Agentic Benchmarks: A broad range of recent benchmarks have emerged to evaluate the performance of agentic frameworks. Recent efforts like ToolBench (Qin et al., 2024) introduce REST APIs and evaluate tool use with metrics such as pass rate and win rate, while APIBench (Patil et al., 2024) assesses accuracy for APIs. Complementing these, APIBank (Li et al., 2023) presents a diverse suite of commonly used APIs, such as search engines and hotel reservations, offering a comprehensive framework for evaluating the planning, retrieval, and execution skills of agents based on LMM. Benchmarks such as ToolQA (Zhuang et al., 2023), Gentopia (Xu et al., 2023), Gorilla (Patil et al., 2024), and AgentBench (Liu et al., 2024b) often repurpose standard data sets or follow restricted evaluation protocols, which limit their utility in assessing open-ended and generalizable tool use capabilities. To address this, GAIA (Mialon et al., 2023) poses conceptually challenging questions for like understanding, GTA (Wang et al., 2024) evaluates agents with executable toolchains in real-world settings, and OSWorld (Xie et al., 2024) offers multi-step tasks based real user behavior. MLGym (Nathani et al., 2025) further adds complex multimodal tasks requiring deep reasoning and tool use. As shown in Table 1, despite promising advances, current. evaluations face two key limitations: (1) most benchnarks focus on final answer accuracy, overlooking interpretability, which is crucial for agentic tasks involving reasoning and tool use, and (2) the absence of standardized and interpretable evaluation protocols hinders meaningful comparison across methods.
Reasoning in Large Multimodal Models: Multiple methods have been introduced to enhance the reasoning capabilities of LMMs, enabling models to handle complex, multi-step problems through explicit reasoning processes (Amizadeh et al., 2020). Following the rise of reasoning techniques, benchmarks have emerged to evaluate reasoning and thhhuutt in models. Early efforts like CLEVR (Johnson et al., 2017) tested basic visual logic, while StrategyQA (Geva et al., 2021) introduced multi-hop reasoning. ScienceQA (Saikh et al., 2022) added multimodal scientific tasks, MathVista (Lu et al., 2023) unified math reasoning datasets, and ShareGPT-40 (Zhang et al., 2024) offered extensive chain-of of-thuught samples to enhance intermediate reasoning. The m&m's 's (Liu ct al., 2024c) benchmark focuses on multi-stop, multimodal reasoning and evaluates various planning strategics. However, many benchmarks, including m&m 's rely on Al-generated queries with predefined tool sequences, limiting their realism. LLaMA-V (Thawakar et al., 2025) introduced benchmark focused on assessing lasks requiring multiple reasoning steps. While these reasoning- focused benchmarks have been introduced, they do not address the agentic framework or evaluate tool usage and the associated reasoning within vision-centric agentic tasks.
-issonnn--ocusedd
3 AGENT-X BENCHMARK
The Agent-X benchmark evaluates the capacity of vision-centric agents to perform complex reasoning tasks that necessitate proficient tool usage across diverse, real-world scenarios. It accommodates broad spectrum of input modalities, including single-image analysis, multi- image comparisons, video interpretation, and multimodal image-(ext interactions. Agent-X presents agents with authentic visual data and requires them to use multi-step reasoning processes. The construction of Agent-X follows a autimadtomated pipeline (illustrated in Figure 2), where an initial set of candidate queries is generated through large multinodal models (LMM) and subsequently refined and validated by human experts.
3.1 BENCHMARK DESIGN
In our proposed Agent-X benchmark, each task is formally defined as structured tuple: S, -V, (1, Tr, A,, where: V; denotes the multimodal context, which may comprise a single image, text, multiple images, or video frames. These visual inputs are carefully selected to reflect diverse real-world scenarios. Q, is the query associated with V/ that necessitates multi-step deep reasoning and the strategic use of external tools. T, S To {tx J is the subset of tools employed to resolve the query, where To is a predefined library of N tools encompassing perception, visual operation, math, and artistic tasks. Further details in
4

---

Published as a conference paper at ICLR 2026
Tool List
AGENTX x
Multimodal data
JSON
Tool Metadata
Human verification
teat
....
LHME
mages
Generated
Human varification
vidcos
querias
Refined queries
Benchmark
Figure 2: Overview of the Agent-X benchmark construction pipeline. Starting from mul- timodal data and predefined toolset, an LMM generates initial queries, which are refined by annotators for realism and correctness. The LMM then produces step-by-step reasoning, which refined to create a high-quality tool-augmented reasoning trace.
Appendix $B. R. = [(t,, ag 7;)] is the deep reasoning trace, capturing the sequence of interactions during problem-solving Here, each step is defined as a triplet (4,, aj r3), where t, is the tool used, aj the input arguments, and ry the resultant output. A, is the final answer derived from the reasoning process, which may take varions forms, including textual explanations, numeric values, or generated content. Finally, the justification, J. explains the reasoning trace in natural language to support A; and enhance interpretability.
In our framework, Te comprises a carefully curated collection of 14 tools distributed across diverse environments. These 14 tools capture the breadth of capabilities required for real-world, vision-centric reasoning.
comprehensive list of these tools is presented in Appendix $B. The queries Q are classified into three distinct categories: factual, interyrelive, and generative. Illustrative examples of these query types are provided in Appendix $I. In the case of factual query, the answer is specific, uniquely determined value, such as number or a phrase. For an interpretive query, the final answer consists of descriptive text. While the answer is not unique, it conveys a general concept idea, with the final response comprising a reference answer. For a generative query, we do not directly utilize the generated output since the LMMs are descriptive models; they only provide textual descriptions rather than generating visual content like images/videoss In these instances, A, = 0.
3.2 TASK PIPELINE
To design the task (V,, Qi, To), we employ semi-automated pipeline (Figure 2) guided by three fundamental principles. First, each lask (V;, 2.) must be solvable using a subset of tools T C Te, ensuring that the problem can be addressed through the functional capabilities of the specified tools. Second, the query Q2 should not explicitly list the required tools or the sequence steps, forcing the model to plan and reason through the task independently. Finally, the tasks are grounded in realistic, meaningful scenarios that mirror complex and real-world challenges.
Our query construction process begins with generating candidate queries using an LMM, which is provided with the visual input V, and the available complete toolset T.. The LMM generates initial queries that human annotators then refine and validate for clarity, coherence, and realism. In the next step, LMM re-engaged with the refined query to produce detailed reasoning trace R,, capturing the sequence of tool calls (inpu/ooupuut) intermediate decisions, and final answer A. Human reviewers further ensure that the trace logically consistent and aligned with the query, while the final answer, A; and its justification, J, are validated for correctness and comprehensibility: The prompts provided to the LMM for query generation and reasoning trace construction included in Appendix $J.
3.3 HUMAN- REFINEMENT
To establish a robust Agent-X benchmark, visual input consists images and videos sourced from publicly available datasets. To maintain diverse and realistic dataset, queries
5

---

Published as a conference paper at ICLR 2026
Stepe vs Tool Mlgge per Overy
Statletdc
Value
Total Tosls
828
Axsorees Image
222
Asswaex Too
Row
1021 / 838
inpuxx
Vicee
Total tool etils
Assenge stops per Laok
2807
5-1
AGENTX
Unique uode nerd
14
Exumples w/ 2/3
tools 2300 / 389 130
24
Barbocamens
Aznortotces
Veriicatioa per
(a)
(b)
(c)
Figure 3: Overview of the Agent-X benchmark, (a) Key data statistics, (b) Overall frequency of tool usage and number of steps. (c) Distribution of tasks across six environments.
Q are crafted to avoid direct references to specific tools. For instance, query such as "Count the number of objects present in this image" is avoided bocause it directly hints at using the ObjectCounter lool. The reasoning steps R. capture the logical flow for each query, with an average of three steps. The answer A provides a final response, while J offers supporting evidence such as URLs or screenshols, especially for queries involving web search tools. This structure ensures that each query is well-defined and verifiable.
Task and Reasoning Construction. Our task construction process begins with automated generation phase, where an LMM produces three candidate queries for each of 1,021 initial visual inputs (images video frames). Annotators then seleet the best query from each set, yielding 1,021 raw queries. Detailed annotation guidelines, including query construction rules and reasoning/answer refinement protocols, are provided in Appendix $H. Each candidate is carefully reviewed for clarity, realism, and suitability to ensure that it cannot be answered directly from the input alone, but instead necessitates meaningful tool use, is expandable to diverse scenarios, and supports multi-step reasoning. Through this refinement process, queries failing to meet these criteria are discarded, resulting in a final pool of 828 validated tasks. Each task is paired with one more unique visual inputs, with no visual input reused across tasks. For web scarch queries, we additionally enforce two safeguards: (1) the query must be answerable using real-time search rather than relying solely on static knowledge, and (2) it should reference credible sources, such as "Whal were the global average temperatures reported in April 2024?", rather than overly general prompts like "What is climate chonge?"
In the second stage, construct the toolchain for each validated task. Each query Q and its corresponding visual input V are provided to an LMM along with the toolsct Te, which generates an initial reasoning trace, final answer A, and justification J. The rcasoning trace specifies the sequence of tool invocations, input arguments, and intermediate outputs, formatted in a JSON-style dialogue. Human annotators rigorously review and refine these LMM-generated traces to ensure logical consistency, correctness of tool use, and factual aligument of the final answer with the reasoning process. Annotators correct errors, replace inappropriate tool choices, and filter out tasks that cannot be solved reliably. Appendix $L illustrates examples of LMM--enerated queries and reasoning traces before and after human refinement.
3.4 DATASET COMPOSITION AND STATISTICS
The final Agent-X benchmark comprises a diverse set of queries covering multiple categories, including factual, interpretive, and generative tasks. The dataset is constructed using 14 executable tools, spanning six diverse environments, e.g., autonomous driving (Cordts et al., 2016; Wood, 2020; Yu et al., 2020), security and surveillance (Aktı et al., 2019; Sultani et al., 2018; Lu et al., 2013; Naphade et al., 2023), math reasoning (Lu et al,, 2023; Ling et al., 2017; Cobbe et al., 2021), web navigation (Sunkara et al., 2022; Zhou et al., 2023), sports
6

---

Published as a conference paper at ICLR 2026
Table 3: Evaluation Metrics. This table outlines the full suite of metrics used in Agent-X benchmark, organized by Step-by-Step, Deep Reasoning, and Outcome modes.
Metric (Symbol)
Description
Step-by-Step
Tool Precision Tp
Grounding
G,
Corvet reference objects, regiono, or aliributes inpal. Accuracy of selecting the correct tool each reasoning step.
Mode
Toal Accuracy (T,)
Corrett use of tools with uppropriate topots and nidputs.
Faitafulinaaa Fuee
Logtou! consistoucy neroas the rassniing process.
Deep
Context Score C,
Effectine use of multimodal and commonsenus contest.
Reasoning
Factual Preciaton Pp
torsettuass of fuctuol ieformation withost ballarination.. Cooerage of semanlicnlly nesessurg
Mode
Semantic Accuracy Suce
Goal Accuracy Gaae]
Final unewer sorwary for fuctust and interpreliie queriess
Outcome
Goal Accuracy w/TsgOen G
Fiuul onewer aceuary for generatie queries.
Mode
Toolset Accuracy Tuce
F1 soore ovsrail correct selection and wse.
(Karpathy et al., 2014; Wu et al., 2022), and generic visual reasoning (Lin et al., 2014; Wang et al,, 2024). Full details of the source datasets are provided in Appendix $C.1.
The annotation effort amounted to roughly 50 hours per annotator, reflecting the full human refinement and validation process. Approximately 800K API tokens were used during benchmark construction. These categories are designed to support wide range of multimodal reasoning tasks. The total number of tasks, query analysis, tool analysis, and annotation labor hours are detailed in Figure 3. Most queries involve the use of two four executable tools. This diverse setup ensures that Agent-X can comprehensively evaluate model performance across various tool-augmented reasoning tasks.
AGENT-X EVALUATION
4.1 EXPERIMENTAL SETUP
Evaluation Modes: We evaluate the models on the Agent-X benchmark using three distinct evaluation modes: 1. Step-by-Step: This mode evaluates the agent's ability to reason through individual steps within reasoning trace (R). It measures how accurately the agent understands and reproduces structured tool-use sequences grounded in visual inputs. 2. Deep Reasoning: This mode assesses the agent's capacity to generate coherent, logically consistent mulli-step reasoning traces. It focuses on integrating visual and textual inputs to produce contextually relevant, semantically complete, and factually accurale reasoning. Outcome: This mode measures the agent's 's overall task-solving ability by evaluating the correctness of the final answer and the accuracy of tool usage.
Metrics: We design a suite of fine-grained metrics (Table 3) to evaluate agentic reasoning across all three modes comprchensively. These metrics capture key aspects of the agent pipcline. For Goal Accuracy, we exclude image generation queries and focus on factual and interpretive queries, using exact matching against the gold answer and descriptive matching against labeled responses using GPT-4o (Hurst et al., 2024) respectively. For image generation, we deline Goal Accuracy/ ImggGe, which assesses the correctness of predicted input parameters, assuming accurate inputs yield suitable images. Detailed metries computation and implementation are provided Appendix $C.
4.2 BENCHMARK RESULTS
Table presents the core evaluation results of Agent-X across three modes: STEP-BY- STEP, DEEP REASONING, and OUTCOME. We evaluate a mix of closed-source models (e.g.. GPT (Achiam et al., 2023; OpenAT, 2025) and Gemini (Team et al., 2023; 2024a)) and strong open-source counterparts (c.g., InternVL (Chen et al., 2024a; Zhu et al., 2025) and Qwen (Bai et al., 2023b;o)). While closed-source models lead overall, open-source models show competitive behavior in select metrics, offering key insights across agentic frameworks, which we summarize below. Beyond these core comparisons, we further extend the benchmark with additional open-source vision LMMs, detailed in Appendix $D. We additionally report results using two alternative evaluation judges, Qwen and human annotators, in Apendix SE. The evaluation prompts provided to GPT-4o, Qwen, and human annotators
7

---

Published as a conference paper at ICLR 2026
Table 4: Overall results on Agent-X. We report performance across three evaluation modes: STEP-BY-STE,, DEEP REASONING, and OUTCOME. Metrics include: G. (Ground- ing Score), Tp (Tool Precision), Tace (Tool Accuracy); F Face (Faithfulness Accuracy), C2 (Context Score), Fp (Factual Precision), Sace (Semantic Accuracy), Gace (Goal Accuracy), G2 (Goal (auracy/IIngen), and Tac (Tbolset Accuracy). The best resulls are highlighted in bold, and secord-best are underlined.
|
Step-by-Step
Deep Reasoning
Outcome
Model
G.
Tp
Tace
F Face
C.
Fp
Sace
Gace
G
Tace
Open-source"
Thi-4-WL-Instrun
0.13
0.21
0.24
0.61
0.19
0.47
0.40
0.11
0.26
0.42
InternVL2.5-B8
0.45
0.31
0.47
0.68
0.47
0.52
0.60
0.28
0.55
0.58
Gemma-3-4B
0.26
0.78
0.61
0.54
0.38
0.54
0.27
0.60
InternVLL3-8B
0.46
0.34
0.54
0.68
0.45
0.70
0.40
0.20
0.59
0.62
VideoLLaMA3-7B
0.45
0.28
0.46
0.65
0.46
0.62
0.54
0.28
0.54
0.54
0.54
0.43
0.63
0.75
0.57
0.56
0.67
0.36
0.65
0.67
Closed-source
Gemini-1.5-Pi
0.43
0.23
0.84
0.62
0.45
0.53
0.62
0.04
0.56
0.48
Gemini-2. 5-Prr
0.40
0.36
0.81
0.72
0.48
0.64
0.73
0.40
0.56
0.62
GPT-4o
0.60
0.47
0.72
0.81
0.57
0.79
0.59
0.37
0.70
0.68
OpenAI-ot-mini
0.42
0.32
0.89
0.71
0.51
0.60
0.80
0.45
0.63
*Resuits for the additional models Appendiar $17 and cross-judge consistency study Appendiz SR.
are included Appendix şK. To mitigate bias, all predictions are cross-checked by multiple graders (GPT-4o, Qwen-14B, and humans), with consistent model rankings across settings. Human scores correlate strongly with automatic grading, and residual discrepancies are uniformly distributed, confirming systematic favoritism. Furthermore, evaluation metrics are explicitly bias-aware (docoupling syntax from semantics, normalizing tool arguments), and task sceds were rewritten under strict QA to avoid leakage. Together, these safeguards ensure fairness, robustness, and reproducibility of the results.
Insight 1: Real-world tool-use tasks remain challenging for current. LMM agents. Despite multimodal capabilities, no model exceeds 50% in Goal Accuracy (Gace). o4-mini, the best performer, achieves only 45%, while most open-source models are below 30%, highlighting the difficulty of tool use and final answer consistency in real-world settings.
Strong reasoning capabilities contribute to higher task success rates. Models that maintain consistently strong scores across reasoning metries are likelier to perform well on final task outcomes. GPT-4o, for instance, achieves high scores in Face 0.81, Fp = 0.79, and Sace = 0.59, which correlates with relatively high Gaec = 0.37 and T Tec 0.68. Likewise, Qwen2.5, with solid reasoning scores (C, 0.57, Sace = 0.67), attains Gace = 0.36, outperforming many open-source peers. These results indicate that deep reasoning and structured execution enhance task success in vision-centric agents.
Insight 2:
Insight 3: Tool invocation and argument prediction remain core bottlenecks. Tool-related metrics show the widest variation, indicating difficulty in accurate and well- formatted tool use. While the GPT family achieves strong reasoning capabilities, its Toolset Accuracy is low. Open-source models like Civen..-VLL-7 show similar inconsistencies as highlighted in Table 4. These results support the observation that argument formatting and tool chaining are the weakest links, disrupting overall pipeline reliability and highlighting a need for more precise execution control.
4.3 ERROR ANALYSIS
Qualitative Analysis: We conduct a qualitative error analysis (Figure 4) to uncover key limitations in current vision-centric agentic models. Our findings reveal four prevalent issues: (1) agents often exhibit shallow reasoning by skipping frame-wise or multi-step analysis, especially in video-based tasks; (2) models frequently misuse hallucinate tools not defined in the metadata, indicating poor tool grounding: (3) many responses violate expected output formats, producing incomplete non-JSON-complian traces that hinder evaluation; and (4)
8

---

Published as a conference paper at ICLR 2026
Prompt: has inidigent mi meld sgent, Yhe presked wit A text query (qeers> As image voles ofilr set loos sau with you reusning with mits daia tost glve as hekket (nsts smaj Your abjettie oxteer query basad as the phen daail contart by exonding are using moe appreptins teel: Yhu mut recson rapelyyrn Each reaszinng
"ppccyy sskess and cheie ftrmst wil ope certaind chearly utsluwwll" "..-feed ffaal pulpul" I'inal anower "Prodd and coesis unove bovel preveux
the selpet", (Nooghh: "troay near. "uilfintant Tronde a soier for the feei prower:
VidenLLLMA3-73
Input Awwe/tt 450.04
uspat Flaee Age.. att Apersi ALE
Query: "Now many womes ore chere to the sideo and andteee the color of their clomaet
Query. "Now much those botties coor?*
the correst resporse formst
(Pheag") Thak" "Costt numher of nemes in the adee i. toot 'Olgettuumn Coutrr
Error Resporse
recsen uq supe [T'sep
The cornect respanse formor
Miral praver" 1550
Errer Responue
("wsp:: tualt Devor the
"retavingg Pape".
"ool puguet 2: "sought" The CejeriCeutter taa
werih auch fooee
Phoph 1. bos"" "Vnnd the price Hereeigg
Héretty
"teed south "Takks Search Teal" "ood La
"ooll" 'Meriiy the iokor their duthes "ndd ose""
exceGeunken kerr "ood augut 'Gee words waaring
seenifle the
wenen
provdc lis product and thet
Tedl auppe? image
"rouge""
each vides from attalbi
deuiity use up peise Hamonyy from talk
bettlet. "haggh"
click tiiking cxter serven TI. "hoa Jnwwer Posker
"The poton tool provlde dier informsian
the vike Fateph
opeer
"sog" "800 "douggh" "Te
"Oxtarnkinng the tym of
and weating doeting Jas Justiiation
('eeph Sud (hteg' a..l]
dirccta andaab from the talee starch reade ]]
cornet prix from prise 1arg Paas II. "wiporrss" festien "The betthr are deriified $1o Thus, bettle s600 388 "
"/sing siject sounting and soene docripn tode, we
the wordnn diibbig" jartfoctionn Taxs
Ieuild .SoN you
dese bed the
idwolies
foom consinentl stowed
Hasing Jusiflation Pormatiing srrof
inceree fial sanver
erar)
WWnn suel Invocaation (lermsting srrr)
Skalke reaseiing Rasossing anoo))
srughet the vid indioniig sun thay come
Rodkadort mar Rhannety areett
No rip ty recceing (aezicing sror)
Engry teel (oormating cmrol
Figure 4: Qualitative comparison of GPT-4o and VideoLLaMA3-7B on Agent-X visual reason- ing lasks. GPT-4o hallucinates tool use and gives incorrect justifications VideoLLaMA3-7B lacks temporal reasoning and frame alignment. More comparisons in Appendix $F.2.
Table 5: Error Breakdown Across Models. Common planning, formatting, and reasoning errors on Agent-X across GPT-4o, Gemini-1.5-Pree and InternVL3-SB. Formatting errors are counted alongside planning and reasoning errors. Extended details in Appendix $P.1.
Error Type
GPT-40
Gemini-1.5-Pro
InternVI.3-8B
Planning Errrrs:
No action, resporse.
157 (17.6%)
(0.2%)
172 (12.8%)
No action, the whole response is a model thought.
2 (0.1%)
0
Pormatling Errors:
Invalid ISON forual in arguneett specification.
235 (26.4%)
755 (44.3%)
454 (33.8%)
Multiple tool calls in single step.
118 (13.2%)
172 (10.1%)
126 (9.4%)
Final answer generation without adhering to the format.
(6.7%)
174 (10.3%)
220 (16.1%)
Reasoning Eryors:
Misinterpreting visual content (e.g., wrong object recognition)
165 (18.5%)
581 (34.3%)
189 (14.1%)
Incorrect spatial ressuing (e-g,, wring relative position)
156 (17.5%)
8 (0.5%)
181 (13.5%)
Total Errors
891(100%)
1695 (100%)
1342(100%)
agents tend to hallucinate reasoning steps or bypass visual verification, resulting in factual inaccuracies. These insights highlight the need for better temporal reasoning, tool schema adherence, structured output enforcement, and visually grounded planning.
Quantitative Analysis: We examine failure modes across three models (GPT-4o, Gemini- 1.5-Pro, and InternVL3-SB), summarized in Table 5. Formatting errors are tracked separately to focus reasoning ability. GPT-4o demonstrates strong structural awareness with fewer formatting errors (13.2% multiple tool calls, 6.7% final format), but often hcsitates to act: 17.6% no response and 18.5% visual misinterprotation, indicating it plays safe but hesitates. Gemini-1.55Pro is aggressive but format-fragile, with 44.5% JSON errors, 10.3% final formatting issues, and 34.3% visual misinterpretattons:; only 0.2% no response. InternVL3-88 shows a balanced yet challenged profile: 33.8% JSON errors, 16.4% final format errors, and struggles in spatial (13.5%) and visual reasoning (14.1%), suggesting it suffers both structural and perceptual issues.
5 CONCLUSION
We present Agent-X, a comprehensive benchmark for evaluating the reasoning capabilities of vision-centric agents. The tasks in Agent-X are context-rich and span diverse multimodal scenarios. Our evaluation framework includes executable tools categorized across six agentic environments, ensuring a wide range of task coverage. We evaluate more than 12 state- of-the-art LMMs on Agent-X and find that even strong closed-source models struggle with agentic tasks. Our insights provide actionable guidance for improving agentic capabilities.
9

---

Published as a conference paper at ICLR 2026
We believe that Agent-X will drive future research in enhancing multimodal reasoning and robust tool integration for vision-centric agents.
Limitations. While Agent-X covers six diverse environments, it is currently monolingual and may inherit certain distributional bias. The semi-automated approach to query and tool-chain generation improves efficiency but occasionally produce lower-quality samples. Additionally, although Agent-X offers broad coverage, there remains ample opportunity for scalability.
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXw preprint arXiv:2412 12. 08905, 2024. 25
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023. 1, 7, 26
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arX.:.440.000073, 2024. 23, 27
Şeymanur Aktı, Gözde Ayşe Tataroglu, and Hazm Kemal Ekenel. Vision-based fight detection from surveillance cameras. In 2019 Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA), pp. 1-6. IEEE, 2019. 6, 21
Saeed Amizadeh, Hamid Palangi, Alex Polozov, Yichen Huang, and Kazuhito Koishida. Neuro-symbolic visual reasoning: Disentangling. In International Conference on Machine Learning, pp. 279 290. Pmlr, 2020. 4
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a. 23
Jinze Bai, Shnai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Foi Huang, et al. Qwen technical report. arXiv preprint arXiv:2300.1600 2023b. 7, 7, 23, 25, 26, 27
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large lisounlanguuae model with versatile abilities. arXiv preprint arxio:2308. 12966, 2023c. 7, 25
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. orXiv preprinl ar Xiv::4401 04222440,, 2024.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVP conference on computer vision and pattern recognition, pp. 11621-11631. 2020. 21
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Loon Maksin, Tejal Patwardhan, et al. Mlo-bench: Evaluating machine learning agents on machine learning engineering. arxiv preprint 1Xxx:241000700000
2024.7 2
Harrison Chase. Langchain, October 2022. URL langchain. 1,3 3
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic disual-linguisti tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185 24198 1919 2024a. 7, 25
10

---

Published as a conference paper at ICLR 2026
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. Scienceagent Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint arXiv:2440.. 05080, 2024b. 2
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 168, 2021. 6, 21
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognétion, pp. 3213-32222 2016. 6, 21
Adrien Deliege, Anthony Cioppa, Silvio Giancola, Meisam J Seikavandi, Jacob V Dueholm, Kamal Nasrollahi, Bernard Ghanem, Thomas B Moeslund, and Mare Van Droogenbroeck. Soccernet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer vidcos. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 4508-45191 2021. 21
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategics. Transactions of the Association for Computational Linguistics, 9:346-366 1846-361,-- 2021. 4
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint orXiv:2407.2 21 283,, 2024. 1, 23
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentrie video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18995 19012, 2022. 21
Significant Gravitas. Autogpt, 2023. URL AutoGPT. 1,
wtt.:tttwb..oo//igiiiiaaaaa Graniitss/
Shanshan Han, Qifan Zhang, Yuhang Yao, Weizhao Jin, Zhaozhuo Xu, and Chaoyang He. Lim multi-agent systems: Challenges and open problems, arXiv preprint 2rXic:2402.00 03578, 2024.7 2
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Conference on Machine Learning, pp. 20271-20309, 2024. 2
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpl-4o system card. arXiv preprint orXiv:24100 10.21266, 2024. 7, 23, 25, 26, 27
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXio:2310.068: 2023. 1
Carlos E Jimencz, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issucs? In The Twelfth International Conference on Learning Represenlations, 2024. 2
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
11

---

Published as a conference paper at ICLR 2026
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014. 7, 21
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In European conference computer vision, pp. 235 251. Springer, 2016.
Douwe Kiela, Tristan Thrush, Kawin Ethayarajh, and Amanpreet Singh. Plotting progress in ai. Contertual AI Blog, pp. 3, 2023. 2
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32 73, 2017. 21
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsine Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training. A deep dive into reasoning large language models. Xiv preprint or 20: 21321, 2025.
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li API-bank. A comprehensive benchmark for tool-augmented llms. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1002-1166,-011111111111111111111 2023. 2,
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp. 740 755. Springer, 2014. 7, 21
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 158-167, 2017. 6, 21
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et, al. Llava-plus: Learning to tools for creating multimodal agents. In European Conference on Computer Vision, pp. 126-142. Springer, 2024a. 3
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs agents. In The Twelfth International Conference on Learning Representations, 2024b. 2,
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216 233. Springer, 2024c. 2,
Cewu Lu, Jianping Shi, and Jiaya Jia. Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE international conference on computer vision, pp. 2720-2722 2013. 6, 21
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math- ematical reasoning of foundation models in visual contexts. In The 3rd Workshop on Mathematical Reasoning and Al at NeurlPS 23, 2023. 4, 6, 21
Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. Aios: Llm agent operating system. arXiv preprint ar X0x: 16971, 2024.
12

---

Published as a conference paper at ICLR 2026
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023. 2,
Emanuele Musumeci, Michele Brienza, Vincenzo Suriani, Daniele Nardi, and Domenico Daniele Bloisi. Lim based multi-agent generation of semi-structured docu- ments from semantic templates in the public administration domain. In International Conference on Human- Computer Interaction, pp. 98 917. Springer, 2024.
Yohei Nakajima. Babyagi, 2023. CRL httlt//pee/gggtt...ooc/////oaaan
3
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vincet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answering with human feedback. arXie preprint arXiv:21 11011122099332 2021.
Milind Naphade, Shuo Wang, David C Anastasiu, Zheng Tang, Ming-Ching Chang, Yue Yao, Liang Zheng, Mohamnee Shaiqur Rahman, Meenakshi S Arya, Anuj Sharma, et al. The 7th ai city challenge. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognition, pp. 5538 5548, 2023. 6, 21
Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, et al. Mlgym: A new framework and benchmark for advancing ai research agents. arXiv preprint arXio:2502. 14499, 2025. 1, 2, 4
OpenAl. Chatgpt plugins.
htt.:pppa.....oe/e/eee///aaaaaaa 2023. URL. https://openai. coom/index//aie httpphhttttttttpiiie
OpenAl. OpenAl o3 and 04-mini system card, April 2025. Version 1, released April 16, 2025. 7, 25, 26, 27
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37:126544-12656, 2024. 2, 4
Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, et al. Webepm: Interactive web search for chinese long-form question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8968 1888-8988,44 2023. 3
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating large language models master 16000+ real-workd APIs. In The Twelfth International Conference on Learning Representations, 2024. 2,
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3)2889-300 2022. 4
Yifan Song, Weimin Xiong, Dawei Zhu, Cheng Li, Ke Wang, Ye Tian, and Sujian Li. Restgpt: Connecting large language models with real-world applications via restful apis. ar preprint orXiv:2306 06624, 2023. 3
Yueqi Song, Frank Xu, Shuyan Zhou, and Graham Neubig. Beyond browsing: Api-based web agents. Xixiv preprint ar arXiv:2410 11.:2410.16444,, 2024.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6479 6488, 2018. 6, 21
13

---

Published as a conference paper at ICLR 2026
Srinivas Sunkara, Maria Wang, Lijuan Liu, Gilles Baechler, Yu-Chung Hsiao, Jindong Chen, Abhanshu Sharma, and James WW Stout. Towards better semantic understanding of mobile interfaces. In Proceedings of the 29th International Conference on Computational Linguistics, pp.
2022. 6, 21
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023. 1, 7, 25, 26, 27
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhuleng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arviv preprint or arXie:2403.0550 2024a. 7, 25, 27
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprinl ar Xix:244033. 08295, 2024b. 25
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, al. Kimi-vl technical report. arXio preprial Nttr:2504.074990 2025. 23
Lagent Developer Team. Lagent: InternLM lightweight open-source framework that allows users to efficiently build large language (ldel([lm) -basd agents. https://githut com/ InternLM/lagent, 2023. 2
Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav- ol: Rethinking step-by-step visual reasoning in 1lms. arXiv preprint arXiv:2501.0118 2025. 1, 2,4
Chenyu Wang, Weixin Luo, Sixun Dong, Xiaohua Xuan, Zhengxin Li, Lin Ma, and Shenghua Gao. Milm-tool: A multimodal large language model for tool agent learning, In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 6678 6687. IEEE, 2025.
Jize Wang, Ma Zerun, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. Gta: a benchmark for general tool agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. 2, 4, 7, 21
Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. Re-bench: Evaluating frontier ai r&ad capabilities of language model agents against human experts. arXie preprint arxiv:2411. 151111111, 2024.
Joanne M Wood. Nighttime driving: visual, lighting and visibility challenges. Ophthalmic and physiological optics, 40(2):187-25) 201, 2020. 6, 21
Fei Wu, Qingzhong Wang, Jiang Bian, Ning Ding, Feixiang Lu, Jun Cheng, Dejing Dou, and Haoyi Xiong. A survey on video action recognition in sports: Datasets, methods and applications. IEEE Transactions on Multimedio, 25:7943-7966 2022. 7
Shirley Wu, Shiyu Zhao, Qian Huang, Kexin Huang, Michihiro Yasunaga, Kaidi Cao, Vassilis loannidis, Karthik Subbian, Jure Leskovec, and James Zou. Avatar: Optimizing I1m agents for tool usage via contrastive reasoning. Advances in Neural Information Processing Systems, 37:25981 126000, 2024. 3
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Newral Information Processing Systems, 37:52040 52094,, 2024. 4
14

---

Published as a conference paper at ICLR 2026
Binfeng Xu, Xukun Liu, Hua Shen, Zeyu Han, Yuhan Li, Murong Yue, Zhiyuan Peng, Yuchen Liu, Ziyu Yao, and Dongkuan Xu. Gentopia. ai: A collaborative platform for tool-angmented lims. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 237-245, 2023. 4
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 1100000..118,,, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards
scalable real-world web interaction with grounded language agents. Advances in Newrul Information Processing Systems, 35:20744-200 40-75777777 2022. 3
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and
Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Represenlations, 2023.
Jiabo Yo, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-soquence understanding in multi-modal large language models. arviv preprint ar Xiv::2400.. 04840, 2024. 23, 27
Fisher Yu, Haofeng Chen, Xin Wang, Wengi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogencous multitask lcarning. In Proccedings of the IEEE/CVP conference on computer vision and pallern recogniilion, pp. 2636 2645, 2020. 6, 21
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Vidcollama 3: Frontier multimodal foundation models for image and vidco understanding. arXiv preprint ar arx::22001.1110 2025a. 25, 26, 27
Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang. Api agents vs. gui agents: Divergence and convergence. arXiv preprint arXiv:2503. 1 11069, 2025b. 3
Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pp. 20, 2025c. 3
Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thouge reasoning. arXiv preprint arv: 16199, 2024. 4
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert To, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXie preprint arXiv:2307.1330 13855, 2023. 6, 21
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvi3: Exploring advanced training and test-time recipes for open-source multimodal models. ar preprint ar Niv:22000 10,79, 2025. 7, 25, 26
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: dataset for Ilm question answering with external tools. Advances in Neural Information Processing Systems, 14:50117-50144 2023. 4
15

---

Published as a conference paper at ICLR 2026
Appendix for Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
Section A
Datacard for Agent-X
Section B
Tool Definition
Section C
Metrics and Implementation Details
Section D
Evaluation of Additional Models
Section E
Cross-Judge Consistency Study
Section G
Additional Experiments and Ablations
Section F
Additional Experiments on Error Analysis
Section H
Instruction for Annotators
Section I
Agent-X Task Examples
Section J
Generation Prompt Details
Section K
Evaluation Prompt Details
Section L
Benchmark Curation Examples
A DATACARD FOR AGENT-X
A.1 MOTIVATION
For what purpose was the dataset created?
The Agent-X X benchmark is designed to evaluate the multi-step and deep reasoning capabilities of vision-centric agents in real-world, multimodal settings. It features agentic tasks with real-world objectives that require implicit tool use. The benchmark includes executable tools across diverse environments and uses authentic images, videos, and text as context input. These components help bridge the gap between existing benchmarks and realistic tool-use scenarios.
Who created the dataset (c.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
The authors of this paper.
Who funded the creation the datasct?
The source of funding will be made available once the paper is no longer under anonymity.
A.2 COMPOSITION
What do the instances that comprise the dataset represent (c.g., docu- ments, photos, people, countries)?
Each instance in Agent-X is stored in JSON format. It contains natural language query, one or more input files (image, textual image, or video), and a set of tool descriptions available to the agent. It also includes reference reasoning chain consisting of step-by-step tool calls, cach with its input, output, and thought process. Each instance ends with final answer and justification that summarizes the agent's conclusion and explains how it was reached.
How many instances are there in total (of each type, if appropriate)?
There 828 instances in AgentX, with 716 images and 112 videos
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from larger sct?
We will provide all instances in GitHub repository for Agent-X.
16

---

Published as a conference paper at ICLR 2026
What data does each instance consist of?
Each instance includes natural language query, one or more input files, a list of available tool descriptions, a step-by-step reference reasoning chain, and a final answer with justification.
there a label or target associated with each instance?
Yes. Each instance includes reference tool chain, a step-by st-e reasoning trace, and a final answer with justification, all serving the ground truth for the given query.
Is any information missing from individual instances?
No.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
No.
Are there recommended data splits (e.g., training, developmenttvallice tion, testing)?
The whole datasct is test set.
Are there any errors, sources of noise, redundancies in the dataset? The dataset created using semi-automaled pipeline, but verified by human. Any noise in the data may be the result of human error.
Is the dataset self-contained, external resources (c.g., websites, tweets, other datascts)?
does it link to otherwise rely
The dataset is self-contained. While the image and video inpuls are sourced from existing datasets, all queries, reasoning steps, tool definitions, and annotations are newly created as part of this benchmark.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient con- fidentiality, data that includes the content of individuals' non-public communications)?
No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
No.
A.3 COLLECTION PROCESS
How was the data associated with each instance acquired? The queries and reasoning steps were generated using GPT-4o and then reviewed and refined by human annotators. Image and video inputs were collected from publicly available datasets. All tool chains, reasoning traces, and final answers were also produced by GPT-4o and verified for correctness by humans.
What mechanisms or procedures were used to collect the data (e.g., hard- apparatuses sensors, manual human curation, software programs, software APIs)?
The dataset was created using semi-aulomated pipeline. Queries, reasoning steps, and tool chains were generated using the GPT-4o API, and all outputs were manually verified and refined by human annotators. Image and video inputs were sourced from existing datasets using standard dataset APIs and tools.
Who was involved the data collection process (e.g., students, crowd- workers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? Researchers and student annotators.
Over what timeframe was the data collected?
The data constructed in 2025.
17

---

Published as a conference paper at ICLR 2026
Were any ethical review processes conducted (e.g., by an institutional review board)?
Yes. All images within Agent-X are available for academic use. Should any authors request the removal of their images from Agent-X, we will promptly comply.
A.4 PRETROCESINN/CELANNNI/AEEEEII
Was any preprocessing http//llannnnaaallliien of the data done (e.g., discretiza- tion or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?
The dataset is created using hybrid approach and verified manually.
Was the "raw" data saved in addition to the data (e-g., to support unanticipated future uses)?
There is no scparate raw data. The reasoning, queries, and annotations are created from scratch. The image and vidco inputs are filtered from existing datasets to ensure coverage of diverse environments and visual scenarios. Only the selected subset relevant our benchmark included.
Is the software that was used to preprocess/clenn/lbbe the data available? We created our Agent-X tool for annotation and used Microsoft Excel and VSCode to create the data.
A.5
USES
Has the dataset been used for any tasks already? No.
Is there repository that links to any or all papers systems that use the dataset?
No.
What (other) tasks could the dataset be used for? Agent-X used for evaluating the visoin centric reasoning ability of LMMs in real- world scenarios.
Is there anything about the composition of the dataset the way it was collected and uses?
cacnne///dcccee///aallllel that might impact future
No.
Are there any potential negative social impacts?
While Agent-X uses public datasets, potential societal risks remain, such as privacy concerns from images with people, conffict-related scenes, and possible hallucinations unsafe outputs during evaluation. There's also minor risk of generating harmful code in reasoning or coding tasks without proper safeguards.
A.6 DISTRIBUTION
Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?
No.
How will the datasct will be distributed (c.g., tarball on website, API, GitHub)?
The dataset will be released at GitHub.
Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? The dataset is released under the Apache License.
18

---

Published as a conference paper at ICLR 2026
Have any third parties imposed IP-based other restrictions the data associated with the instances?
No.
A.7 MAITTENANCE
Who will be hupporting//osttnggmmaittiiiiint the dataset?
The authors of this paper.
How the wwwer/curator/maan mmanger of the dataset be contacted (e.g.,
email address)?
Please contact with authors through emails in the paper.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?
Yes, users can propose issues and the dataset will be updated on Github.
If others want extend /aggment// build on/contribute to the dataset, is there mechanism for them to do so? Contact the authors of the paper.
19

---

Published as a conference paper at ICLR 2026
B TOOL DEFINITION
Table 6 provides a comprehensive overview the 14 tools categorized under four distinct functional domains: Perception, Visual Operation, Math, and Artistic. Each tool is described, highlighting its functionality, the nature of imputs it requires, and the output format it generates. These tools collectively enable wide range of automated visual and textual processing tasks, making them versatile for applications in both analytical and creative contexts.
Table 6: Detailed Definitions of Tools Across Categories
Name
Description
Input
Output
Perception
UCR
Extracts all visible text along with bounding box coordinates.
Image
Text with bounding boxes
KathDCR
Recognizes mathematical expressions and returns La- TeX format.
Image (with math)
LaTeeX string
SceneDeacriber
Generates brief natural language summary of the seene.
Image
Scene caption
RegionDescriber
Describes specified at- tributes of given image region.
Image, bounding box, tribute
Description of the region's attribute
Locate@bjeettByTxt
Identifies and localizes jects based on textual queries.
Image, object description
Bounding box and detec- score
ObjectCounter
Counts occurrences a specified object the im- age.
Image, object description
Integer count
Visual Operation
DrawBoundingBox
Draws bounding box on the image, optionally with a label.
Image, bounding box, notation (optional)
Image with bounding box
OverlayText
Overlays text at specified position on the image. Retrieves top search sults for given query.
Image, text, position, color
Image with text overlay
WebSearch
Query, top-k results Search results turn (optional)
MMtt
Calculator
Evaluates a single Python math expression. Only module functions are allowed; imports disal- lowed.
Text expression
Computed result
Solver
Executes SymPy code to symbolically solve equa- tions. Code define solution() function re- turning string.
SymPy code markdown format
Solution
CodePlotter
Executes Python code us- ing Matplotlib gener- ato plot. Requires solution() function that returus figure object.
Python vode markdown format
Generated plot image
Artistic
ImageGenerator
Generates an image based on given text prompt. Modifies image appearance Image, style instruction
Text keywords
AI Al-generated image
ImageStylization
Stylized image
using a text instruction.
20

---

Published as a conference paper at ICLR 2026
Table 7: Datasets Used Across Agent-X Environments
Environment
Datasets
Autonomous Driving
Cityseapes (Cordts et al., 2010), BDD100K (Yu et al., 2020), Nighttime Driving (Wood, 2020), nuScenes (Caesar et al., 2020)
Security and Surveillance Surveillance Camera Fight Dataset (Altt et al,, 2019), Anomaly Detection (Sultani
et al., 2018), Abnormal Events (Lu et al., 2013), AI City Challenge 2023 (Naphade al., 2023), YouTube-hosted surveillance footage
Math Reasoning
MathVista (Lu et al, 2023), A12D (Kembhavi et al,, 2016), Math Program Syuthe- sis (Ling et al,, 2017), Math Training (Cobbe et al., 2021)
Wesh Navigation
Mobile Interfaces (Sunkara et al., 2022), WebArena (Zhon et al., 2023)
Sports
SocrerNet (Deliege et ol., 2021), Sports-1M (Karpathy et al., 2014), Ego4D (Grauman et al., 2022)
Generie Visual Reasoning
COCO (Lin et al,, 2014), Visual Genome (Krishna et al., 2017), GTA (Wang et al., 2024)
C METRICS AND IMPLEMENTATION DETAILS
Hardware and Setup. All model evaluations were conducted on a single NVIDIA A100 GPU (40GB), ensuring consistent hardware conditions across experiments. Each evaluation run processes visual inputs and queries through the selected LMM agent, which intoracts with predefined set of tools using standardized reasoning framework.
Tool Call Execution. We implement tool calls as callable Python functions with strict input/outpin schemas. We simulate tool-augmented reasoning for each model using model- generated reasoning steps formatted in JSON. Tools are executed sequentially based on the model's output. A tool call is considered successful if it executes without input formatting errors, tool mismatch, or empty outputs.
Metrics Overview. As outlined in Table 3, Agent-X uses fine-grained evaluation metrics across three modes:
Step-by-Step Mode: Evaluates intermediate reasoning quality using metrics such as Grounding Score, Tool Precision, and Tool Accuracy.
Deep Reasoning Mode: Assosses coherence and factual alignment of multi-step reasoning via Foithfulness Accuracy, Contert Score, Factual Precision, and Semantic Accurcy.-
Captures final task correctness through Goal Accuracy, Goal Accuracy/ImgGen and Toolsel Accuracy.
Outcome Mode:
Evaluation Pipeline. Each model's outputs are parsed and compared against verified ground truth reasoning traces. Metries are computed using both exact matches (c.g., for tool names) and similarity-based scoring (e.g., cosine similarity for final answers). Tool failures, including invalid calls, mismalches, or missing outputs, are logged and included in failure rate statistics.
Reproducibility. All scripts, model APIs, tool definitions, and formatted JSON outputs will be made available in our public repository to enable consistent evaluation and replication of results.
C.1 SOURCE DATASETS
The environments Agent-X are constructed from a diverse set of publicly available datasets spanning autonomous driving, security and surveillance, mathematical reasoning, web navigation, sports, and generic visual reasoning. These datasets provide the multimodal contexts (images, videos, and diagrams) from which tasks are derived, ensuring both diversity
21

---

Published as a conference paper at ICLR 2026
and realism. In particular, they enable the benchmark to capture real-world challenges such as object detection in complex scenes, anomaly recognition, multimodal mathematical problem solving, interactive navigation, and fine-grained visual reasoning.
Table summarizes the datasets used in each environment along with their references. Each dataset was selected to contribute unique scenarios that demand perception-groundee reasoning and tool-based interaction, aligning with the agentic goals of Agent-X.
C.2 ERROR CATEGORIZATION METHODOLOGY
To compute the detailed error breakdown in Table 11, compared model-predicted outputs against the ground truth annotations provided in the benchmark JSON files. Each sample consists of a sequence of reasoning steps and final answer. We classify errors into three broad categories: Sicoss:Planna Planning, Format, and Reasoning, based on discrepancies observed during step-by- sypep comparison:
Planning Errors: These oceur when the model either produces no valid reasoning steps and final answer (No action, response) reaches answer without taking any actions (Whole response is model thought). These were detected by checking for missing empty tool usage and absence of concrete outputs.
Format Errors: These include structural issues such as:
Invalid JSON format in argument specification, where the tool call ar- guments could not be parsed did not conform to the tool schema.
Multiple tool calls in a single step, where more than one tool was in- voked simultaneously, violating the single-call constraint.
Reached final answer with format errors, where the model reached answer but lacked required fields (e.g., value, justification) or deviated from the expected JSON schema.
These were identified using schema validation scripts and regular expression checks.
Reasoning Errors: These include semantic mistakes in visual understanding or spatial reasoning. They were detected using keyword matching between the GT and model outputs. Specifically:
Misinterpreting visual content was flagged when the described objects or attributes significantly differed from ground truth labels.
Incorrect spatial reasoning included errors idemifying relationships like position, count, or arrangement.
Error counts were aggregated over all examples for each model. Total error percentages were normalized by the number of failed or erroneous reasoning steps to ensure fair comparison across models.
Mitigating Evaluation Bias. Although Agent-X relies on automatic graders for efficiency, explicitly address potential evaluation bias through multi-pronged protocol that combines diverse automatic judges, human verification, fwas-aware metric design, and full reproducibility. First, every prediction is scored not only by our primary GPT/4o grader but also by an open-source LMM, all released under permissive licences and executed entirely on-premise. These models, which were not involved in data creation, preserve the same relative ordering of systems as GPT4o, demonstrating that our headline trends are grader- agnostic. Second, we conduct a human study on stratified subset of 100 tasks (about 18% of the benchmark), scored independently by four expert annotators who then reconcile disagroements shown in Supplementary Material. Human Goal-Accuracy and Grounding scores correlate strongly with GPT4o grading, and the small set automatic-grading errors is uniformly distributed across models, indicating systematic favouritism. Third, our metrics themselves are designed to reduce bias: we separate format compliance from semantic correctness, normalise tool arguments to avoid spurious mismatches, and use seaiidence-weighter aggregation for subjective queries, falling back to a human vote below threshold. Finally, we will publicly release all evaluation scripts, prompts, and grading
22

---

Published as a conference paper at ICLR 2026
logs, enabling researchers to rerun the pipeline with alternative judges, audit borderline cases, and reproduce every figure the paper. Under these bias controls, relative model rankings remain unchanged, absolute scores shift average (within the reported confidence intervals), and failure-mode distributions vary, providing strong evidence that our conclusions robust rather than artefacts of particular evaluation setup.
Leakage & Data- Genertonn Integrity. A legitimate concern that tasks initially seeded by LMM might advantage that same model family at evaluation time. We therefore executed a four-stage geakag--itigation protocol:
Version isolation. Task seeds were produced with GPT-40 using a closed prompt template. All evaluation is performed later with entirely different checkpoints (e.g. GPT-4o, Qwen2.5) that had no exposure to the prompt or the generated drafts.
2. Aggressive human rewriting. Each seed query and reasoning trace rewritten by four independent annotators who were instructed to "retain the task spirit but discard every exact phrase from the draft." Less than 7 % token overlap remains between seeds and final prompts.
Human quality-assurance (QA). All final tasks passed a threo-way review: content review, (ii) tool-integration review, and (iii) language review.
Taken together, these safeguards ensure that models are evaluated on content they have not seen, while the extensive human rewriting and QA eliminate verbatim near-verbatim leakage. We therefore believe that Agent-X provides a fair and realistic test bed rather than memorisation probe.
Table 8: Results Agent X with additional models (GPT- and Qwen-14B (Bai et al., 2023a) as judges). We report performance across three evaluation modes: STEP- BY-STEP, DEEP REASONING, and OUTCOME. Metrics: G, (Grounding Score), T (Tool Precision), Tace (Tool Accuracy), Face (Faithfulness Accuracy), C, (Context Score), Fp (Factual Precision), Sace (Semantic Accuracy), Gaee (Goal Accuracy), G (Goal Accura- cy /v/mgGen), and Ts ace (Tbolset Accuracy). The best value in each column is in bold, the second best. is underlined.
Step-by-Step
Deep Reasoning
Outcome
Model
G,
Tace
F
C2
F
Sace
Guee
G;
T
GPT as judge
Pixtral-12B (Agrawal al., 2024)
0.12
0.55
LLMM-3.2-11BB-1ee (Grastafiori al., 2024)
0.03
0.15
0.14
0.70
0.08
0.70
0.24
0.07
0.28
0.42
BLni--L--ABB-Taae 2PL00-0v18-7B-240777 (Ye et al., 2024)
(Team al., 2025)
0.26
0.19
0.59
0.62
0.42
0.52
0.65
0.29
0.29
0.48
0.10
0.14
0.30
0.49
0.25
0.32
(1.17
0.11
0.26
0.50
Qwen- 14B as a judge
Pixtral-121 (Agrawal al., 2024)
0.30
0.50
0.68
LLAMA-3.2 2------ii----iiioo (Grataaiori al, 2024)
0.16
0.06
0.12
0.49
0.17
0.74
0.20
0.10
0.11
0.15
Muna-VVL-43B-Taiakiing (Team al., 2025)
0.17
0.20
0.59
0.79
0.64
0.68
0.74
0.35
0.60
0.62
BPL0G-OW33-TB-266 (Ye et al., 2021)
0.30
0.11
0.31
0.39
0.18
0.48
0.56
0.16
0.45
0.48
D EVALUATION OF ADDITIONAL MODELS
Table 8 extends benchmark with four open source! vision LMMs (Pixtral-12B. LLaMA-3. 1-....21111 2-11B-Vision, Sini--L-A3B-Thineee and mPLUG-Or13-7B)and reports their scores under two independent evaluators: the proprietary GPT-4o (Hurst al., 2024) ("GPT judge") and the open-source Qwen-14B 14B (Bai et al., , 2023b) ("Qwen judge"). Three main takcaways emerge.
All systems come from publicly released weight checkpoints that can be run locally; did not fine-tune them for Agent-X.
23

---

Published as a conference paper at ICLR 2026
(i) Cross-judge consistency. The GPT and Qwen columns largely agree the relative strengths of the newcomers. Pixtral-128 achieves the strongest tool handling metrics (Taces Tp) according to both judges, whereas LLaMA-3 2.....1B--iiiiii performs best factual precision (Fp) no matter which one evaluates it. Although absolute values fluctuate (a known consequence of LLM-scoring variance), ranking reversal is observed.
(ii) Kimi VL is the most balanced model. Under the Qwen rubric, Kimi obtains the highest G5 Tp, Faces Saces Gase and the best macro loolsel-accuracy (Tace 0.62). Even the stricter GPT judge still awards Kimi the second-best scores for grounding and semantic accuracy, confirming that Kimi-VL is presently the most well-rounded openly available model on our benchmark.
(iii) Room for improvement. Despite isolated strengths, none of the four recent releases comes close to the closed-source baselines (Table 4): Oulcome scores, particularly Goons remain below 0.35, signalling difficulties in converting partial reasoning successes into fully correct end answers. This gap, detected by both judges, underlines the importance of future research on end-to-end robustness rather than isolated tool-call accuracy.
Summary. The additional comparison demonstrates that our evaluation pipeline can casily accommodate new models, and the insights drawn remain correct: (a) open-source systems stiil trail behind state-of-the-arl closed models, yet (b) certain communily releases (in our case Kimi-VL and Pixtral) already display competitive performance on individ sub-metrics, suggesting the gap narrowing. The near-identical conclusions reached by proprietary (GPT) and an open (Qwen) judge testify to the robustness of the analysis.
IMAGE-BASED VISION LANGUAGE MODELS
Qualitative analysis of additional open-source VLMs. We probed six recent image-based vision Sssionlanguage models: LLaVA-OneVision LLaVA-NeXt. CogVLM, Yi-VL-6B, DeepSeek-VL, and Fuyu-8B, with the same -S0N-coostraiined, tool-augmented evaluation protocol used throughout this study. Each system was required to reason step-by-step, call tools only when justified, and emit nothing but a valid JSON object. The results were uniformly disappointing:
Instruction non-compliance: LLaVA-OneVision and LLaVA-NeXt reproduced the entire system prompt verbatim, omitting the mandatory keys query, reasoning_steps, and final_answer,
Tool hallucination: Yi-VL-6B and DeepSeek-VL invented calls to non-existent ObjectCounter APT and returned hard-coded object lists, despite never being presented with tool catalogue.
Semantic errors: CogVLM aborted the procedure after single line ("Final answer: No other steps"). leaving the query unresolved, while other models produced price estimates that bore no relation to the visual input.
Output gibberish: Deepseek-V1, and Fuyu-8B emitted stray quotation marks and Markdown fences, demonstrating fragility when confronted with nested or unconventional formatting.
These failure modes underscore three persistent weaknesses of current open VLMs: (i) poor format fidelity, (ii) unreliable tool-oriented reasoning, and (iii) limited visual grounding and conteztual understanding. The models appear over-fitted to entrenched demonstra- tion patterns; even slight deviations in the evaluation template trigger hallucinated tools, unsupported assertions, or syntactically invalid outputs.
E CROSS-JUDGE CONSISTENCY STUDY
Table (automatic scores produced with the Qwen-14B judge) and Table 4 (the same protocol scored by the more powerful but proprietary GPT-4o judge) paint almost identical portraits
24

---

Published as a conference paper at ICLR 2026
Table 9: Additional results on Agent-X (Qwen-14B (Bai et al., 2023b) as judge). We report performance across three evaluation modes: STEP-BY-STEP, DEEP REASONING, and OUTCOME. Metrics: G2 (Grounding Score), Tp (Tool Precision), Tace (Tool Accuracy), Face (Faithfulness Accuracy), Cs (Context Score), Fp (Factual Precision), Sace (Seman- lic Accuracy), Gace (Goal Accuracy), G (Goal Accuracy/Iie /Imggen)) and Tece (Toolset Accuracy). The best value in each column is in bold, the second best is underlined.
Step- by-Step
Deep Reasoning
Outcome
Model
G2
Tp
Tacc
F
C₁
P
Sace
Gaoe
G
T acc
Open-source
Phi-e-V--Instruct (Abdin et al,, 2024)
0.27
0.11
0.32
0.51
0.39
0.59
0.46
0.16
0.35
0.39
InteraVL2. 5-88 (Chen et al,, 2024a)
0.38
0.16
0.49
0.63
0.51
0.61
0.55
0.29
0.53
0.53
Geama-3-4B (Team et al., 2024b)
0.50
0.24
0.67
0.74
0.66
0.59
0.74
0.30
0.68
0.68
InternVL3-88 (Zhu et al., 2025)
0.41
0.16
0.51
0.71
0.61
0.60
0.69
0.23
0.51
0.62
VideoLLaMA3-78 (Zhang et 2025a)
0.39
0.15
0.40
0.68
0.56
0.60
0.68
0.27
0.53
0.56
Qwen2. 55VL77 (Bai al., 2023c)
0.51
0.27
0.63
0.77
0.66
0.61
0.77
0.37
0.62
0.67
Closad-source
Semini-1.5-Pre (Team al., 2024a)
0.57
0.36
0.80
0.82
0.73
0.76
0.63
0.05
0.77
0.71
Gemini-2. 5-Pro (Team ul., 2023)
0.63
0.40
0.84
0.86
0.76
0.80
0.83
0.50
0.74
0.72
CPT-4o (luust et al,, 2024)
0.46
0.27
0.63
0.72
0.59
0.75
0.69
0.44
0.48
0.56
OpenAI-od- (OpenAI, 2025)
0.63
0.35
0.78
0.79
0.88
0.58
of the current landscape Agent-X. Below, we walk through the evidence along the three evaluation axes and demonstrate that our conclusions remain unaffected by the choice of evaluator.
Across the DEEP REASONING (Faces C., Fp, Sace) and OUTCOME (Gaee; G Tace) axes, the two independent evaluations, one powered by the open-source Qwen-14B judge and the other by the closed-source GPT-4o judge, paint an essentially identical picture: ope -i-mmni dominates every reasoning metric and remains first (or, worst, close second) in every final-answer metric; GPT-4o consistently eclipses OpenAI-o4-min on factual precision and semantic accuracy; and, the open-source camp, Qwen2. 5-VL-7B reliably outperforms InternVL3-89 and VideoLLaMA3-7B.
Consistency across evaluators. Taken logether, these observations satisfy the three desiderata set out above:
(a) Relative ranking: no pair of models swaps positions ions between the two tables on more than one metric.
(b) Highlighting of leaders: the same entries appear in bold or underline across judges, flagging identical winners and runners-up.
(c) Small absolule deviations: score deltas rarely exceed 0.06, confirming robustness.
Pul differenlly, both evaluators tell the same story: closed-sowrce systems still lead, but open-soure source contenders (especially the Quen family) are rapidly closing the gap, validating the central thesis of our paper.
Implications. Because Qwen- 14B is open-source and deterministic, its near-perfect align- ment with GPT-4o demonstrates that our benchmark can be audited and reproduced without proprietary APIs. Conversely, the agreement validates our earlier claims that (i) closed-source agents still dominate, yet (ii) top open-source models such as Qwen2. 5-VL-7B are closing the gap. The consistency across evaluation protocols therefore, strengthens the empirical foundations of our conclusions.
Agreement on the global winner: Both judges agree that Samini-2.5-Pre is the overall champion. When scored by the open-source Qwen-14B judge it achieves the highest values for Taees Fp. Sace and T the GPT-4o rubric yields the same outcome, with Gemini-2.8 5-Pro either retaining first place or falling only marginally to second. This convergence supports claim that Gemini -2. 5-Pro currently sets the bar for vision-centric agents Agent-X.
25

---

Published as a conference paper at ICLR 2026
Table 10: Results on Agent X with human evaluation. We report performance on 50 Agent-X tasks across three evaluation modes: STEP-BY-STEP DEEP REASONING, and OUTCOME. Metries: G. (Grounding Score), Tp (Tool Precision), Tace (Tool Accuracy), Face (Faithfulness Accuracy), Cs (Context Score), Fp (Factual Precision), Sace (Semantic Accuracy), Gaee (Goal Accuracy), G (Goal Accuracy / Inggeen) and T Tace (Toolset Accuracy). Best values are in bold; second best are underlined.
Deep Reasoning
Outcome
Model
Тр
Tace
F Face
C,
Fp
Sacc
Gace
G
T acc
Open-source
VideoLLaMA3-78 (Zhang al., 2025a)
0.44
0.40
0.66
0.68
0.55
0.67
0.68
0.52
0.61
0.54
InternVL3-88 (Zhm al., 2025)
0.50
0.50
0.68
0.71
0.66
0.70
0.68
0.58
0.83
0.63
Qven2. 5-VL778 (Bai et al., 2023b)
0.59
0.62
0.82
0.76
0.67
0.78
0.77
0.68
0.85
0.70
Closed-source
S.nini-2.5-P22 (Team et al,, 2023)
0.74
0.66
0.86
0.84
0.70
0.87
0.82
0.75
0.75
0.76
GPT-4o (Hurst al,, 2024)
0.43
0.39
0.66
0.65
0.54
0.66
0.67
0.55
0.47
0.53
OpenAl systems show identical trends: Across both evaluations OpenAI-o4-mini secures the top Tace score (0.86 vs. 0.89), while GPT-4o leads in G. and Fp. Although small mumerical variations arise from different random seeds inside each LMM judge, the relative ordering is preserved, confirming the stable performance gap between the two OpenAI releases (Hurst et al., 2024; Achiam et al., 2023; OpenAT, 2025). Open-source models follow the same pattern: Community systems display similar consistency: Qxen2.5-VL-7B ranked first or second by both judges for G,, Tp. Face and Ta whereas InternVL3-8E systematically outperforms VideoLLaMA3-7B. Absolute scores differ slightly, but no pairwise ranking is ever inverted.
Why use Qwen-14B as an auxiliary judge? Because Qwen-14B is fully open-source, deterministic under fixed hyperparameters and open license, our entire evaluation becomes reproducible. The near-perfect concordance with the proprietary GPT-4o demonstrates that a transparent and inexpensive alternative can deliver equally reliable verdicts, allowing other rescarchers to replicate paper, most notably that precise tool use remains the key bottlencck, where closed-source models still lead (yet the Quen family is closing the gap), are fully corroborated by both judging protocols. The consistent rankings underscore the robustness of our conclusions.
numbers without API costs. The qualitative insights of the main
Complementary human study. Table 10 reports scores assigned by two human graders who independently inspected all 120 test episodes, then reached consensus after discussion. Three observations stand out.
1. Macro-level agreement with automatic judges. The human ranking exactly reproduces the pattern seen with Qwen-14B and GPT-4o evaluators: Gemini-2.5 5-rro > (wen2.5-VL-78 > InternVL3-8B > VideoLLaMA3-7B. Likewise, humans confirm the every STEP-BY-STEP metric and the global
sizcable lead of Gemini-2.5 5-roPr OUTCOME columns (Gace and T.
2. Consistent strengths of the Quen family. Human judges reward 5 with the highest G (0.85) and the second-best. scores for six additional columns, echoing the automatic evaluation, where Qwen dominated grounding and faithfulness metrics. This suggests that Qwen's open-source architecture translates into genuinely strong visual reasoning, not merely alignment with a particular L.L judge.
3. Why absolute numbers are higher. Human annotators can interpret graphics, verify OCR text and ignore benign formatting glitches, capabilities absent from current LLM judges. Consequently, perfectly valid tool calls that were penalised as "ill-frmatted"" by antomatic scripts are credited by humans, pushing absolute scores upward (c.g. Tace rises
'The annotators followed the same rubric used by the automatic judges, but were allowed to replay tool outputs and visual inputs for verification; Appendix $ H.
26

---

Published as a conference paper at ICLR 2026
Table 11: Error Breakdown Across Models. Common planning, formatting, and
reasoning mistakes on Agent-X, shown separately for source and closed-source systems.
Firee Type
Pistrai-158 Agronai et 201l VideolzaMaa-7a plavsy (6604) Queed 5-11-T8 (1ki
-PLAb-buia (Vo 2101)
Pesiing
No rriki, wirk miqure thought
1a (1855)
1808 aggasatee
112
(32
<2K
755 136 (81)
yhu
Frxal srew regiteal benee
(3 (9-406)
3 (1.009
79M (3k 192)
Reveslig
becmel
resal adoggemaat
300 (93A0))
(02
091)
06.R1
(6-55)
Chves--vane vuslele
Firse Type
GPT-4k (frasd
Denses-1. [leass (006))
SpecAE (UppecA 2001)
Demins --.5- (Vloss
No crina,, zogetar
5 (00228)
IM (URR))
92 (2.4R)
Finsating erter
books 180N
789 (44.85)
(SL.53) (SNIPS)
S2N (48 STR)
188 (6.75)
122 (88 (18)
710 (3) (XL
Mutresigg emons
Wirze sieal reeeper
162 (80)
S41 (38 (85)
S4 (6 '81
(a ER01 THK (1005)
Total eireee
891 (UM65))
1185 (1585)
S58 (5074)
from 0.84 to 0.86 for Gemini-2. 2-...-r.)))) Crucially, the relative ordering is unaffected, reinforcing the robustness of our conclusions.
Taken together, the human study corroborates the main paper's message: closed-source agents still set the bar on Agent-X, yet modern open-source models, particularly the Qwen line, are rapidly closing the gap, trend observed by all three independent evaluation protocols.
F ADDITIONAL ERROR ANALYSIS
F.1 QUANTITATIVE ANALYSIS
Table 11 dissects the failure modes of eight representative agents and surfaces three systematic trends. (1) Format fidelity as the open source ceiling. Fully 57% of Pixtral's and 60% of mPL.G-OW133 errors are pure syntax violations, missing braces, double tool calls, or free-form prose where a JSON key is required. These errors are fatal for any downstream pipeline because they prevent the trace from even executing. VideoLLaMA mitigates the synlax issue, but the extra "safety buffer" shows up as planning laxily: roughly one in twenly traces never leaves the "thought" stage and thus contribute no real work. Qwen 2.5 the only community model to keep format errors below the 30 % mark, yet it still trips on fine- aneegraiied grounding, miscounting objects, swapping left fftt/rigtt in spatial references. (2) Closed trade-offs. Gemini-1.5-Pa sacrifices precision for recall: it almost never refuses task, but its cagerness translates into the highest JSON error rate of any proprietary system. GPT-4o shows the opposite bias; its structured output is tidy, yet more than one-third of its slips involve misidentifying entities, suggesting the vision adapter rather than the language head is the new bot bottleneck. The lighter o4-mini and Gemini-2.5 familics replicate their larger siblings' error profile, confirming that these behaviours stem from architectural choices rather than sheer scale. (3) Silent refusal as research risk. "no-action / no-response" outcome is equivalent to a system crash an autonomous workflow. Although rare for commercial models (S 0.3 %), the phenomenon still affects double-digit percentages of open-source traces. Eliminating this headroom could immediately yield a significant boost in end-to-end success without touching visual reasoning at all.
Implications. Collectively, the numbers echo the qualitative findings of $4: state-of-the- art closed models win primarily through reliable structure, not mystical reasoning leaps. For open-source efforts, investing in strict output proteprocessors, JSON-aware RLHF, or titema-guided decoding may be the fastest path to parity. Conversely, commercial systems now face diminishing returns from format tuning and must push on visual grounding and multi-turn consistency to stay ahead.
27

---

Published as a conference paper at ICLR 2026
F.2 QUALITATIVE ANALYSIS
Figures and reveal two distinct error profiles. Figure shows that Qwen2. Swen2.5-VL-7B often hallucinates tools (e.g., fake ObjectCounter) and then builds confident but unfounded ex- planations on top of those hallucinated outputs. Repeated use of the generic SceneDescriber across video frames further crodes temporal grounding. Figure 5 highlights InternVL3-BB While adheres to the required step-tool format, it miscounts objects, redundantly reinvokes tools, and contradicts its own earlier steps, symptoms of weak short-term memory rather than hallucination. Together, the examples confirm our quantitative results: current open- source VLMs struggle either with tool fidelity (Qwen) or cross-step coherence (InternVL), underscoring key directions for future work.
Prompt
You are an intelligent multi-modal agent. You are provided with: A text query <query> An image or video <files> A set of tools to assist with your reasoning with meta data of tools given as follows: [meta, dataj. Your objective is to answer the query based on the given visual content by choosing and using the most appropriate tools. You must reason step-by-step. Each reasoning step should include:
[f'task": "Describe the sub-task being performed.",, {"taoi_used": "Specify the tool selected and justify its choice,") ["tool_ output": "Provide the tool's output.", ("thought": explained.", "f.....'fiiall, output": ("final annserr: "Provide a clear and concise answer based on all previous steps", "justification:: "Provide justification for the final answer.")
"Ensure each step is self-contained and clearly
InternVL3-8B
Input Files: AgentxX AgentX_612.av) 612.avi
Query: "How many different types of vehicles are seen in the video? How many of them are already parked?"
Error Response
and determine if they are parked or moving.". "tool_used": 'ObjectCounter', "tool_output": "The image contains 2 motorcycles and cars. motorcycles parked.", "thought": 'The initial counting helps determine types and of of distinct vehicles, necessary for confirming parked vehicles."), ('task": 'Analyze vehicle locations being parked not,", "tool, used": "SceneDescriber', "tool_outpin _otpptttptt "There are motoraycles along the walkway and cars near entrance, indicating they are parked.", "thought"; scene description confirms vehicles are in parked condition. 'Il.
beaasoning_stes"" [f'task": "Identify count different types vehicles in
"final answer": ("vehicle I"vehicle.types": ("motorcycles", "cars"], "number, distinct_vehickeee "parike_vehhilee""
vehicles": 2, justiiicatton": "The ObjectCounter tool identified counted the vehicles, and SceneDescriber clarified their parking status, providing complete details needed for the query."3
The response format
"reasoning_steps" [["step": 1, "task": "Get a general idea the scene the video.", "tool": "SceneDescriber"" "tool_ cotoll_utpttttt "The scene shows a courtyard with several motorbikes, pickup van, and car parked and a group of people interacting around tables.", "thought": "Understanding the general context of the images will help identify distinct vehicles and they are parked or moving. ("sstepp"""""ttp"" 2, ..., ("step": 3, ...), ["step": 4, ...,, ("step": 5, ...,, ['step": 6, ...J. ["step": 7, ..3, ["step": 8, "task": "Verify vehicle presence throughout the video." "tool": "ObjectCounter", "taol_oee ooupptt: "2 matoraycles, pickup van, 1 car", "thought". "Canfirming that the all the vehicle count and types remain constant throughout the video finalizes the analysis."].
"final answer": ["value"; '4: 2 motorcycles, pickup van, car, all parked", "justification": "Using scene description, object counting, and region description, we identified parked motorcycles, carry or pickup van, and car consistently across all videa.")
Incorrect final
Incorrect json format
Shallow reasoning
Qualitative comparison of InternVL visual reasoning tasks from Agent-X. InternVI. generally follows the correct step structure but exhibits internal inconsistencies in final answers and justifications. It occasionally miscounts objects and repeats tool usage redundantly, revealing challenges in cross-step memory and consistent entity tracking.
Figure 5:
28

---

Published as a conference paper at ICLR 2026
Prompt
You are intelligent multi-modal agent. You provided with: A text query <query> An image video <files> A of tools to assist with your reasoning with meta data of tools given as follows: mntta_data]. Your objective is to answer the query based on the given visual content by choasing and using the most appropriate tools. You must reason step-by-step. Each reasoning step should include: "reasoning_step_foee formatt: [f'task" "Describe the sub-task being performed."i. ["tool_used" "Specify the tool selected and justify its choice." "). I'tool_output": "Provide the tool's output.",, ], ('thought": ..... "Ensure each is self-contained and clearly explained." "......finnl__utty" output": ["final_answer answer": "Provide a clear and concise answer based on all previous steps', 'justification:: "Provide justification for the final answer."]
Qwen2.5-VL-78
Input Files: SigentX116.jppp
"Which color line shows the greatest overall change in the y-direction and approximately what is its average height across the chart?"
Query:
Error Response
"reasoning_stepp" :ttrppi: ("step": "task": 'Identify the line that grows most rapidly and first overtakes all others value.", :ooo_" "CodePlotter", "tool_output": "The Cornflower line grows most rapidly and first overtakes all others in value.", 'thought": "By analyzing the graph, the fastest and surpasses the others."]. f'step:: 2, "task": "Calculate the approximate average height of the the chart.' "tool_used": "CodePlotter", "topp_outtut": "The approximate average height the chart around 35.", "thought": "Using the CodePlotter tool, we calculate
visually identify which line increases
Cornflower line of the Cornflower line the average height of the Cornflower line by integrating the area under the and dividing by the length of the x-axis." 'J
f"value": 'The Cornflower line grows most rapidly and first overtakes all others in value, with approximate average height 35 the chart., "justification": "The Cornflower line identified the growing most rapidly and surpassing others due its steep slope and early lead. The average height calculation provides quantitative measure of its performance throughout the chart.")
The correct response format
the chart identify the trends of different colored lines", "tool": "SceneDescribee", outppttt 'The image line chart with different colored lines.", "thought": "The SceneDescriber confirms the image is line chart and identifies the calor-coded lines, which is necessary to analyze their trends."), "J, ["step": 2, "task": "Extract the y-axis values understand the numeric scale and support trend comparison.", "tool": "OCR", "output": 'f'text': [['value:: 'Cornflower "thought": 'The OCR extracts y-axis values to establish numerical scale and matches line labels to their corresponding colors, enabling meaningful comparison of changes across lines."), ("step": 3, "task": "Determine the minimum and maximum values for each line calculate overall change.", "toal": "SceneDescriber", "tool_ Tooolouutut".: "Cornflewer ranges from 17 to 42, Bubblegum goes from 17 to Orange Red keeps it tight from 18 to 17. Medium Orchid from 17 to 27. Orange shoots from 39 47. Brown starts at 17 and climbs to 32.", 'thought": 'The RegionDescriber retrieves the minimum and maximum values of each line to calculate their overall change."), ("step": 4, "task": 'Estimate the average height af the fastest-growing line across the chart.", "tool": "Calculator", "tool_ output": "The average height of the Cornflower line is approximately 30 the y-axis,, "thought": "The Calculator computes the change and average height of each line, allowing with the greatest overall change and report its average position on the chart."]..
"reasoning_stept [["step": "task": "Describe the
identify the one
"final_and answer": !"value": "Cornflower, 30', "Justification": "The line chart was analyzed to compare the overall change in each line. Cornflower showed the greatest change, and its average height was calculated to be approximately 30.']
Incorrect final answer
Wrong tool call
Wrong usage of the tool, CodePlotter a generative tool that outputs the image of plot
Figure Qualitative comparison of Qwen2.5 on visual reasoning tasks from Agent-X. Qwen2.5 often hallucinates tool behavior and produces overconfident justifications without numerical evidence. It struggles with temporal reasoning, overuses general-purpe tools like SceneDescriber, and fails to maintain consistency between reasoning steps and final answers.
29

---

Published as a conference paper at ICLR 2026
Table 12: Run-to-run variance analysis for (a) Qwen2.5-VL-771. (b) InternVL3-8B,
and (c) Gemini-2.5-Pro.
Run Metric
G,
Tp
Tacc
acc
C
S acc
G
G
T ser
(a) 1ren2.5-VLL-771
Run
0.51
0.27
0.63
0.77
0.66
0.77
0.37
0.62
0.67
Run
0.48
0.25
0.66
0.77
0.65
0.65
0.77
0.38
0.61
0.66
Run 3
0.48
0.26
0.66
0.76
0.65
0.65
0.76
0.38
0.62
0.66
Variance (%)
0.020
0.0067
0.020
0.0022
0.0022
0.0022
0.0022
0.0022
0.0022
0.0022
(b) InternVL3-8B
Run
0.41
0.16
0.51
0.71
0.61
0.60
0.69
0.23
0.51
0.62
Run 2
0.36
0.15
0.50
0.62
0.50
0.62
0.56
0.20
0.51
0.52
Run
0.36
0.15
0.50
0.62
0.50
0.62
0.56
0.20
0.50
0.52
Variance (%)
0.0556
0.0022
0.0022
0.1800
0.2689
0.0089
0.3756
0.0200
0.0022
0.2222
(c) Gemini-2.5-PM9
0.63
0.40
0.84
0.86
0.76
0.80
0.83
0.50
0.74
Run
0.62
0.38
0.86
0.85
0.75
0.82
0.84
0.49
0.73
0.71
Run 3
0.61
0.37
0.86
0.85
0.75
0.82
0.84
0.50
0.73
0.72
Variance (%)
0.0067
0.0156
0.0089
0.0022
0.0022
0.0089
0.0022
0.0022
0.0022
0.0022
Table 13: Inter-judge agreement across GPT-40, Qwen- 14B, and human judges.
Statistic Metric
G
T acc
F
acc
Sace
Gace
Pearson (r)
0.34
0.96
0.32
0.72
0.98
Spearman (p)
-0.07
0.99
0.39
0.63
0.96
Mean Dev.
0.13
0.05
0.09
0.11
0.04
Cohen's (R)
0.62
0.83
0.09
0.31
1.00
G ADDITIONAL RESULTS AND ANALYSIS
This section presents comprehensive analysis of the proposed evaluation pipeline across mul- tiple large multimodal models (LMMs) and judges. We report results from multi-run variance studies, inter-judge agreement, automated evaluation outcomes, and reasoning perception ablations.
reasoning-perooptionn
G.1 RUN-TO-RUN VARIANCE ANALYSIS
To ensure the reliability of aulomated evaluation setup, we perform a run-to-run consistency check on three representative models: Qwen2.5- VL-7B, Intern VL3-8B, and Gemèni-2.5-PM Table 12 summarizes the variance across three independent evaluation runs for each model. Overall, the variance remains minimal across all metrics, confirming the stability of our pipeline. For instance, Qwen22.5-VL-7B and Gemini-2. 2.5-Pro both exhibit less than 0.02% deviation across metrics, whereas Intern VL3-8B shows slightly higher fluctuation (up to 0.38%) in semantic accuracy (Soco). A high degree of determinism is therefore ensured across runs. A consolidated summary in Table 14 further highlights that Gemini-2 2.5rini-2.55- 2.5-Pro and Qwen2.5- VL achieve near-identical results across trials, confirming that the observed differences occur only beyond the third decimal place. Intern VL33-B exhibits minor instability, attributed to its stochastic sampling process during decoding, but the overall ranking remains consistent across repetitions.
G.2 INTER-JUDGE ACREEMENT
To assess the robustness and reliability of our evaluation framework, compare the judgments produced by three distinct ovaluators: GPT-4o, Qwen-14B, and human experts. Table 13 presents four complementary agreement statistics, Pearson correlation (r), Spearman rank correlation (p), mean deviation, and Cohen's K, computed across five representative metrics (G,: Tarc: Faces Suce, and Gaee).
30

---

Published as a conference paper at ICLR 2026
Table 14: Summary of multi-run variance analysis using Qwen-14B as judge.
Model
Max Variance (%)
Observation
0.02
Stable all metries; differences appear only the 3 decimal place. Slightly higher flnrmaation in semantic acmuracy, but consistent overall ranking. Practicaly identical scores acroes runs, confirming evalustion determinism.
InternVL.-88
0.38
Gem121-2.5-Pro
0.02
High Alignment with Human Judgments. The results reveal remarkably strong consistency between automated judges and human evaluators, particularly in task accuracy (Tace) and global accuracy (Goco). Both dimensions exhibit near-perfect correlations (r 0.96, p = 0.99 for Tace and r = 0.98, x = 1.00 for Gaee), suggesting that Qwen-14B and GPT-40 capable of replicating human evaluative behavior with high fidelity. These metrics primarily assess correctness and goal achievement, objectively quantifiable properties that leave little room for subjective interpretation. Hence, the high agreement indicates that automated judges can reliably approximate human consensus for structured, outcome-based criteria.
Moderate Agreement in Semantic and Functional Dimensions. For semantic accuracy (Sece) and functional accuracy (Face), the correlations remain moderately high (r 1.32-0.72) reflecting partial but imperfect alignment. These metrics rely nuanced linguistic and reasoning cues, such as whether model responses maintain contextual consistency or functional correctness, where even expert human raters may differ in interpretation. The observed variance thus highlights the intrinsic ambiguity of multimodal reasoning evaluation rather than deficiency in the automated judges.
Lower Agreement in Goal Success. The goal success metric (G.) demonstrates lower correlation values (r = 0.34, م = -0.07), revealing that high-level subjective goals are more difficult to standardize across evaluators. This finding underscores the challenge of open-ended, multimodal tasks, where different judges may focus distinct aspects such as reasoning depth, visual grounding, task completion. It also motivates the need for multi-perspective evaluation frameworks that combine quantitative accuracy with qualitative interpretability.
quantifying "success"
Key Insights. Overall, the results establish that automated evaluation using Qwen- 14B achieves near-human reliability across most quantitative dimensions. The strongest aligument occurs for metrics grounded in explicit correctness (e.g., Tace and Gace), demon- strating that LLM-based judges are highly effective at assessing factual and structural validity. However, metrics involving contextual, semantic, open-ended reasoning still benefit from human oversight. This reinforces the complementary nature of human-AN evaluation synergy: while automated judges provide scalability and consistency, expert human review remains crucial for capturing subtle interpretive aspects of reasoning quality.
G.3 AUTOMATED EVALUATION ACROSS MODELS
We benchmark a wide range of vision laaion-aangage models using our automated judge, Qwen- 14B. As shown in Table 15, GPT-40 achieves the highest overall performance, with strong results in task accuracy (Tace = 0.45) and functional accuracy ( (Face = 0.34), followed by Qwen2.5- 7B- Instrrcct and Intern VL33-88 The bascline Qwen1. Soven15-7B-- 5-B--B performs significantly lower across all metrics, highlighting the benefits of multimodal fine-tuning. Interestingly, Qwen2.5 5- VL performs comparably on low-level metrics such as C. and F, but lags in high-level goal accuracy (Gace). This reinforces that pure visual grounding alone is insufficient. for complex multimodal reasoning.
G.4 REASONING VS. PERCEPTION ANALYSIS
To isolate and quantify the contribution of reasoning to multimodal task performance, conduct an ablation study on representative subset the dataset. Specifically, we compare two configurations: (1) a perception-only model that relies solely on visual understanding
31

---

Published as a conference paper at ICLR 2026
Table 15: Automated evaluation results across models on Agent-X.
Model
Instalign
T Fuce
Argace
Su1aace
F
C
Suce
Guee
Gstar
0.65
0.39
0.11
0.12
0.34
0.27
0.29
0.49
0.33
0.30
GPT-4o
0.72
0.45
0.15
0.12
0.34
0.31
0.36
0.51
0.36
0.35
Qwen1.5-72 5-7B-Chat
0.24
0.09
0.04
0.12
0.00
0.02
0.08
0.35
0.34
0.25
(wen2.5-7B- I-B--nstrutt
0.46
0.29
0.02
0.13
0.15
0.09
0.10
0.39
0.33
0.26
fwen2.5-7B-VV
0.42
0.20
0.14
0.09
0.18
0.32
InternVL3-88
0.49
0.28
0.04
0.12
0.14
0.09
0.13
0.41
0.02
0.02
without explicit reasoning steps, and (2) our proposed reasoning-angmentedd model that
incorporates structured reasoning chains during evaluation. As shown in Table 16, the inclusion of reasoning improves the overall goal accuracy (Gace) from 0.33 to 0.43, a relative gain of approximately 30%.
This improvement reveals that reasoning plays a crucial role in bridging low-level perception with high-level task understanding. While perception-only models can identify visual entities and basic relationships, they often fail in multi-hop inference, causal understanding, or context-dependent decision making. In contrast, models equipped with explicit reasoning chains can decompose complex agentic queries into interpretable intermediate steps, verify intermediate evidence, and better align their final predictions with the intended goals.
These findings validate our design choice of incorporating reasoning- centric evaluation protocols, as they more accurately reflect an agent's true multinodal competence rather than surface-level visual recognition. In other words, the ability to reason explicitly, not just perceive, what allows multimodal systems achieve coherent, explainable, and goal-directed behavior across diverse scenarios.
Table 16: Reasoning vs. Perception Ablation on Subset..
Setting
Gacc
Perception-only (no explicit reasoning) Full reasoning chain (ours)
0.33
0.43
G.5 DISCUSSION
Our analyses collectively establish three key insights:
1. The evaluation pipeline deterministic and consistent: (Tables 12-14), ensuring reproducibility across runs.
2. Automated judges like Qwen-148 14B show strong agreement with human evalua- tions (Table 13), supporting scalable benchmarking.
3. Explicit reasoning significantly contributes to higher multimodal task per- formance (Table 16), emphasixing the importance of reasoning supervision in multimodal agents.
G.6 REASONING CHAIN EXPERIMENTS
Goal Accuracy vs. Question Difficulty. To better understand model performance across varying levels of task complexity, we categorize queries into easy and hard subsets using difficulty classifier based on LMM confidence scores and step trace complexity. As shown in Figure all models perform better easy queries, with noticeable drop in Goal Accuracy for harder examples. For instance, Qwen2. 5 drops from 39% to 31% and InternVL3 from 28% to 14%, indicating that model reliability significantly declines reasoning difficulty increases. Gemini-2.. 2.5 performs the strongest overall, suggesting better robustness across query diffieulty levels.
32

---

Published as a conference paper at ICLR 2026
Average racy (EsoyHard Only)
Anerge Coal Aooreey
02
01
Figure 7: Average Goal Accuracy by difficulty level across models.
Goal Accuracy vs. Reasoning Depth. We also analyze how the number reasoning steps (chain length) affects task success. Figure shows that deeper reasoning traces (i.e., 5-6 steps) generally correlate with higher or more stable Goal Accuracy for strong models like GPT-4o-mini and Qwen, suggesting their improved ability to handle multi-hop reasoning, In contrast, performance for InternVL3 and VideoLLaMA3 dips at deeper depths, exposing their limited capacity for long-horizon reasoning. These findings underscore the importance of both depth-aware reasoning and intermediate step consistency.
Goal Accuracy vs. Reasoning Chain Depth
(PI-ot-mini
GPT4c
Intervv..3 5
06
Videollaa
Average
0.4
0.3
01
Reasoning Stops
Figure 8: Goal Accuracy reasoning chain depth.
Tool Call Success and Failure Analysis. Figure 9 shows the number of successful and failed tool invo- cations per model. successful call executes without format or argument errors. The GPT family achieves the highest success rate (83.8%) relative to its total calls, with GPT-4o demonstrating a strong balance between usage and reliability. InternVL models lead to abso- lutely successful calls but exhibit high failure counts, primarily due to invalid tool usage. shows the highest precision, with only 109 failures out 2241 calls, reflecting robust format compliance. In con- trast, o4-mini the most aggressive, issuing 3374 calls but suffering the highest failure rate (1038). Gemini-
(ree2.5-VL-81)
GF1
Figure 9: Tool Call analysis across LMM families on Agent-X.
33

---

Published as a conference paper at ICLR 2026
3-4B and VideoLLaMA3-7B show moderate tool use, with Gemini-3-4B maintaining a lower failure count. These trends highlight three behavioral patterns: aggressive models (e.g., InternVL3-8B) invoke tools frequently but are prone to errors; conservative models (e.g-, Qwen2.5-VT--BB) prioritize accuracy over volume; and balanced models (e-g.., GPT-4o) combine moderate tool use with high reliability.
34

---

Published as a conference paper at ICLR 2026
H INSTRUCTION FOR ANNOTATORS
Query Construction Guidelines
Input Selection Rules:
Select inpuls (images or videos) from publicly available dalasets to ensure broad coverage real-world scenarios.
Include diverse environments such sports, autonomous driving, web brows- ing, surveillance footage, etc.
Use image files that range from clean to cluttered scenes to test for occlusions and visual complexity.
For video files, ensure that there is meaningful variation motion across frames (e.g., object movement or temporal change).
Query Design Requirements:
Design queries that require at least three distinct reasoning steps.
Each query must involve at least two different tools from the provided tool list (Table 6).
Avoid trivial or single-glance questions; queries must require reasoning grounded in the input.
Do not mention tool names explicitly in the query. (e.g. UUee OCR to read the sign" is nol allowed).
If referencing dynamic or timo-sensitive information, specify a fixed date, time frame, source.
For multi-image queries, ensure that the question requires reasoning across the images and not individually.
For video queries, ensure that reasoning involves content across multiple frames.
Language and Verification Rules:
Write all queries and answers in English.
Ensure that each answer can be verified consistently by human evaluators.
Avoid ambiguous questions that could lead to inconsistent answers.
Figure 10: Annotation instruction document used during the query construction stage.
35

---

Published as a conference paper at ICLR 2026
Reasoning Trace and Final Answer Annotation Guidelines
Your Task:
You are provided with set of image- or vidco-based queries along with tool metadata. Your job is to verify and improve the reasoning steps, final answer, and justification for each sample. Ensure that the steps are logical, human-like, and grounded in the given lools and inputs.
Reasoning Trace Requirements:
Each reasoning trace must contain at least three distinct steps.
Use at least two different tools from the provided tool list.
Each step should inchude:
Thought: What the agent is trying to figure out.
Tool: The tool being used.
Input: What is passed to the tool (e.g., image name, region, text).
Output: What the tool returns.
Steps should form coherent, human-like progression. Each step should follow logically from the previous.
Ensure tool use is appropriate and consistent with its defined capabilities.
Do not invent outputs: use plausible results based on the image and tool functionality.
Final Answer and Justification Guidelines:
The final answer must be consistent with the reasoning steps.
For objective queries, the answer should be accurate, verifiable, and clearly correct.
For subjective or generative queries, the answer should be reasonable, coherent, and grounded in the input.
Avoid vague or incomplete answers: the answer should fully resolve the query.
The justification should summarize the reasoning clear natural language.
Annotation Rules:
1. Do not edit or rewrite the query. Focus only on the reasoning steps, answer, and justification.
2. Use fluent, human-like language for the thoughts and justifications.
3. Make sure each reasoning chain is clear, logically structured, and not overly long.
4. Flag samples if the image, video, or query ambiguous, irrelevant, or unsolvable.
Figure 11: Instruction document for reasoning step and answer annotation.
36

---

Published as a conference paper at ICLR 2026
I AGENT-X TASK EXAMPLES
Example 1
185
fettlager
Dumige
Hocea Oriia
Trown
[Sennt__76.ppp
Which color line shows the greatest overall change the y-direction and approximately what is its average height across the chart?
Query:
Involved Tools: SceneDescriber, OCR, RegionDescriber, Calculator
Steps:
1. Describe the scene in the chart to identify the trends of the different colored lines.
2. Extract the y-axis values understand the numeric scale and support trend comparison.
3. Determine the minimum and maximum values for each line to calculate
overall change.
4. Estimate the average height of the fastest-growing line across the chart.
Reasoning Trace for steps:
1. The SceneDescriber confirms the image line chart and identifies the color-coded lines, which is necessary to analyze their trends.
2. The OCR extracts y-axis values to establish a mumerical scale and matches line labels their corresponding colors, enabling meaningful comparison of changes across lines.
3. The RegionDescriber retrieves the minimum and maximum values of each line calculate their overall change.
4. The Calculator computes the change and average height of each line, allowing us to identify the one with the greatest overall change and report its average position the chart.
Answer: Cornflower, 30
J Justification: The line chart was analyzed to compare the overall change in each line. Coruflower showed the greatest change, and its average height was calculated to be approximately 30.
Figure 12: Example 1
37

---

Published as a conference paper at ICLR 2026
Example 2
AgentX_ 10o0_100.pp
Is this kitchen more likely set up for a cooking class or for a private dinner?
Query:
Involved Tools: SceneDescriber, ObjectCounter, LocataObjecttyText
Steps:
1. Describe the scene to understand the overall layout and setting of the kitchen.
2. Count the number of cooking stations or setups visible.
3. Examine the arrangement and positioning of kitchen equipment.
4. Assess the kitchen's 's design and features to determine its suitability for cooking class or private dinner.
Reasoning Trace for steps:
1. The SceneDescriber tool highlights multiple cooking stations, suggesting layout intended for simultaneous use by several individuals, typical of a cooking class setup.
2. The Counter tool confirms the presence of seven stovetops, support- ing the idea that the kitchen designed for group-based cooking activities rather than a private dinner.
3. The LocateObjectByylex tool is used to find specific items like cut- ting boards in the image, showing that each station is individually ceiipeed-reinfoccinng the sctup's suitability for a cooking class.
4. The overall utilitarian and organized layout, observed through the SceneDescriber, indicates the space is structured for instructional use or demonstrations, not for private dining experience.
Answer: A cooking class.
The kitchen features multiple cooking stations, including seven stovetops and individual cutting boards, all arranged in structured layout. This setup is characteristic of a cooking class environment, where several people cook simultaneously, rather than setting intended for a private dinner.
Justification:
Figure 13: Example 2
38

---

Published as a conference paper at ICLR 2026
Example 3
0.4
0.
02
x rpp.ouuerrpppllsssaaa/.,,, 3, 32), np.ones(32))
= x.coOy().T
np.sin(x np.cosly 22
0.3
0.4
AgentX_211 212. jpg
4202
AgentX 2...2.jp8
Query:
What is the highest point visible on the plotted surface, and where does occur the 3D space?
Involved Tools: RegionDescriber, MathOCR, Solver
Steps:
1. Describe the region corresponding to the peak of the surface plot. to determine approximate and values.
2. Extract the equation for the surface to calculate the exact value of z.
3. Compute the z-value at the identified (x, y) coordinates.
Reasoning Trace for steps:
1. The RegionDDescriber tool is used to analyze the highest point the plotted surface. This helps estimate the corresponding x and coordinates, which are visually identified approximately (2,3).
2. The MathOCR tool extracts the surface equation 2 = sin(x?) + cos( from the image. This needed to compute the exact value of z at the identified location.
3. The Solver tool evaluates the equation at x-2 and y=3, y=3, z-value of approximately 0.24. This confirms the height of the highest visible point in the 3D space.
yiclding computed
Answer: The highest point the plotted surface occurs at coordinates (x, y, (2.0, 0.0, 0.24).
J Justification:
The highest point visually located at (2,3). Using the extracted equation, the valalue this point was computed to be approxinately 0.24, con- firming its position in the 3D space.
Figure 14: Example 3
39

---

Published as a conference paper at ICLR 2026
Example 4
NASHVILLE BIG USAT
TENNESSEE
Agentx_0.1p0
Query: What kind of room is the poster located in, and what does it say?
Involved Tools: SceneDescriber, Locate@bjactByText, DCR
Steps:
1. Describe the scene to determine what lype of room is shown.
2. Locate the poster within the scene to enable reading its content.
3. Extract text from the poster the scene.
Reasoning Trace for steps:
1. The SceneDescriber tool identifies typical kitchen elements like cabinets, sink, a dish rack, microwave, and stove, which indicates that the room is kitchen.
2. The LocateObject By Text tool helps determine where the poster is in the image, allowing me see what is written on it.
3. The OCR tool allows extraction of the visible text from the poster, making it possible for me to read what says.
Answer: The poster is in a kitchen and reads: "NASHVILLE MUSIC CITY USA! TENNESSEE""
Key kitchen elements such as a microwave, oven, stove, sink, and cabinets identified, confirming the room is kitchen. The poster's position in the image was located, and its text was extracted, allowing the content to be read and verified.
Justification:
Figure 15: Example 4
40

---

Published as a conference paper at ICLR 2026
Example 5
(x-
EgentX_133.11
What is the solution the equation, and is the solution real, complex, or integer?
Query:
Involved Tools: MathOCR, Solver, Calculator
Steps:
1. Extract the mathematical expression from the image.
2. Solve the extracted equation to find the values of x
3. Analyze the solutions to determine whether they are real, complex, or integers.
Reasoning Trace for steps:
1. The MathOCR tool extracts the equation (x- +4* = from the image, enabling symbolic processing.
2. The Solver tool is used to solve the equation, yiclding the solution x -sqrt(2).
3. The Calculator confirms that the value sqri((2) real number, which is irrational but not complex or integer.
Answer: -2gr((2). Real
J Justification: The equation was extracted, solved, and evaluated. There is just one solution of this equation. The resulting value is real and irrational, not complex or integer.
Figure 16: Example 5
41

---

Published as a conference paper at ICLR 2026
Example 6
Mesntisia Line
FroR
Blar
Rabbiz
Mouse
Ceebrt
AgentX 555. jpg
Query:
How many species does the top predator prey on?
Involved Tools: OCR, SceneDescriber, ObjectCounter
Steps:
1. Extract all text from the image to identify species and connections.
2. Analyze the food web structure to find species with no outgoing arrows indicating top predators.
3. Count the number of incoming arrows to the top predator to determine how many species it preys on.
Reasoning Trace for steps:
1. Reading the species names helps establish which organisms are part of the food web and which ones connected by feeding relationships.
2. Analyzing the direction of arrows reveals the top predators species with no outgoing arrows- indicatin that nothing preys on them. this case, the Hawk and Mountain Lion meet that condition.
3. Counting the number of arrows pointing to each top predator shows how many species they prey ou: four for the Hawk and two for the Mountain Lion.
The top predators in the food web are the Hawk and the Mountain Lion, which prey on four and two species, respectively.
Answer:
The Hawk and Mountain Lion are top predators with outgoing arrows. Counting the arrows pointing to each confirms that the Hawk preys on four species and the Mountain Lion on two.
Justification:
Figure 17: Example 6
42

---

Published as a conference paper at ICLR 2026
Example 7
X_values 3,, 5]
y_ v_values= 3, 5, 7, 11]
400
bubble sizes = [30, 80, 150, 200, 300]
200
AgentX_ DeantX3399.0 1. jip
AgentX_3ae 339.2.jpP
Using the provided data, verify the x, y, and size values against the bubble chart. Identify if there are any discrepancies in the data representation, and plot a corrected version of the chart if necessary.
Query:
Involved Tools: OCR, SceneDescriber, Locate@bjectByTextt, CodePlotter
Steps:
1. Extract the lists of x-values, y-values, and bubble sizes from the data.
2. Analyze the layout and structure of the bubble chart to understand how the data is represented visually.
3. Compare the extracted values with the chart to identify any mismatches in position size.
4. Generate corrected version of the chart using the accurate data values.
Reasoning Trace for steps:
1. The OCR tool used extract the lists of x_ values, y_ values, and bubble_sizee from the data, which are needed to verify the accuracy of the plotted chart.
2. The SceneDeseriber helps interpret the layout of the bubble chart, including axis scales and the overall distribution of points, which provides context for comparison.
3. By visually inspecting the chart and referencing the extracted values, mismatch is observed between the actual positions or sizes of the bubbles and what. the data specifies.
4. Re-plotting ensures that the visual representation now matches the ex- tracted x, y, and bubble sizes to produce an accurate visual representation.
Answer:
The extracted x, y, and size values did not match the visual chart. Re-plotting the data corrected these discrepancies, ensuring the chart accurately reflects the provided values.
Justification:
Figure 18: Example 7
43

---

Published as a conference paper at ICLR 2026
Example 8
When you send message friend who's online and right after that, they go offline
AgentX_ SeentX_334.jjp 534.jpg
Query: What emotion is being depicted in the image? What is the meme about?
Involved Tools: SceneDescriber, OCR, RegionDescriber
Steps:
1. Describe the scene in the image to identify the depicted emotion.
2. Extract text from the image to understand the context of the meme.
3. Analyze combined information to conclude the meme's theme.
Reasoning Trace for steps:
1. The scene description helps identify that the cal appears sad, indicating the emotion of sadness or disappointment.
2. The text provides context for the image, explaining why the depicted emotion is sadness or disappointment.
3. Combining the cat's sad expression with the text content, the meme humorously conveys the disappointment of a common online interaction.
Answer: The emotion depicted is sadness. The meme is about the disappoint ment felt when friend goes offline right after receiving message.
Justification:
The steps provided clarity on the cat's expression (sadness) and the textual context (meme scenario), confirming the theme of disappointment.
Figure 19: Example 8
44

---

Published as a conference paper at ICLR 2026
Example 9
Custoniing char
import natplotlib, pyslet as
ordered
plotted counter-clockiiiee
continents ['Msie", 'Europe', Meriica', "South Averica', "Mustrolla',, 'Africa" 'Antarctice"]
10/2
POTO
[25, 18,25,18,5] explede (6.1, 8, 0.1, 0,8.1,0.1,8.3)
AgentX, 340.1.
AgentX_ SgentX_340.22 340.2.JpP
Query: Based on the provided code snippet and the pic chart, which continent represents the largest area from the given data, and which car brand reprosents the largest percentage in the pie chart? Calculate the difference between these two percentages.
Involved Tools: MathOCR, Calculator, OCR, SceneDescriber
Steps:
1. Identify the continent with the largest area from the code snippet.
2. Calculate the percentage that this area represents out of the total.
3. Analyze the pic chart to identify the largest slice in the pic chart.
4. Extract the percentage value and brand name associated with that slice.
5. Calculate the difference between the largest area percentage and the largest pie chart percentage.
Reasoning Trace for steps:
1. To determine which continent dominates in size, the area values from the code need to be examined so that we use the correct value for comparison.
2. Calculating the percentage of the largest area provides a normalized value, making it possible to directly compare it with percentages shown in the pie chart.
3. The SceneDescriber helps interpret the pie chart structure, which neces- sary to visually identify the largest share based on the relative size of each slice.
4. The OCR tool is used to confirm the brand and exact percentage of the largest slice, ensuring that the visual estimate is backed by precise data.
5. Finding the difference between the two percentages answers the core question by quantifying how the largest values from each source compare.
Answer: Asia, Mercedes, 1.1%
The continent with the largest area is Asia (25%), and the car brand with the largest percentage is Mercedes 1%). The difference between these two percentages is 1.1%.
Justification:
Figure 20: Example 9
45

---

Published as a conference paper at ICLR 2026
Example 10
AgentX 630 63j_6.jjp
Agent.X 630 30.jpg
6300 6300_54.jpg
Query: Is the person in the video an employee?
Involved Tools: SceneDescriber, RegionDescriber
Steps:
1. Identify the type of location shown in the video.
2. Determine whether the setting appears open to customers or closed.
3. Observe the placement of objects for operational context.
4. Analyze the person's behavior and interaction with the environment.
5. Consider whether the actions match expected employee routines.
6. Decide if the person's 's presence and behavior indicate they are an employec.
Reasoning Trace for steps:
1. Identifying the place restaurant or café sets clear expectations for how staff typically behave.
2. If the place appears closed, anyone inside should either be staff or unau- thorized.
3. The chairs placed upside down tables indicate the location is not open to customers.
4. The person's 's cautious movement and drawer searching suggest they are not performing routine tasks.
5. Staff usually act with purpose, doing things like cleaning or closing up, not wandering or poking around
6. The closed setting and unusual behavior make it unlikely that the person is an employee.
Answer: No
The restaurant appears closed, and the person's behav- ior cautiously searching through drawers -owe does not match typical staff duties. This suggests they are likely an employee.
Justification:
Figure 21: Example 10
46

---

Published as a conference paper at ICLR 2026
Example 11
Aumoerat for KAPSPEEA EC EETEWEE
NJ WERT June GRVIR GPareend
AgentX_ 358_5211 ________pp 5.jpg
AgentX 358 358_25.jppg
AgentX -58 45.jpg
Query: How many people entering the elevator?
Involved Tools: SceneDescriber, RegionDescriber, ObjectCounter
Steps:
1. Undersland the overall context of the sceni and confirm that the setting involves an elevator.
2. Analyze the body positions and movement direction of individuals across the frames to identify who is entering the elevator.
3. Count the number of people who move into the elevator space.
Reasoning Trace for steps:
1. Recognizing the environment how people should move relative to the door.
clevator area helps set expectations for
2. Analyzing the region near the elevator door across frames shows one woman stepping forward into the space, while others remain stationary are already inside.
3. Counting only the individuals who cross the threshold into the clevator shows that one person enters.
Answer:
Justification: woman is moving into the elevator while others remain in place already inside, confirming that only one person is entering.
Figure 22: Example 11
47

---

Published as a conference paper at ICLR 2026
J GENERATION PROMPTS
Query Generation
"You are an annotator tasked with generating realistic and verifiable user query for benchmarking a multimodal LMN-based assistant.
You provided with the following tools the assistant
tools "ImageDescription" "GommtGivenObeett,, "OCR", "DrauBox", "Calculator", "DetectGivenlbject",, Piagionnttrbbuutbeerripppa "KathOCR", "Solver", "Plot", "GoogleSearch", "TextToImage", "AddText", "ImaeStylization"
Your job is to generate one single complex and human-evaluable user query that satisfies all of the following:
If you are given multiple images, the query must require reasoning across multiple images (not just one)
you are given multiple frames, the query must involve comparing, tracking, or analyzing across multiple video frames.
The query must require least distinct reasoning steps be answered.
It must require at least 2 different tools from the list above. not mention tool names explicitly.
If the query involves online time-sensitive content, include fixed timeframe source.
The query must be realistic and grounded in common user intentions.
Use only English.
All queries must be suitable for evaluation by a human the answer should not vary arbitrarily across individuals.
Return your result as a single JSON object on one line. { "input": (file_repr), "query": "Knatural language query>" }
Figure 23: Prompt given to GPT4-o to for the initial query generation.
48

---

Published as a conference paper at ICLR 2026
Reasoning and Final Answer
"You are an agent performing multimodal reasoning using the tools listed below.
Your goal is to answer the given query step-by-step by selecting the right tools for each stage of reasoning.
For each step:
Clearly state what you're re trying to do.
Specify the tool you are using.
Explicitly include the input provided to the tool and the output
received from the tool.
Explain what you learned why this step was necessary.
Make sure your tool usage follows tool descriptions provided. Do not hallucinate tools or skip intermediate reasoning steps.
Available Tools How to Use Them: [toolmeta_section)
Query: "[query)"
Images: (imagel, image2, ...)
Return your output as JSON list steps, formatted like this:
"step": <step number?,
"task": "<Short description of what this step is trying to do>",
"tool": "<Tool name (s)>",
"input": "<Input provided to tool>"
"output": "<Dutput from the tool>",
"thought": "<Why this step was needed, or what you learned>" },
"final_ answer": {
"value": "<Final conclusion>", "Justification": "<How all steps lead the answer>"
}}
Figure 24: Prompt given to GPT4-0 for the initial reasoning and final answer generation.
49

---

Published as a conference paper at ICLR 2026
K EVALUATION PROMPTS
Grounding Score
You are an evaluation assistant measuring the groundedness each reasoning step performed by vision-language agent.
"You given Ground Truth (GT) block containing the original query, sequence reasoning steps, and the final answer along with justification. Each reasoning step includes the task being attempted, the tool used (with its input and output), and the agent's s thought process.
You also given the full reasoning trace produced by the agent, including each step's a task, selected tool its output, thought.
Your task is to assess whether each agent step is visually and contextually grounded that is, whether the task, the selected tool, the tool output, and the agent's thought process are supported by the GT.
Evaluation Guidelines:
If the number of steps in the GT and the model differ:
Any extra model step beyond the GT steps should receive a score of 0 (hallucinated or unjustified).
Any GT step that the model omits should also receive a score of (missing reasoning).
The Grounding Score for the query is computed as average of the per-step scores.
Assess the following per step:
Is the task relevant to the query and aligned with the GT?
Is the tool appropriate for this task?
Is the output consistent with what's 's in the GT?
the thought grounded and logically aligned?
Scoring Criteria:
1 All aspects grounded (task, tool, output, and thought)
0.5 Partially grounded (some align, some don't)
0 Ungrounded hallucinated
Output format:
Score: <0, 0.5, or 1>
Justification: <1-2 sentence explanation for the score>"
Figure 25: Prompt for computing the grounding score of reasoning steps produced by vision- langugge agents.
50

---

Published as a conference paper at ICLR 2026
Tool Precision
"You are an evaluation assistant measuring tool precision in an agent's reasoning step.
You given Ground Truth (GT) block containing the original query, sequence of reasoning steps, and the final answer along with justification. Each reasoning step includes the task being attempted, the tool used (with its input and output), and the agent's 's thought process.
You also given the full reasoning trace produced by the agent, including step's task, selected
and output, and thought.
Your task is to assess whether the tool selected by the agent in each reasoning step is the most appropriate tool for the task, by comparing it with the GT.
Evaluation Guidelines:
If the number of steps in the GT and the model differ:
Any extra model step beyond the GT steps should receive a score of 0 (hallucinated or unjustified).
Any GT step that the model omits should also receive a score of (missing reasoning).
The Tool Precision score for the full query computed the average the per-step scores.
Assess the following per step:
the tool selected by the agent the same as the one used GT?
Is the tool the most appropriate for the task being attempted?
Scoring Criteria:
1 - The selected tool matches the GT tool and is appropriate for the
task.
The tool does not match the or an inappropriate choice.
Output format:
Score: <0 1>
Justification: <1-2 sentence explanation the score>"
Figure Prompt for assessing whether the tools selected by the agent each step match the ground truth.
51

---

Published as a conference paper at ICLR 2026
Tool Accuracy
"You evaluation asaistant measuring tool accuracy in agent's reasoning step.
You are given the task the agent is trying to accomplish, the tool it along with its input and output, and the tool metadata containing description
tool's purpose and its expected input/output formats.
Your task is to assess whether the tool used by the agent was applied correctly in each step by checking:
Whether the output format is valid consistent with the tool 's specification (based on tool metadata).
Whether the output relevant and meaningful for completing the step's task.
Evaluation Guidelines:
The Tool Accuracy score for the full query is computed as average the per-step scores.
Assess the following step: the tool's output correctly formatted according to the tool metadata?
Is the output meaningful and appropriate for the stated task? Scoring Criteria:
1 - Output is valid, properly formatted, and clearly relevant to the task.
0.5 Output is partially valid or partially relevant.
0 Output is incorrectly formatted, irrelevant, unhelpful.
Output format:
Score: <0, 0.5, or 1>
Justification: <1-2 sentence explanation for the score>"
Figure 27: Prompt for verifying correct tool usage and outpul formatting based metadala.
52

---

Published as a conference paper at ICLR 2026
Faithfulness Accuracy
"You evaluation assistant measuring the Faithfulness Accuracy of agent's 's reasoning process.
You given Ground Truth (GT) block containing the original query, sequence of reasoning steps, and the final answer along with its justification. Each reasoning step includes tool used (with its input and output), and the agent's 's thought process.
task being attempted,
You also given the full reasoning trace produced the agent, including each step's a task, selected tool, output, and thought.
Your task is to assess how faithful the agent's reasoning trace to the GT, i.e., whether the steps follow logically sound plan that aligns with the structure, intent, and direction of the GT.
Evaluation Guidelines:
Focus on the structure and logical flow the agent's 's reasoning steps.
Determine whether the steps collectively form coherent strategy answer the query.
Faithfulness is about consistency with the GT's method, not necessarily correctness of individual steps.
Scoring Criteria:
The reasoning is faithful to the GT: it follows logically sound plan that mirrors the GT structure and direction.
0.5 The reasoning partially follows the GT's structure, but contains some deviations, redundancies, inconsistencies.
The reasoning is not faithful to GT lacks logical progression or alignment with the intended plan.
Output format:
Score: <0, 0.5, Justification: <1-2 sentence explanation score>"
Figure 28: Prompt used to evaluate the structural faithfulness of agent reasoning to the ground truth.
53

---

Published as a conference paper at ICLR 2026
Context Score
"You evaluation assistant measuring the Context Score of agent's reasoning step.
You are given Ground Truth (GT) block containing the original query, sequence of reasoning steps, and the final answer along with its justification. Each reasoning step includes the task being attempted, the tool used (with its input output), and the agent's thought process.
You are also given the full reasoning trace produced by the agent, including each step's task, selected tool and its output, and thought, well the agent's final answer justification.
Your task is to assess whether the agent's reasoning step is grounded in the input context, and whether that context was effectively used in the reasoning process. Inputs may include image, video, text, audio, or combination.
Evaluation Guidelines:
the number of steps in the GT and the model differ:
Any extra model step beyond the GT steps should receive of (hallucinated or unjustified).
Any GT step that the model omits should also receive of (missing reasoning).
The Context Score for the full query is computed as the average of the per-step scores.
Assess the following per step:
Does the agent correctly relevant parts of the input?
the input used appropriate for the tool selected the task being attempted?
Does the reasoning show effective and meaningful of the input?
Scoring Criteria:
The agent the input fully and appropriately.
0.5 The agent uses some relevant input but misses important details or misuses others.
The agent ignores or misinterprets input entirely.
Output format:
Score: <0, 0.5, or 1>
Justification: <1-2 sentence explanation for the score>"
Figure 29: Prompt for evaluating the contextual grounding of each reasoning step.
54

---

Published as a conference paper at ICLR 2026
Factual Accuracy
"You are an evaluation assistant measuring the factual accuracy of the agent's reasoning steps.
You are given Ground Truth (GT) block containing the original query, a sequence of reasoning steps, and the final answer along with its justification. Each reasoning step includes the task being attempted, the tool used (with its input and output), and the agent's thought process.
You are also given the full reasoning trace produced by the agent, including each step's s task, selected tool, tool input/output, and thought.
Your task is fabricated, factually incorrect information its reasoning when compared to the GT.
whether the agent introduces any hallucinated,
Evaluation Guidelines:
Compare the model's reasoning steps the GT identify factual hallucinations or incorrect claims.
Focus on whether the output or thought includes details not supported the GT.
not penalize for minor omissions unless they lead to a factual error.
Scoring Criteria:
1 No factual errors or hallucinations compared to the GT.
0.5 Minor inaccuracies vague mismatches.
0 Major factual errors or hallucinated content.
Output format:
Score: <0, 0.5, 1>
Justification: <1-2 sentence explanation
score>"
Figure 30: Prompt for evaluating factual correctness of the agent's reasoning steps.
55

---

Published as a conference paper at ICLR 2026
Semantic Accuracy
"You are an evaluation assistant measuring the Semantic Accuracy of an agent's 's reasoning process.
You given Ground Truth (GT) block containing the original query, sequence of reasoning steps, and the final answer along with justification. Each reasoning step includes the task being attempted, the tool used (with its input and output), and the agent's thought process.
You also given the full reasoning trace produced the agent, including step's task, selected
output, and thought, as well as the agent's final answer and justification.
Your task is to assess whether the agent's reasoning and final output semantically align with the Ground Truth, i.e., whether the agent has covered all the essential parts the query demonstrated the GT.
Evaluation Guidelines:
Compare the agent's reasoning trace and final ansver with the GT to check whether all key components of the query addressed.
Credit should be given for meaningful semantic coverage, not superficial similarity.
the model ignores misunderstands parts the reasoning or final answer, penalize accordingly.
Scoring Criteria:
agent addresses key components of the query, matching GT's semantic scope.
0.5 Some parts covered, but the response misses weakly handles other essential elements.
0 - The agent's reasoning or answer omits or misrepresents major parts of the query.
Output format:
Score: <0, 0.5, or 1>
Justification: <1-2 sentence explanation for the score>"
Figure 31: Prompt for evaluating semantic consistency of the agent's reasoning and answer.
56

---

Published as a conference paper at ICLR 2026
Goal Accuracy
"You evaluation assistant measuring the Goal Accuracy of vision-language agent's 's final answer.
You are given the original query, the agent's final output, the ground truth (GT) final answer, and the type of query either "objective" "subjective".
Your task is to assess how well the agent's final output matches the ground truth answer, based on the nature of the query.
Evaluation Guidelines:
The Goal Accuracy score is computed once per query (not per step).
Use exact match evaluation for objective queries.
Use semantic similarity evaluation subjective queries.
Scoring Criteria:
For objective queriea:
The final output matches the GT exactly or is clearly equivalent.
0 The output is incorrect, incomplete, or unrelated.
For subjective queries:
Score Cosine similarity between the agent's answer and the GT answer (range: to 1)
Output format: If objective:
Score: <0 1>
Justification: <0ptional 1-2 sentence explanation>
If subjective:
Score: <cosine similarity>
Justification: <0ptional 1-2 sentence explanation>"
Figure 32: Prompt for evaluating alignment of final answer with the ground truth.
57

---

Published as a conference paper at ICLR 2026
Toolset Accuracy
"You evaluation assistant measuring the toolset accuracy of agent's reasoning process.
You given Ground Truth (GT) block containing the original query, sequence of reasoning steps, and the final
along with its justification. Each reasoning step includes task being attempted, tool used (with its input and output), and the agent's thought process.
You also given the full reasoning trace produced the agent, including each step's 's task, selected tool and output, and thought.
Your task is to evaluate whether the agent used the correct tools overall by comparing the set of tools it used to the set used in the GT.
Evaluation Guidelines:
tools that tools that are, it should be penalized.
the agent
present in the GT misses
The score reflects how well the agent's 's toolset aligns with the GT toolset
the full reasoning trace.
Output format:
Score: <F1 score rounded to decimal places>
Justification: <1-2 sentence explanation for the score>"
Figure 33: Prompt for evaluating alignment of the agent's toolset with the ground truth.
58

---

Published as a conference paper at ICLR 2026
L BENCHMARK CURATION EXAMPLES
The raw LMM-generated queries were often suboptimal: they merged unrelated tasks (e.g., "identify objects, find government symbols, count frames, and explain political significance"), contained ill-posed demands such as "circle the damaged part" without an annotation interface, asked for information invisible in the image (e.g., a jet's full landing-geat specification from à single profile shot). As illustrated in Fig. 34, therefore, every prompt underwent mandatory human-in-therr in-the-looop rewrite. Curators distilled each draft into single, objectively answerable question that aligns with the available tools, for example, "How many people are carrying patterned umbrellas, and what pattern on the umbrella. closest to the camera?" This step removes speculation, narrows the scope to verifiable visual evidence, and converts noisy LLM proposals into reliable, evaluation-gradd tasks.
Query
posser and Then, on visible estimate liaaly cay this photo was takenn Finally, craw box kitchen apollance
that pwrind.
Human Befined Query
What kind of room is the poster locatad in, and what closs it way?
AgenMXOO1p)
Query
by indivicuals.
symbo' governmentt organizations steps, signtficance image many
Query
manyy people ur carrying palterned olles, identify the pallerr closss career?
Apenot
Generated
Idantiff the sutivity being performed the imaghe count the number of visisle tootill caliee, and determine the reflections and Considering imerninlevveing dental that fits this
Human Refined
multiple
Generated Query
What the ande between the bont pols holcing the stop sian and the ground? Provide a brief deser lpsion the surroundi-g enwironment, and count the number cars visiate in image.
Human Refined Query
any of
meinterance? Pleass indicate cirdinng
Ageiti Pijag
Generated Query
Idantify ourrent dition image due fog, calculate whether driving lights necossary accorcing typical traffic ragulations.
visible.
Query
Does the endronmenn appear safe for riding matorcycle at this moment?
Apeio 83
Generated
imagl, trigin type airowel cornespord typical goar
Query
Identiff the country origin bype
Figure 34: Illustaatiee corrections: original LMM queries versus the refined, task-focused prompts used in benchmark.
59

---

Published as a conference paper at ICLR 2026
Generated and Answer
Human Refined
Tool Spenedesso lberr
heat: idensity species indoocdaati
How many individaals of this spacias woull be required to match the weight 150,000 Blue whate?
Thought uncerasaade rg the apecisa in the imsgee necessary to determine the welght of individual pangalr, which needed notve pribo
avergee weight penguut.
Weodearch oh
Thought the buse whole
alowe calculation how necded match
Stept 3
Taok: Calralate numoer pengui nwedee to maich we gig ot 160 nno
Thouught
aright
average seight pengut given
pangure
Final Arswer
valuee 5000 penguima
juctificarlon: The roene describee penguins and the suerge weight of renginn in caloutating 150,000 kg tenight of base senaioi divited by resuts 3000 per quina neadee to maten the whele's weith.
AgentX 150
Human Refined Reasoning and Answer
image.
Toot SceneCleso
Thought The SeecaDrec toot identihes the animaal humptack whale, which eusmial ceterninig weight should ueed for compariacn bluee wi als.
Step: 2
auvrnge weight of humptooct
Thoughtt
evorage seight
bepprso mommnayy
necemary swight meithinng
Tusk: Galculate humpbace whals noeded weight of 000 tuue whale
Tool Caltulstor
Thought The Cavosator divides ta the bus whale's welats t:50 000 kai by the tumbback chate's average welaht to dete mine how mary individubl are neodedd cquas mass
Arswerr
using
kg.
humpbact
while
Figure 35: Before-after refinement of an automatically generated reasoning trace. Left pane: the raw LMM output mis-identifies the species (penguin + humpback whale), fabricates an ObjectCounter call, and propagates the error to an absurd "5000 penguins" answer. Right pone: after human verification, each tool is appropriate to its sub-task, the correct constant answer is retrieved, and the calculator produces the logically consistent reasoning. The revision phase thus excises hallucinated steps, fixes fact look-ups, and restores end-to-end coherence.
L.1 REASONING TRACE GENERATION
fferr a query was finalised, we asked an LMM to auto-erpand into complete reasoning trace: thoughts, tool calls, and numerical answer. These traces were often self-contradictory: a model might identify penguins in Step 1, retrieve the average weight of a humpback whale in Step 2, hallucinate ObjectCounter call, and then feed the spurious value into the calculator. Accordingly, every Trace underwent a mandatory Phase-2 human pass in which annotators:
1. verified that each tool matched the declared sub-task,
2. replaced incorrect factual look-ups, and
3. recomputed downstream results whenever earlier constants changed.
In the whale- whale-weight vignette, the editor corrected the species mismatch, supplied defensible 30,000 kg 30,000 kg estimate for humpback whale, and updated the calculator output from "5000 penguins" to the logically sound "5 humpback whales". This clcan-up ensures that every released trajectory is logically coherent, numerically sound, and fully erecutable. Figure 35 juxtaposes the LMM-generated trace (left), replete with species confusion and cascading arithmetic errors: with the human-refinee version (right), where each tool invocation, intermediate value, and the final answer are perfectly aligned. The comparison underlines the necessity of Phase-2 curation for converting noisy, auto-generated chains of thought into dependable, evaluation-gras grade ground truth.
60

---

Published as a conference paper at ICLR 2026
AGENTX
Annotation Tool
Image
Annotations
Figure 36: Agent-X Annotation Tool Example
L.2 HUMAN VERIFICATION TOOL
To ensure every sample in Agent-X meets our quality bar, we developed a small web-based "Agent-X Reasoning Annotation Tool." When an annolator drags a folder onto the page the interface instantly loads the raw media on the centre canvas, the agent's 's step-by-step rationale in a panel to the right, and a navigation list of all remaining tasks on the left. Each reasoning field, including the task description, chosen tool, tool 1/O, and final answer, can be edited inline; a single Mark as verified tick confirms the trace now correct. The Save button writes changes back to the original Excel file, and global Export action adds "verified" column plus colour-codes every modified coll so downstream scripts can compute inter-annotator agreement. In practice reviewers process roughly fifly samples per hour, and Figure 36 illustrates two typical sessions: identifying an Adobe Photoshop logo. The tool's tight coupling of evidence, metadata, and editing shorteuts turns what would be cumbersome spreadsheet task into fluid, human- incann-i-teee wn-the-loopp verification workflow.
61


Reviews:
[{"review_id": "orNWdru2nR", "reviewer": "Reviewer_bQ33", "summary": "The paper proposes Agent-X, a benchmark for evaluating deep multimodal reasoning in vision-centric agentic tasks. It assesses how well multimodal models perform multi-step reasoning using images, videos, and tools across six environments — visual reasoning, web browsing, surveillance, driving, sports, and math. The paper includes 828 multimodal tasks, 14 executable tools and evaluated 10 LMMs. Models are evaluated in three modes i.e. Step-by-step, Deep Reasoning Outcome. Also, multiple metrics are reported for each of these modes. The findings suggest that even the best models achieve <50% full-chain success and models struggle with tool usage and reasoning consistency.", "questions": "1) All the metrics are derived from LLM-as-judge i.e Qwen14B and GPT-4o. Also results for human evaluations are presented. But this approach is not scalable, as the number of tasks and models increase. Do the authors have a method to scale these evaluations?\n2) How consistent and unbiased is the human refinement process, and how do the authors ensure annotation quality or diversity across annotators?\n3) Were multiple runs conducted for each model to assess reproducibility and statistical significance of reported metrics?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) Covers six distinct domains and integrates both image and video data\n2) Presents detailed analysis of failure modes of SOTA models\n3) Evaluates not just end answers but intermediate reasoning, tool grounding, and logical coherence\n4) Multiple metrics are reported in three distinct evaluation modes", "weaknesses": "1) Prompts need to be designed for LLM-as-judge. Is there an automated way to do this? Or do the authors need to manually create them every time a new metric is introduced?\n2) Human involvement at multiple stages makes it challenging to scale such a data generation approach. Also this is prone to errors and inconsistency? Any discussion around this would provide clarity.\n3) The dataset is built on top of publicly available datasets, it is hence unclear if this dataset is truely novel apart from some processing done on these datasets.\n4) No mention of variance, significance testing, or confidence intervals in Table 4 results. The results may vary due to use of LLMs in multiple places from task generation, task solving to evaluation.\n5) From the third paragraph in section 3.3, it seems like humans evaluate the correctness of tool calls and its coherence with final answer. Why are humans involved here, can't the tools calls be executed and answer returned to LLM and hence the model proceeds to complete the task.\n6) In the \"Step-by-Step\" process evaluation \"Tool Precision\" seems to be an unnecessary metric as same task can be completed via very different sequences of tool calls. Comparing a tool call sequence at test time to a \"ground truth\" sequence might not be an ideal way if evaluation for several task like web surfing where the environment can be very dynamic and hence change across runs.\n7) Metrics related to hallucination in \"Deep Reasoning Mode\", could be wrongly estimated if the LLM judge itself is prone to hallucination.\n8) Point 6 and 7 render some of the metrics unreliable and their variation needs to be studied across runs.\n9) Average steps per task is 3.4 (from Figure 3a ), which shows that the tasks are rather simple. How does this pipeline scale to tasks with more difficulty and longer trajectories. Also, evaluation of such trajectories seems to be a challenge with the proposed metrics and would incur costs as LLMs are used as judges.\n10) There’s no clear baseline showing how much performance improvement stems from reasoning vs pure perception. A sample experiment would be to take web tasks and evaluation LLMs using the access tree or DOMs instead of the image\n11) Human validation time (~50 hours per annotator) is low given 828 tasks, raising concerns about depth of verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HV1YGWcGQA", "reviewer": "Reviewer_YEeg", "summary": "The paper introduces Agent-X, a large-scale benchmark for evaluating deep multimodal reasoning in vision-centric agentic tasks. The benchmark consists of 828 tasks across six environments (general visual reasoning, web browsing, security/surveillance, autonomous driving, sports, and math reasoning). It features multi-step reasoning traces, tool-use sequences, and fine-grained metrics evaluating reasoning quality, coherence, and tool effectiveness.\nThe authors evaluate 12 state-of-the-art large multimodal models (LMMs) — including GPT-4o, Gemini 2.5, and Qwen2.5-VL — showing that even leading models perform below 50% on complex reasoning chains. The paper claims to be the first to combine vision-first multimodal reasoning with stepwise tool-augmented evaluation in real-world settings.", "questions": "1. Could you provide quantitative measures of inter-judge agreement (e.g., Cohen’s κ or correlation scores) to establish evaluation consistency among GPT-4o, Qwen-14B, and human annotators?\n2. How do you mitigate the potential circularity bias that arises when GPT-4o, a system under evaluation, is also used as a grader?\n3. Have these metrics been validated through human judgment or ablation studies to confirm that they align with actual reasoning quality?\n4. How do you ensure that the benchmark does not inherit linguistic or reasoning biases from the LMMs used in the task generation process?\n5. Did you measure inter-annotator agreement during human refinement, and how were disagreements resolved?\n6. Why are there no ablations comparing against random or shallow reasoning traces to validate that Agent-X genuinely measures deep reasoning complexity?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses a timely and relevant problem by filling a clear gap in evaluating deep reasoning and tool use for multimodal agents, an increasingly important topic as LMMs evolve into embodied and interactive systems.\n2. It presents a comprehensive benchmark design that spans six diverse domains and modalities, including images, multi-image comparisons, videos, and text.\n3. The tasks are realistic rather than synthetic, enhancing ecological validity compared to prior benchmarks like GAIA or GTA.\n4. The paper introduces fine-grained, multi-level metrics—covering step-by-step, deep reasoning, and outcome modes—that enable nuanced evaluation beyond final-answer accuracy.\n5. These metrics are particularly useful for analyzing reasoning coherence and the consistency of tool usage.\n6. The experimental analysis is thorough, evaluating both open and closed models, incorporating multiple judges (GPT-4o, Qwen, and humans), and providing detailed error analyses.\n7. It offers valuable insights into common failure modes such as formatting errors, shallow reasoning, and hallucinated tool calls.\n8. Finally, the benchmark is reproducible and openly available, with detailed documentation of the data pipeline, annotation processes, and tool specifications.", "weaknesses": "1. The paper primarily integrates existing components i.e. LMM reasoning, multimodal datasets, and tool evaluation, without introducing a fundamentally new evaluation paradigm, making it appear incremental compared to GAIA, GTA, and MLGym.\n2. The evaluation setup relies heavily on GPT-4o and Qwen-based automatic grading, raising bias and circularity concerns since the same model families are both evaluated and used as judges, with no quantitative inter-rater agreement reported.\n3. Despite its claim of real-world grounding, all tasks remain within predefined toolsets, offering limited evidence of generalization to unseen tools or domains, and no transfer or few-shot results are provided.\n4. The semi-automated dataset construction pipeline may inherit biases and linguistic artifacts from the generating LMMs, and the human refinement process lacks quantitative validation or agreement metrics.\n5. The study omits ablation experiments or baseline comparisons using shallow or random reasoning traces, leaving it unclear whether Agent-X truly measures deep reasoning rather than surface pattern matching; key metrics are also GPT-based without formal definitions or validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nuDi11ZYwQ", "reviewer": "Reviewer_5f8j", "summary": "This paper presents Agent-X, a benchmark for evaluating vision-centric AI agents' multimodal reasoning abilities. The benchmark contains 828 tasks spanning six environments (visual reasoning, web browsing, security, autonomous driving, sports, and mathematics) using real-world images, videos, and text inputs. Unlike existing benchmarks that use synthetic queries, Agent-X requires multi-step reasoning with tool usage in authentic scenarios. The authors develop a step-level evaluation framework that measures reasoning correctness, logical coherence, and tool effectiveness. Testing on 10 large multimodal models shows that even top-performing systems achieve less than 50% success on complete reasoning chains. The results indicate significant limitations in current models' tool usage and multi-step reasoning capabilities.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The authors have designed a comprehensive tool subset for the benchmark that covers most tools required for visual tasks. The evaluation criteria are also well-rounded, encompassing multiple dimensions of assessment.\n2. The benchmark demonstrates a significant improvement in sample size compared to previous benchmarks, which represents a notable contribution of this work.\n3. The benchmark proves to be quite challenging for most state-of-the-art open-source and closed-source models. From the results, no model achieves 50% on the Goal Accuracy metric, demonstrating the benchmark's effectiveness in evaluating model capabilities.", "weaknesses": "1. The template format used in the paper appears to differ from the official template provided. I am uncertain whether this may violate ICLR's formatting requirements. I recommend that the authors carefully review and address any formatting issues.\n2. The font in Figure 3(b) is quite blurry.\n3. I am not familiar with dataset construction in the agent domain. Could you please explain why JSON-formatted dialogue output is adopted? What is the rationale behind this design choice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tkDMxd4md", "reviewer": "Reviewer_KPXX", "summary": "The paper introduces Agent-X, a new, large-scale benchmark designed to evaluate the deep reasoning and tool-use capabilities of vision-centric AI agents. The paper argues that existing benchmarks are insufficient because they often rely on synthetic, single-turn queries, have limited visual modalities (mostly static images), and fail to assess the step-by-step quality of an agent's reasoning process.\n\nTo solve this, the paper makes two key contributions:\n* Agent-X provides 828 complex, multi-step tasks across six diverse, real-world environments: general visual reasoning, web browsing, security/surveillance, autonomous driving, sports, and math reasoning. These tasks involve rich multimodal inputs, including images, multi-image comparisons, videos, and instructional text.\n* A fine-grained, step-level evaluation framework. Instead of only checking the final answer, this framework analyzes the entire reasoning chain for:\n  * Step-by-Step correctness (e.g., tool selection and grounding).\n  * Deep Reasoning quality (e.g., logical coherence, factual precision).\n  * Outcome success (e.g., final answer accuracy).\n\nThe authors test several leading large multimodal models (LMMs), including the GPT, Gemini, and Qwen families. Their key observations are as follows:\n\n* No model tested achieves a Goal Accuracy rate above 50%. The best-performing model, OpenAI-4o-mini, only reached 45%, with most open-source models scoring below 30%. This indicates that complex, real-world, tool-augmented tasks remain extremely challenging.\n\n* Models that scored higher on the intermediate \"Deep Reasoning\" metrics (like faithfulness to the task, factual precision, and semantic accuracy) were more likely to produce a correct final answer. This supports the paper's hypothesis that evaluating the reasoning chain is critical.\n\n* Tool Use is a Core Bottleneck: Models struggle significantly with tool invocation and argument prediction. The error analysis shows that even when a model's thought process is logical, it often fails by selecting the wrong tool, hallucinating a tool that doesn't exist, or providing incorrectly formatted arguments.\n\n* Distinct Error Profiles for Top Models:\n  * GPT-4o: Showed strong formatting skills but was overly hesitant, often failing to act or misinterpreting visual content.\n  * Gemini-1.5-Pro: Was aggressive (rarely refused a task) but format-fragile, with a very high rate of JSON formatting errors (44.5%) and visual misinterpretations (34.3%).", "questions": "* Have the authors considered maintaining a leaderboard for this benchmark?\n* Are the authors motivated to extend this dataset to address the biases, shortcomings and to harder problems as the frontier models get better at solving the existing problems?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The Agent-X benchmark is comprehensive and well-thought-out:\n  * The inclusion of videos, multi-image comparisons, and text alongside single images is a significant step up from most vision benchmarks.\n  * Using six practical domains (like autonomous driving and security) makes the tasks more realistic and relevant than purely synthetic tests.\n  * Queries are designed to be \"tool-agnostic\" (i.e., they don't tell the agent which tool to use), forcing the agent to perform genuine reasoning about tool selection.\n\n* The three-mode evaluation (Step-by-Step, Deep Reasoning, Outcome) provides a powerful diagnostic tool. It allows researchers to move beyond a simple \"pass/fail\" score and understand why and where an agent is failing.\n\n* The paper details a semi-automated pipeline with human-in-the-loop refinement. The examples in the appendix clearly show how this process turns low-quality, ambiguous, or multi-part LMM-generated queries into focused, high-quality, verifiable benchmark tasks.", "weaknesses": "* While a clear improvement over single-step tasks, an average of 3-4 steps may not be sufficient to discover breakages in long-horizon planning, memory, and context management.\n* Reliance on model based critique can be brittle (e.g., known to be reward hacked). Perhaps using inference time techniques (like majority scoring) can be used to alleviate such risks.\n* The authors correctly identify the monolingual and potential dataset biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)