InfoMosaic-Bench: Evaluating Multi-Source Information Seeking in Tool-Augmented Agents
Authors: Yaxin Du
Yuanshuo Zhang
Xiyuan Yang
Yifan Zhou
Cheng Wang
Gongyi Zou
Xianghe Pang
WenHao Wang
Menglan Chen
Shuo Tang
Siheng Chen
Venue: ICLR 2026 Poster (2026)
Keywords: LLM Agents
Tool-Augmented Agents
Model Context Protocol
Primary area: datasets and benchmarks

Abstract: Information seeking is a fundamental requirement for humans. However, existing LLM agents rely heavily on open-web search, which exposes two fundamental weaknesses: online content is noisy and unreliable, and many real-world tasks require precise, domain-specific knowledge unavailable from the web. The emergence of the Model Context Protocol (MCP) now allows agents to interface with thousands of specialized tools, seemingly resolving this limitation. Yet it remains unclear whether agents can effectively leverage such tools—and more importantly, whether they can integrate them with general-purpose search to solve complex tasks.
Therefore, we introduce InfoMosaic-Bench, the first benchmark dedicated to multi-source information seeking in tool-augmented agents. Covering six representative domains (medicine, finance, maps, video, web, and multi-domain integration), InfoMosaic-Bench requires agents to combine general-purpose search with domain-specific tools. Tasks are synthesized with InfoMosaic-Flow, a scalable pipeline that grounds task conditions in verified tool outputs, enforces cross-source dependencies, and filters out shortcut cases solvable by trivial lookup. This design guarantees both reliability and non-triviality.
Experiments with 14 state-of-the-art LLM agents reveal three findings: (i) web information alone is insufficient, with GPT-5 achieving only 38.2\% accuracy and 67.5\% pass rate; (ii) domain tools provide selective but inconsistent benefits, improving some domains while degrading others; and (iii) 22.4\% of failures arise from incorrect tool usage or selection, highlighting that current LLMs still struggle with even basic tool handling.

Published as a conference paper ICLR 2026
INFOMOSAIC-BENCH: EVALUATING MULTI-SOURCE IN- FORMATION SEEKING IN TOOL- AGENTS
Yaxin Du
Yuanshuo Zhang'
Xiyuan Yang'
Yifan Zhou?
Cheng Wang' Shuo Tang*
Gongyi Zou'
Xianghe Pang
Wenhao Wang
Menglan Chen'
Siheng Chen 1+
IShanghai Jiao Tong University
The Chinese University of Hong Kong 7Zejiang University
UUniversity of Oxford
ABSTRACT
Information seeking is a fundamental requirement for humans. However, existing LLM agents rely heavily open-web search, which exposes two fundamental weaknesses: online content is noisy and unreliable, and many real-world tasks require precise, domain- specific knowledge unavailable from the web. The emergence of the Model Context Pro- tocol (MCP) now allows agents to interface with thousands of specialized tools, seemingly resolving this limitation. Yet it remains unclear whether agents can effectively leverage such tools ond more importantly, whether they can integrate them with general-purye search to solve complex tasks. Therefore, we introduce CeoMMosiic-Bnnhhe the first bench- mark dedicated to multi-source information seeking in tool-augmented agents. Covering six representative domains (medicine, finance, maps, video, web, and multi-domain in- requires agents to combine general-purpose search with
tegration), InfoMosaic-Bente domain-specific tools. Tasks are synthesized with AifoMosaic-Floww scalable pipeline that grounds task conditions in verified tool outputs, enforces cross-source dependencies, and filters out shortcut cases solvable by trivial lookup. This design guarantees both re- liability and non-triviality. Experiments with 14 state-of-the-art LLM agents reveal three findings: (i) web information alone is insufficient, with GPT-5 achieving only 38.2% ac- curacy and 67.5% pass rate; (ii) domain tools provide selective but inconsistent benefits, improving some domains while degrading others; and (iii) 22.4% of failures arise from in- correct tool usage or selection, highlighting that current LLMS still struggle with even ba- sic tool handling. Code is available here: https: www.ggthhub.... www.://w..ooooooooottttutot
Liffo-Mosaico
INTRODUCTION
Access to high-quality information is the fundamental driver of enhanced cognition, optimized decision- making, innovation, and societal progress. Each major advance in intelligent systems has been closely tied to progress in how they acquire and organize information: the advent of PageRank search engines (Page et al. et al. 2020, OpenAl 2023 Kaplan et al.) 2020) shifted information access from explicit retrieval to lever- aging vast pre-trained knowledge; and most recently,
1999) made the web navigable scale; the breakthrough of large language models (LLMs) (Brown
LLM agents (Nakano al.) 2021 such as various deep research product (OpenAI] [0255] |Perllxxity AI 2025) G have transformed information seeking into an iterative process of querying, browsing, and synthesizing evidence.
"Corresponding Author: Seeenn@@sjuu.edu.cm

---

Published as a conference paper ICLR 2026
InfoMosaic:
Kmnkk2
GP75
GL34-5
Groh4
Multi domuin
Evaluating Multi-Source Information Seeking in Gool-AuummentedAgem
Query:
Dataset:
Tiaunce
Video
Identify the multicenter trial testing a topical commensal Staphyoccccuus from healthy skin, located with study site north Interstate 90 in Chicago....
621 real-world samples
across domains
5 Single & Multi Domain
Modical/iic
Single Source
X Tools:
Web
77 MCP Tools across 5 domains
Multi Source
7 MCP Servers
Models:
6CT06504160
close-sourced model open-sourced model
Figure 1: Overview of InfoMosaic-Bema The benchmark evaluates multi-source information seeking in tool-augmented agents. (Left) Example query illustrating that single-source web search often fails, while multi-source tool use is required. (Center) Dataset statistics, including 621 samples across six domains, 77 MCP tools, and 14 models (7 closed- and 7 open-sourced). (Right) Radar plot showing domain-wise accuracy across models and the pie chart illustrating sample distribution across domains.
Already in wide use, these agents are now becoming indispensable, powering high-frequency workflows science (Chai al. 2025), business, and everyday decision decision-making -akinng (Shen 2024).
Despite their growing adoption, today's agents remain fundamentally limited by their heavy reliance on open-web search. Online content is noisy, inconsistently formatted, and often unreliable (Wenzek et al. 2020 Vosoughi et al.) 2018), making it insufficient for high-stakes applications. More importantly, many weal-world tasks require precise, verifiable, and domain-specife knowledge that web search simply cannot provide. For example, a financial analyst risks misleading conclusions without structured access to corporate filings and market data (Loughran & McDonald 2011); medical assistant cannot ensure patient safety without curated drug-side effect databases (U.S. Food and Drug Administration 2022); ; and even route planning requires geospatial applications and transport schedules that cannot be recovered from fragmented web pages. These scenarios reveal deeper challenge: general-purper purpose web search is not enough-reliate agents must integrate both general web information and specialized, domain-specific sources.
In parallel, the ecosystem of intommatioonseekiing tools is expanding rapidly. With the advent of the Model Context Protocol (MCP) tools (Hou et al., (025). LLM agents can access thousands heterogeneous data sources, ranging from biomedical databases (Flotho et al., 2025) to financial feeds (Zeng 2025) and map- ping services. Such advancements substantially enrich information-seeking potential. While this shift appears to overcome the limitations of relying solely on pprppose web search, it also raises two critical open questions: (1) How effectively can LLM agents leverage domain-speen -eeeaaii-ppeciiccccc tools to access information within each individual field? (2) More importantly, can they seamlessly integrate general general-purpose purpose search with multiple specialized tools to tackle complex, multi- source iioormation-seeingg tasks?
interaction and broaden their
2

---

Published as a conference paper ICLR 2026
To answer these questions, we propose InfoMosaic,Bench, the first benchmark dedicated to evaluating the ability of LLM agents to perform multi- source information search using external tools. InfoMosaic- Bench comprises 621 synthesized tasks and 77 tools across six domains-meedcallee medicaa/eo/daccicoooooo.. finance, maps, video, web and multi-source seeking. This benchmark directly targets the two open challenges identified above and enables evaluation of both domain-specinne tool usage and the harder setting of seeking of multi- source information. Unlike existing benchmarks, which either focus generic web search in single source with single tool (like BrowseComp (Wei et al. 2025) and WebWalkerQA (Wu et al., 2025)) or correctness of isolated tool calls (r-Bench (Yao et al. 2024), MCP-Bench (Wang et al. (025b)). InfoMosaic- BenncccceBechh uniquely evaluates agents' ability to solve multi-source informatiin-sseeknng tasks using contemporary and domain-specific MCP tools, with verified outputs ensuring reliability and non-triviality.
A key challenge in constructing such a benchmark is how to design tasks that inherently require multi- source search, rather than being solvable by single tool trivial web lookup. In practice, human curation has two limitations: no single author has broad cross-domain expertise, and crafting coherent multi-source tasks demands dozens of iterative tool calls, which rarely sustainable by hand. Therefore, we propose Info.Mosaic-Flow, an agentic data synthesis pipeline for multi-source information seeking task. The key idea to leverage an organizer-workers architecture, where a single organizer acts the commander, co- ordinating multiple domain-specific workers to enable scalable, cross-tool data synthesis. The organizer handles high-level planning, while each worker, tied a particular domain, executes assigned tasks with its tools and returns precise results. This design enables integrative use of domain tools while maintaining ro- bust reasoning, producing coherent and cross-tool grounded outputs. At last, we enforce multi-stage quality control combining automatic and carefully guided manual checks guarantee reliability and difficulty.
After conducting extensive experiments, the results reveal three key findings: (1) Web information alone is insufficient for precise domain reasoning: even GPT-5 attains only 38.2% accuracy, showing that open-web search cannot meet the information needs of domain-specifie tasks. (2) Domain tools offer selective but limited benefits: they improve performance in Map and Video but degrade in Medical, Finance, and Multi- domain, indicating that current agents are still far from being able to effectively exploit domain domain-specifie tools within each field. (3) Many failures come from incorrect domain-tool usage and selection; nearly 22.4% of failures come from wrong tool usage and tool selection, demonstrating that agents lack the competence to reliably in even basic tool handling.
Looking forward, the benchmark exposes fundamental gap: today's models excel at web search yet re- main unable reliably exploit domain tools or combine them effectively. Closing this gap not minor improvement, but prerequisite for deploying trustworthy agents in high-stakes domains such as medicine, finance, and scientific discovery.
Our contributions are as follows:
We identify the challenge that reliance on general-purposs web search is inadequate, and that no benchmark evaluates whether agents can leverage diverse domain-specific tools for reliable infor- mation seeking. We propose MrMMooai--Bnnel fill this gap.
We propose InfoMosaic- an automated two-stage synthesis pipeline that grounds tasks in domain-wise wise tool evidence and refines them with web-based verification.
Experiments show that relying web search alone is insufficient for precise reasoning, while domain-specific tools can unlock additional capabilities, but current agents fail to robustly use them, leading to selective and inconsistent gains.
3

---

Published as a conference paper ICLR 2026
2
RELATED WORK
2.1 TOOL-USING LLMS
Early work tool-augmented reasoning explored how to disentangle internal reasoning from external tions. ReAct Yao et al. 2023a) pioneered this idea by interleaving
with explicit tool calls (e.g., search, calculator), enabling models iteratively refine answers with external evidence. Building on this, Toolformer (Schick et (023) showed that L.LMs can self-supervise when and how to call APIs, while systems such as TooILLM (Qin al. 2023) and EasyTool (Yuan al. 2024) and MCP-FLow (Wang et al., 2025a) scaled the breadth of API coverage and improved robustness of invocation. As LLM capabiliities advanced, the focus shifted from invoking single APIs to handling long-horizon search and orchestration. Works such as Search-ol (Li et al. 2025b), WebThinker (Li et al., 2025c), and R1 R1-Searcher (Song et al., 2025) focus on persistent retrieval and orchestration web search, highlighting the strengths and limitations of single-channel search-augmenter reasoning. In contrast, the introduction of the Model Context Protocol (MCP) (Hou et al. 2025) expands tool use from web-only retrieval broad ecosystem of heterogeneous domain-specifie tools, raising the challenge of coordinating and integrating evidence across multiple sources. Our work targets precisely this gap, proposing a benchmark dedicated to evaluating multi-source information seeking in tool-augmented agents.
2.2 BENCHMARKS FOR TOOL-USING AGENTS
There are three parallel lines of work for benchmark tool-augmented LLMs: 1) API-centric benchmarks. ToolBench(Qin et al. 2023) and related datasets (Patil et al.) Yao et al. 2024 Chen et al. 2025) evaluate an agent's ability to discover, select, and call APIs correctly. These efforts provide valuable coverage of API functionality and invocation robustness, but the evaluation typically centers on single-tool correctness rather than multi-source synthesis. 2) Weblsearch-orienter benchmarks. Datasets such BrowseComp (Wei et al. 2025), Web WebWalkerQA (Wu et al. 2025), and MM-Bu MaM--rroseoopppp (Li et al., 2025a) evaluate agent's 's ability to engage with the open web, combining capabilities such as query reformulation, long-horizon reasoning, and information extraction from complex webpages. These benchmarks highlight the reasoning dimension of search-augmented agents and have advanced our understanding of how models operate in noisy and partially observable environments. However, the scope of tool use remains narrow: agents are restricted to web search and browsing, without evaluating whether they can coordinate multiple types of tools or integrate evidence beyond single retrieval channel. 3) MCP-style tool suites. More recently, benchmarks have emerged around the MCP ecosystem, including MCP-Universe (Luo et al. (025). MCP-Radar (Gao et al.] (025), MCP-Zero (Fei et al., 2025), and MCP-Bench (Wang et al., (025b). These benchmarks expose agents large-scale, heterogeneous tool environments and focus on aspects such as tool invocation correctness, exccution robustness under complex tool spaces, zero-shot tool discovery. However, they generally stop short of evaluating information seeking and long-horizon reasoning across tools. In short, none of them systematically evaluate whether LLM agents reliably seek, combine, and reason over heterogeneous evidence sources. InfoMosaic- Bench fills this gap, providing a benchmark where every task is grounded in tool evidence and demands genuine multi-source reasoning.
3 METHODOLOGY
To construct benchmark that reliably evaluates the ability of LLM agents to integrate evidence across multiple heterogeneous tools, we propose Info.Mssaic-Flow, scalable synthesis pipeline that generates tasks requiring non-trivial multi-source reasoning. This section introduces the overall design in Sec. 3.1 and describes our quality control procedure in Sec. 3.2
4

---

Published as conference paper ICLR 2026
Stage I: Information Seeking
Stage Il: Iterative Refinement
Synthesizer
Executor
Refiner
Verifier
Input: seed data
Input: find info
Inpuc: Initil Question
Input: find gro
information Bgelow
Scenario Proposing
Tool calling &
Condition Decomposing
Tool calling
xxx
Domain Info Gathering
Iterate
Sell search
plt_place__a
Verifier
search
gio
Regener
search video info
Macpotus
Summarize
Condition Fuzzing
vads Summarize
Output:
Fucord cus citice
Output:
Exccutor Response:
Srreecic
village WaR vc- 7:09 pin
Verifier Response:: fownd onumer by seanching beyaard
130.
Integrating
Concluding
Executors
Verifler Set
Outpuc:
Intea Question
Answer
Outputt:
Question
Figure 2: Overview of InfoMosaic-Flox, The synthesis pipeline is laid on an organizer- workers architec- ture, where a single organizer acts as the commander, coordinating multiple domain-spectie workers. Stage 1: Information Seeking composing interdependent constraints and grounding them with verified multi-tool outputs to form initial QA pairs; Stage 2: Iterative Refinement revising drafts, pruning shortcuts, and en- forcing multi-source reasoning.
3.1 DATASET CONSTRUCTION
Fig. 2 shows the overall generation with organizer-worker system. Specifically, an organizer is responsible for reasoning and formulating constraints/veriiiicttonn while worker is activated tool-calling event and follows instructions from the organizer to perform continuous tool calls and return consolidated evidence. This dual-agent designation has two advantages: (1) it isolates execution, preserving reasoning depth. The organizer handles decomposition or constraint reasoning; the worker handles fine-grained tool calls and evidence consolidation. This functional separation mitigates execution-induced noise in multi-step inference and reduces retrofitting of constraints to available tools. (2) It also expands exploration, improves multi- source coupling. The organizer remains tool-agnostic and selects only the target domain, while the executor freely chooses among all tools within that domain to satisfy the plan. This decoupling turns each subtask into combinatorial search over the domain toolset, increasing tool diversity and coverage (see App. A.7
Overall, the pipeline proceeds in two stages: (i) Information Seeking, the synthesizer composes interdepen- dent constraints and the executor grounds them with verified outputs from multiple tools to form the initial QA pair; and (ii) Iterative Refinement, where drafts are repeatedly challenged and revised to prune single- source shorteuts, leaving only tasks that genuinely require multi-source reasoning. Fig. 2 illustrates the multi-domain instantiation: the other domains generation process simply by eliminating executor toolsets with medical/biology, finance, maps, vidco, and web, respectively.
3.1.1 STAGE 1: INFORMATION SEEKING
This stage takes seed data as input and generates a coherent multi-condition problem grounded in domain- wise evidence. The core idea of this stage is to use seed data propose various scenarios that could diversify domain-wise tool callings to include different domain-specific information for problem synthesis. In addi- tion, instead of relying static templates noisy web content, actively query specialized tools and
5

---

Published as a conference paper ICLR 2026
compose their retrieved information into tasks. This design guarantees that every problem is grounded in verifiable evidence and requires reasoning across multiple sources.
In this stage, the organizer in the agentic system is the synthesizer, and the worker the executor with re- spective domain tools. The process of synthesizer works in the following steps: (1) Scenario Proposing, Starting from various seeds (e.g., Wikipedia or Baidu Baike, Qunar web, NCI IDs), the synthesizer proposes candidate scenarios that guide the construction of problems. This helps to provide diverse contexts that naturally invoke heterogeneous tools, increasing diversity beyond narrow or contrived tool-calling flows. (2) Domain Information Gathering. The synthesizer reasons step by step and emits high-level instruc- tions (subtasks) that trigger tool calls, i.c. "executor(subtask domain)". The executor, equipped with the domain toolset, selects and composes tools to retrieve verifiable domain-speciff facts and returns organized evidence. The synthesizer consumes this evidence, updates the plan, and issues the next instruction. (3) In- tegrating. Finally, the synthesizer organizes the validated tool results into a coherent multi-sourced problem which requires multiple tool calls and cross-condition reasoning.
Overall, this design has two benefits. (i) First, by hiding tool internals, the synthesizer focuses on maintaining coherence and naturalness in the problem statement, rather than overfitting to tool quirks. (ii) Second, the information gathering loop enlarges the exploration space and includes diverse tools. As a result, Stage 1 ensures the generated problems are both coherent and inherently multi-source.
3.1.2 STAGE 2: ITERATIVE REFINEMENT
While Stage ensures that every synthesized problem is executable, the resulting tasks may still be trivial in practice. In particular, some problems can be answered by satisfying only a single clue, or even by issuing a generie web query without invoking multiple tools. Such cases do not reflect the real challenges faced by agentic systems, where reliable reasoning requires integrating evidence from multiple heterogeneous sources.
To eliminate these trivial cases, we introduce an iterative refinement stage. In this stage, the worker is a Verifier with only web-search tool, attempting to answer the Refiner-assigned problem through web search. The refinement proceeds in three steps: (i) Condition Decomposing: the Refiner breaks down the synthe- sized problem into individual conditions and asks the Verifier solve them independently; (ii) Condition Fuzzing: if any condition proves too revealing (e.g., directly exposing the answer via single search), the Refiner rewrites, augments, or combines it with others to reduce shortcut solutions; (iii) Concluding: once condition can independently yield the answer and the Verifier fails to solve the task via search alone, the refined set of conditions is recomposed into the final question.
The refinement process is repeated until two criteria are simultancously met: ii the problem cannot be solved by web search alone, and (ii) no single condition is sufficient to determine the answer. This ensures that each admitted question is both challenging and robust, demanding genuine multi-source reasoning. This refinement stage guarantees difficulty, since trivial shortcuts are removed and the remaining tasks genuinely require reasoning over multiple sources, which ensures that InfoMosaic- Bench provides robust testbed for evaluating multi-source information seeking.
3.2 QUALITY CONTROL
To ensure the reliability of InfoMosaic-Bencee apply two automated checks, including Tool-Call Filtering, Answer-Evidence Consistency, and Coherence Filtering, to remove trivial, noisy, or ill-formed tasks. After automatic filtering, further conduct manual screening and revision by human annotators, who correct or discard problematic cases to improve factual alignment, coherence, and difficulty.
adopt series of quality control processes. We first
6

---

Published as a conference paper ICLR 2026
The automatic quality checks are as follows: (1)TooI-Call Filtering. We first enforce a minimum tool- call threshold in Stage 1, discarding samples below to eliminate under-constrained, low-seeking tasks and keep the benchmark focused on non-trivial multi-source reasoning. (2) Answer-Evvience Consistency. To guarantee traceability, we retain only items whose final answers are exactly derivable from collected tool outputs, ensuring every sample grounded verifiable information sources. (3) Coherence Filtering. We further remove tasks that exhibit incoherent or ill-formed conditions, such as contradictory constraints or unnatural phrasing. to guarantee that each problem maintains semantic coherence across its question, intermediate conditions, and final answer.
Human Selection and Refinement. After automatic filtering, we further review a sample for consistency, coherence, and difficulty I.I problematic items revised or discarded. A dedicated user study (Sec. 5.4) further confirms reliability for evaluating multi-source seeking.
4 DATASET
Statistics. Table summarizes the composition of wrfoMosaic-Benchh The dataset contains 621 problems across 5 domains (medicine/biology. finance, maps, video. web), plus an additional set of explicitly cross-domain tasks. Each prob- lem is paired with condition-level gold labels and traces of tool calls, enabling both final-answer evaluation and fine-grained diagnostic analysis. The benchmark is multilingual, contain- ing 621 instances in total. Among these, 229 queries are writ- ten in Chinese (36.9%) and 392 in English (6).1%) Crucially, the evaluation pipeline (agent framework, tools, and metrics) is identical for both languages, as described in this section. In to- al,, InfoMosaic- -eecch-Beecchhh incorporates 77 distinct tools spanning 7 servers, combined with condition-level supervision, ensuring that the benchmark provides a challenging and reliable testbed for evaluating multi-source information seeking. Detailed tool
Table 1: Domain distribution and key
statistics of InfoMosaic-Bench. Refer to App. A. for more detailed statistics.
Sample Number
MCP Tool
Domain
Number
Medical/Biology
83
15
Web
100
Video
100
Finance
100
29
Map
135
20
Multi-Domain
103
Total
621
77
information are described in App. A.9.2
Evaluation Metrics, We report both Accuracy and Pass Rate. Accuracy measuring strict end-to-end task success, reflecting whether the agent can complete information seeking and reasoning holistically. Pass Rate, in contrast, provides a more fine-grained view of agent performance based associated test cases (subquestions with gold answers or subgoal checks, like many coding tasks Du et al. (2025)).
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Models Evaluated. In our experiments, evaluate closed-source and open-source LLMs. Details of these LLMs can be found in Table
Agent Framework. For the agent framework, adapt the most popular framework ReAct (Yao al. 2023b equipped with OpenAl's tool-calling interface (OpenAI) 2025) and a Python Sandbox (Pang et al. 2025 from which LLMs receive the tool execution results. More details can be found in Sec. A.4.2
Evaluation. As introduced in Sec. we use Accuracy (Acc) and Pass Rate (PR) as our evaluation metrics. Instead of relying solely exact match, we leverage an LLM to judge whether the predicted answers align
7

---

Published as conference paper ICLR 2026
Table 2: Comparison of 14 LLM agents equipped with a web search tool on InfoMosaicc-Bnnch evaluated across six domains and the overall average. Metries include Accuracy (Acc) and Pass Rate. The best overall Accuracy and Pass Rate is highlighted bold.
Medical/ Biology
Multi- domain
Overall
Map
Video
Web
Finance
Acc
Passrate
Close Sourced Model
GPT-5
32.59
53.10
36,00
29.00
41.00
41.75
38.18
67.48
03
40.74
44.79
23.00
28.71
45.00
35.78
36.35
64.96
Grok-4
9.63
39.02
33.00
10.00
43.88
19.42
25.42
39.44
Claude-4.0-Sonnet
17.04
20.48
18.00
3.00
27.00
10.68
15.94
36.47
Qwen2.5-Max
7.41
7.23
5,00
0.00
9.00
1.94
5.15
15.72
Gemini-2.5-Flash
11.11
10.84
3.00
2.00
9.00
6.82
7.25
28.63
04-mini
24.44
25.30
24.00
39.00
24.27
24.15
61.67
Open- Weight Model
GLM-4.5
24.44
27.71
24.00
11.00
22.00
14.56
20.61
26.98
Kimi-K2
14.81
19.28
8.00
18.00
0.00
10.14
39.72
Qwen3-235-A222
5.19
19.28
6.00
0.00
23.00
3.88
9.02
31.40
Qwen3-32B
8.15
8.43
4,00
1.00
23.00
1.94
7.73
33.80
DeepSeek-V3 V3
9.63
7.23
1.00
0.00
16.00
2.91
6.28
25.40
Qwen3-Coder
9.63
4.82
6.00
0.00
9.00
1.71
5.44
19.25
Llama-4-Scout
0.74
4.82
0.00
0.00
22.00
2.91
4.83
21.03
with the references, which alleviates where semantically correct outputs cannot be captured by string matching. The detailed evaluation prompts are provided in the App. A.12.7
5.2
MAIN RESULTS
InfoMosaic-Bent demonstrates that web search alone insufficient for multi-source reasoning. Ta- ble reports results for 14 state-of-the-art LLM agents limited to web-search tool. We observe that: (1) Current agent system performs poorly on this task. Even the best closed-source model (GPT-5) attains only 38.2% accuracy and 67.5% pass rate. (2) Closed-source models exceed open-source ones by 15-20% on (3) Pass rates consistently surpass exact accuracy, reflecting that agents often satisfy some conditions but fail integrate all of them into correct final answer.
accuracy, yet both are constrained by web information.
InfoMossac-BBnnll reveals stark differences across domains, especially video, map, and multi- domain. The bottom-right radar in Tig[[[ummarriz domain- accuracy for six models on InfoMosaic- Bench. We find that (1) agents with only web search are highly uneven across domains: best scores reach 53% (Medical/Biology) and 45% (Finance) but drop 36% (Video) and 40.74% (Map). (2) Capabilities vary by domain for example, Grok-4 does relatively well on Video, whereas GPT-5 struggles there.
5.3
ANALYSIS
5.3.1
DOMAIN TOOL ANALYSIS
Comparison with only web search tool. To evaluate how LLMs perform with domain conduct experiments under two settings. For each single-domain evaluation, the agent is provided only with that domain's 's specialized tools (for multi-domain, all tools provided). We report answer accuracy for GLM-4.5 and GPT-5 under only web search tool and domain-tool settings in Table 3 We find that: (1) On average, domain tools yield only marginal gains, indicating that the bottleneck is not tool availability but
tools,
8

---

Published as a conference paper ICLR 2026
Table 3: Comparison between web-only and domain-tool agent accuracy across six domains and the overall average for GLM-4.5 and GPT-5. The colored numbers indicate the difference between domain- momainttool and web-only settings. Values in green denote accuracy improvements when domain tools are used, while values in red denote decreases.
Tool
Map
Medical/Bio
Video
Web
Finance
Multi-domain
Overall
web 24.44
27.71
24.00
11.00
22.00
14.56
20.61
GLM-4,5
domain
30.3745.93
34.9447.23
25.00+1.00
7.00-4.00
20.00-2.00
12.62-12 2..62-1.94
21.51+0.90
web
32.59
53.10
36.00
29.00
41.00
41.75
38.18
GPT-5
domain
40.00+7.41
43.37-9.73
46.00+10.00
32,00+3.00
30.00-9.00
39.81-1.94
38.61+0.43
tool use how agents plan, select, parameterize, and time their calls. (2) Both GPT-5 and GL.M-4.5 see clear gains in map and video domains because these tasks depend on structured, exclusive signals (e.g., spatial queries, video metadata) that web search cannot reliably provide. (3) Accuracy drops on multi-domain tasks with many tools, highlighting cross-source orchestration issues; selecting and chaining tools raises planning complexity and error propagation.
Tool use result analysis. To analyze the reason why the accuracy will drop in certain domains. We collect and categorize the results of tool calls into four types: usage error (wrong function calling), selection error (wrong tool selections), invalid result (successful but irrelevant/unhelppull calling), and valid result (success- ful and useful tool calling). Fig 4a shows the distributions of these types in each domain. We have the following findings: (1) As shown by the line, better tool usage yields more useful information and leads to stronger model performance. (2) Tool usage error rate correlates with tool complexity. Bio and Multi- domain with larger parameter numbers exhibit higher usage-error rates. Finance and Multi-domain host the largest toolsets and show markedly higher selection error rates, implying larger tool inventories increase selection risk. (3) Most tool results are unhelpful and contribute little to answering the question.
5.3.2 SCALING ANALYSIS
We analyze the relationship between the number of tool calls and performance. Fig. 3a and 3b report results of GLM-4.5 across three representative domains (Finance, Bio, and Video), showing how accuracy and pass rate change with the number of tool calls. Fig. [c compares 4 different models, indicating the growth of input token length as the number of tool calls increases. We find that: (1) Ace and PR generally increase with more tool calls. (2) Performance plateaus after 8 tool calls, with further tool calls sometimes reducing accuracy and pass rates due to redundant rather than useful information. (3) Fig 3c input token length rises quickly with the number of tool calls until a turning point, after which extra calls add little context. This turing point reflects each model's effective tool-usage limit. Across models, it correlates with overall accuracy (R² 0.57), indicating moderate positive link between effective tool-call capacity and task performance.
5.3.3 FAILURE MODE ANALYSIS
Fig. 4b shows GPT-5's web-only failure distribution on InfoMosaic- -encch using a six-class, primary-cause label (Retrieval Miss, Tool Misuse, Reasoning Gap, Confirmation Bias, Overgeneralizzatio, Context Mis- read) which is detailed in Appendix A.3 Retrieval Miss (39.6%) and Overgeneralizatioon (28.2%) dom- inate, indicating failures stem mainly from retrieval and evidence selection rather than final-step reason- ing underscrcnn the need for domain tools and stronger search orchestration.

---

Published as conference paper ICLR 2026
Accuracy Number of Tool Calls
Pass Rate Number of Toal Calle
Tool
Domate
10000
LLH Name
Vidw
Phrs Rate
Vilee
2000
Number That Calle (log scato)
Number Teol Calls (tog acaln)
Tool Calls
(a) Num of tool calls v.s. Acc
(b) Num of tool calls v.s. Avg PR (c) Num of too calls v.s. Avg Length
Figure 3: Relationship between performance and input length. (a) and (b) show results of GLM-4.5's Acc and PR across Finance, Bio, and Video domains. (c) compares different models, showing average input token length against the number of tool calls.
Distribution of Tool Call Types
Distributton of Error Types
Bafore
Refinement
Detore Human Aetiseneit
+0.102
Parcestuge (N)
4.8
Score
0.6
Ramocting Cap
Contimaaiion Blac
+0.101
Overgeneraitaaiinn
(851 SPL
Condect Miaread
4.
Video
Finance
Web Multidomain
Bio
4.0
Coberatce
(a) Distribution of tool-call result types in 6 domains.
(b) Distribution of 6 error types only using the web scarch.
(c) Comparison of human evaluation before and after refinement.
Figure 4: Distribution analysis. Fig. 4a shows the distributions of 4 tool calling result categories 6 domains, Fig. 4b shows 6 failure modes of GPT-5 with only web search, and Fig. 4c shows improvements after refinement.
5.4 HUMAN STUDY
We conduct a human study on 120 randomly sampled problems across domains: three NLP-trained graduate annotators independently rated pre- and post-check versions on factual alignment, coherence, and task dif- ficulty (guidelines in Appendix A.II.I). Fig 4c) (I) high pre-check scores, indicating strong baseline quality from tool-grounded synthesis; and (2) the largest post-check gain in factual alignment, correcting evidence-answer mismatches. Agreement is high (Cohen's 's x = 0.92), confirming reliability.
6 CONCLUSION
In this work, we propose InfoMosaic-Bende the first benchmark in evaluating multi-source information seeking in tool-augmented agents, spamning 6 domains with 77 heterogeneous tools. To construct domain- wise and cross-domain information seeking tasks, also proposed InfoMosaic-Flow a scalable synthesis methodology. Our extensive experiments demonstrate that (i) web search alone is insufficient for domain- specific information seeking tasks, (ii) current agents remain disproportionately better at web search than at leveraging domain-specific tools. We expect InfoMosaic-Bent principled, auditable multi-tool information seeking, accelerating progress on high-stakes domains such as finance and science. Future work may extend the synthesis pipeline to additional modalities, interactive environments, pushing LLM agents closer real-world deployment.
to catalyze a shift from web-only search
10

---

Published as a conference paper ICLR 2026
7 ETHICS STATEMENT
Our paper fully complies with the ICLR Ethics guidelines. Specifically, the research is designed to contribute positively to society and human well-being by proposing InfoMosaic-Bente the first benchmark dedicated to multi-source information seeking in tool-augmented agent, with careful consideration of potential risks and mitigation strategies to avoid harm. We make sure our experiment results are reproducible and our code and dataset are accessible. This study does not involve any trade secrets, client data, non-public business strategies, financial information, research data, pre-publication scholarly articles, and patent applications. The data used for the synthesis all publicly available, and citations have been added in the appendix. We comply with the corresponding licenses to ensure respect for the original sources. All human labor involved in this paper has been conducted with full respect for the workers and creators. Since our data come from real-world sources, there is no issue of bias from human annotators. Our research does not involve any harm or discriminatory practices.
8 REPRODUCIBILITY STATEMENT
Our code and dataset available here httph:htttttttiiiee http://www...ooooooooooottnnn/to Our approach and experiments were designed with reprodueibility in mind, and we encourage others to build upon our work using the provided resources.
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, and et al. Language models few-shot learners. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xinyu Zhu, Mengcheng Zhou, Yanfeng Wang, Yuzhi Zhang, Linfeng Zhang, Siheng Chen, al. Scimaster: Towards general-purpose scientific ai agents, part i. x- master as foundation: Can we lead on humanity's 's last exam? arXiv preprint arXiv:2507.05241, 2025.
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang, al. Acebench: Who wins the match point in tool learning? arXiv e-prints, pp. arXiv-2501, 2025.
Yaxin Du, Yuzhu Cai, Yifan Zhou, Cheng Wang, Yu Qian, Xianghe Pang, Qian Liu, Yue Hu, and Siheng Chen. Swe-dev: Evaluating and training autonomous feature-driven software development, arXiv preprint arXiv:2505. 16975, 2025.
Xiang Fei, Xiawu Zheng, and Hao Feng. Mcp-zero: Proactive toolchain construction for Ilm agents from scratch. arXiv preprint arXiv:2506.01055 0000000006. 2025.
Matthias Flotho, Ian Ferene Diks, Philipp Flotho, Leidy-Alejanrra G Molano, Pascal Hirsch, and Andreas Keller. Mcpmed: A call for mep-enabled bioinformatics web services for llm-driven discovery. arXiv preprint arXiv:2507.08055, 2025.
Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. Mcp-radar. A multi -imensidimensionall benchmark for evaluating tool use capabilities in large language models. arXiv preprint arXiv:2505.16700) 2025.
Google. Notebooklm:
Ai research tool & thinking partner, 2025.
URL https:/notebookin
google/|
Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model context protocol (mcp): Landscape, security threats, and future research directions. arXiv preprint arXiv: 2025.
11

---

Available at Published as a conference paper ICLR 2026
Jared Kaplan, Sam McCandlish, Tom Henighan, and al. Scaling laws for neural language models. arXiv preprint, 2020.
Shilong Li, Xingyuan Bu, Wenjie Wang, Jiaheng Liu, Jun Dong, Haoyang He, Hao Lu, Haozhe Zhang. Chenchen Jing, Zhen Li, et al. Mm-browsecomp: A comprehensive benchmark for multimodal browsing agents, arXiv preprint arXiv:2508. 13186, 2025a.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint 2xXi::5501.05366 2025b.
Xiaoxi Li, Jiajie Jin, Guanting Dong. Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.2177. 2025c.
Tim Loughran and Bill McDonald. When is a liability not a liability? textual analysis, dictionaries, and 10-ks. Journal of Finance, 66(1):35 65, 2011. doi: 10,1111/11140-66612220 2211440006666122100066662..... SSRN: https: ssrn. wwwm/bastaattt1333
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang. Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. Mcp-universe: Benchmarking large language models with real-world model context protocol servers. arXiv preprint arXiv:2508.1470. 2025.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, and et al. Webgpt: hwwpesrrraasssceeeeuuusooooannss with
human feedback. arXiv preprint, 2021.
OpenAL. Gpt-4 technical report. arXiv preprint, 2023.
URL https://array..og//ab////
08774
OpenAl. Introducing deep research, February 2025.
URL https://openai.com/nness
detr research/
OpenAl. Function calling openai api guide. httpp://ppaatffomooee.aa. pppeeaai.com/ddcs/ggiidee
Eunction-cal celloo-caalinn 2025. accessed 2025-09-22.
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab, November 1999. URL
Xianghe Pang, Shuo Tang, Rui Ye, Yuwen Du, Yaxin Du, and Siheng Chen. Browsemaster: Towards scalable web browsing via tool-augmented programmatic agent pair. arXiv preprint arXiv:2508.09129, 2025.
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning.
Perplexity AI. Introducing perplexity deep research, 2025. URL Dttp:://ww..ppppeeiiiy.ooo research,
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307. 16789, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Newral Information Processing Systems, 36:68539-6888 2023.
12

---

Published as a conference paper at ICLR 2026
Zhuocheng Shen. Lim with tools: A survey, arXiv preprint arXiv:2409.18807 2024.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Rong Wen. R1-searcher: Incentivizing the search capability in Ilms via reinforcement learning, arXiv preprint arXiv:2503.05599, 2025.
U.S. Food and Drug Administration. Clinical decision support software: Guidance for industry and food and drug administration staff (final guidance, september 2022). https:://w fda. www.ggo////eeeo a/ 109618 down 1 oad 2022. Docket No. FDA-2017-D-6569.
Soroush Vosoughi, Deb Roy, and Sinan Aral. The spread of true and false news online. science, 359(6380): 1146-1151, 2018.
Wenhao Wang. Peizhi Niu, Zhao Xu, Zhaoyu Chen, Jian Du, Yaxin Du, Xianghe Pang, Keduan Huang, Yanfeng Wang, Qiang Yan, et al. Mcp-flow: Facilitating Ilm agents to master real-world, diverse and scaling mep tools. arXiv preprint arXiv:2510.24284, 2025a.
Zhenting Wang, Qi Chang, Hemani Patel, Shashank Biju, Cheng-En Wu, Quan Liu, Aolin Ding, Alireza Rezazadeh, Ankit Shah, Yujia Bao, al. Mcp-bench: Benchmarking tool-using IIm agents with complex real-world tasks via mep servers. arXiv preprint arXiv:2508. 2025b.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging bench- mark for browsing agents. arXiv preprint arXiv:2504. 12516, 2025.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asun- cion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfih Language Resources and Evaluation Conference, pp. 4003-4012, Marseille, France, May 2020. European Language Resources As- sociation. ISBN 979-10-95546- 34-4.
htt::/aaaaaannn htt.t:pps///..aaaaoooooooooo/// org 2t1p: wtw.loooyorrg/22..11eeeee
494/
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang. Zekun Xi, Runnan Fang. Linhai Zhang. Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking Ilms in web traversal. arXiv preprint arXiv:2501.0753 0055020..772,2222 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference Learning Repre- sentations (ICLR), 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Repre- sentations (ICLR), 2023b.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool- agent-user interaction in real-world domains. arXiv preprint arXiv:2406. 2024.
Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. Easytool: Enhancing lim-based agents with concise tool instruction. arXiv preprint arXiv:240..0620), 2024.
Yifan Zeng. Quantmep: Grounding large language models in verifiable financial reality. CoRR, 2025.
13

---

Published as conference paper ICLR 2026
A APPENDIX
A.1 THE USE OF LARGE LANGUAGE MODELS (LLMS)
We acknowledge the use of large language models (LLMs) the this work. Specifically, GPT-5 was em- ployed for language polishing and refinement of the manuseript, while GPT-4o was used tool for eval- uation and error analysis. The LLMs did not contribute to research ideation, methodology development, or experimental execution.
A.2 USED DATA SOURCE
Table lists the data sources used to generate seed data and seeking information. We adhere to all knowledge-use policies of these websites and products.
Table 4: List of data sources.
Name
Website
Type
Wikipedia
www.wikipedia.org
Seed data source
Baidu Baike
sakeebbadu.com
Seed data source
Qunar
www.qunar.cmm
Seed data source
NORD
rarediseases.ong
Seed data source
Clinica/Trials.ory
clinicaltrials.go
Seed data source
AMap MCP
MCP server
Google Map MCP
MCP server
Bio MCP
gibubb.cm//aaomooooolyyyonnn
MCP server
YouTube MCP FMP MCP
MCP server
ggthub.con/ccaa///mmmmppeeerrr
MCP server
A.3 PROBLEM FORMULATION
Notation. Let the domain be the queries in D be QD and the user query be q E 2p. Let the of available tools be
Tavail (T....., Tm
where each tool T; is specified by interface metadata (name, description, parameter schema, output schema). Let GT denote the ground-truth answer to q and K be the max tool calling limit. A task instance denoted as
(g, Tavail K, GT).
Evaluation. Let M (M1, M2, ...,. Mn} denote the of LLMs and A {A1, A2, Ap} the set of agent frameworks. For a given (M, A) E M x AA and task T, the interaction generates a message history
H (h2, h2,...,.... hr),
where each h, = (r. (M..)) (oool) records the responses from agents and any tool invocations. The evaluation function
E: (H,,)) + 10.1}
assigns 1 if the query is correctly solved under predefined success criteria, and otherwise. Correctness is determined by automated checks (e.g., verifying output content or exactly matching). Details of evaluation could be found in Sec. 5.1
14

---

Published as a conference paper ICLR 2026
A.4 EXPERIMENT SETUP
A.4.1 MODELS
We evaluate a broad spectrum of current top tier LLMs, including both close-sourced and open-sourced models. Soble their key features such as model size, release date, openness, and access links. Through the evaluation of these latest state-of-the-art LLMs, we ensure the validity and reliability of our main conclusions.
Table 5: Details about evaluated LLMs.
Model
Size
Releuse Date
Status
Link
GPT-5
2025-08-07
Closed
3stps://ple Form. www.apeeaa......... con/ docs mmsdes/mmeeeeeee h/pptteees//p/pleee//eeegg 5
Grok-4
2025-07-09
Closed
netpa: att////..////////////0/
Claude-4.0-Somee
2025 (15 23
Closed
https: Via and hnoprropi com/ c aude/ sonnet
o3
2025-04-16
Closed
hEt //plttt//att// Form, openaa cos. does mode 13/63
04-mini
2025-04-16
Closed
btips://plas
openal ccm/ doca mmoce
5aax
2025-01-25
Closed
2 /qwen im hibb 10/
qwen max
Gemini-22.5flas
2025-06-17
Closed
SiEtpp://deep.n... wd....g..oooo........... goog 1e mode /et.el/ggeeennaaaaaaa
GI.M-4.5
355B
2025-07-28
Open
Qwen3-235B-AN 235B-A22B
235B
2025-04-22
Open
ht pst huggungggggg Face 20 Vwen/ Veren 225--
Qwen3-32b
32.8B
2025-04-29
Open
hhicss///hignnnee www.eee....... C07 co/ owen 323
Llama-4-
109B
2025-04-05
Open
Pteps://hagen ngrace,
ama/
DeepSeek V3
685B
2025 03 24
Open
Ettps: /hoqg ne
nofaaat Face cp/ deepseek bocpseeos--aooepeeeeee
Kimi-K2 K2
2025-07-11
Open
https://hogg ng aaca wtt.ee.oooooooooooooooooooo hoop/ooooooaaaaaaooooooooo
nusr
Qwen3 Coder
30.5B
2025-07-22
Open
ht htops:/ hpg://huggaggtae
A.4.2 AGENT FRAMEWORK
We build our agent framework based on ReAct (Yao et al., 2023a), which serves as the foundation of the agent's reasoning and actions. The framework integrates Python Sandbox and follows the OpenAl function calling interface (OpenAl 2025), enabling the L.MM to invoke external tools and consume their outputs in a structured manner.
Concretely, we implement multi-turn interactions where tool metadata is serialized into JSON Schema for- mat and provided to the LLM through the function calling interface. The LLM responds with a structured tool call request, which we automatically translate into Python code snippet. This code is then executed inside the Python Sandbox (Pang al. 2025), and the execution results (standard outputs or errors) are captured. Finally, the returned results are appended to the dialogue history and passed back to the LLM as additional context for subsequent reasoning steps. We set the tool calling limit to 20, ensuring that the agent terminates tool calls after repeated unsuccessful attempts to solve the problem.
This design allows the framework (i) support multiple domain-specific tools in a uniform schema, (ii) enforce execution safety through sandbox isolation, and (iii) tightly couple the reasoning trace of the LLM with actual tool outputs.
A.5 DETAILS OF FAILURE MODES ANALYSIS
We define six categories of failure: (1) Retrieval Miss, where the agent fails to extract key information already present in the tool knowledge base, often producing "cannot determine" answers or overlooking relevant facts; (2) Tool Misuse, where the agent misinterprets tool outputs or provides incorrect parameters commands, leading to irrelevant, distorted, erroneous results (e.g., miscalculations); (3) Reasoning Gap, where correct information fragments are retrieved but not coherently linked, resulting in logical jumps, causal confusion, inconsistent conclusions; (4) Confirmation Bias, where the agent selectively emphasizes
15

---

Published as conference paper ICLR 2026
Table Effect of planner-executor design. Removing the executor collapses tool usage and reduces task complexity.
Executor Setting
Avg. Tool Calls
Avg. Tools Used
With Executor 59.1
43.1
Without Executor
7.8
5.6
evidence supporting its initial hypothesis while downplaying ignoring contradictory tool outputs; (5) Overgeneralization and Hallucination, where insufficient evidence leads to unfounded guesses or fabricated details not supported by tool results; and (6) Instruction and Context Misunderstanding, where the agent misinterprets the user's intent task context, producing answers that may be partially correct but irrelevant to the actual query.
A.6 T-SNE VISUALIZATION OF QUERY EMBEDDINGS
Fig. 5 reveals clear domain-level clustering of query embeddings, indicating strong semantic separability across domains. Limited overlap suggests occasional cross-domain ambiguity, while dispersion reflects intra-domain diversity.
A.7 VALIDATING THE SYNTHESIS PIPELINE
We perform ablations to verify the necessity of the two core components of InfoMosaic-Flow: web-based verification and planner-executor interaction.
t-SNE Visualization of query different domains
Domain
Effect of Planner-Executor Interaction. The default pipeline employs GPT-5 as the planner and executor pair. As shown in Table 6 removing the executor collapses the synthe- sis space. With exccutor, synthesized tasks involve on average 59.1 tool calls across 43.1 unique tools. Without executor, this drops sharply to 7.8 calls across 5.6 tools, indicating severe loss of heterogeneity and task richness.
tic
map
dutdoomah video
Effect of Web-Evolving Verification. Stage-2 verification is implemented with GPT-5 acting both the main synthesizer and executor. Table 7 show that without this step, many tasks collapse to trivial single-source queries, yielding inflated ac- curacy (45.1%) To test pruning strategies, we substitute the executor with GPT-5. Quick Fuzz mode, the GPT-5 execu- tor rewrites conditions without invoking web tools, modestly reducing shortcuts but leaving residual triviality (39.7%). Ad- vanced Fuzz applies iterative probing and constraint rewriting, lowering accuracy to 31.3%, which indicates stronger resis- tance to shortcuts and closer alignment with true multi-source reasoning.
-10-
-20
Figure 5: t-SNE visualization of query em- beddings across domains. Each point is a query, colored by domain.
Web-evolving verification prevents trivial shortcuts, and planner-executor interaction ensures sufficient task richness. Both are indispensable for generating a benchmark that reflects realistic multi-source reasoning challenges.
16

---

Published as conference paper ICLR 2026
Table Effect of web-evolving verification. Without pruning, many tasks collapse to trivial web lookups, inflating accuracy. Advanced fuzzing most effectively suppresses shortcuts.
Verification Setting
Accuracy (%)
w/o Condition Pruning
45.1
Quick Fuzz (GPT-5, no web)
41.7
Advanced Fuzz (GPT-5, web)
31.3
A.8 AGENT PROBLEM-SOLNVE ANALYSIS
A.8.1 PATTERNS ANALYSIS IN SEARCHING PROBLEMS
After analyzing the trajectories Agents solving dataset problems, we can find that advanced Agents with higher evaluation scores can already exhibit clear, structured thought processes and steps, which are not prompted by humans. Based on our analysis, GPT-5 demonstrates a specific, sequential long-range trajectory when solving problems in the map domain, which is: "Broad Search + Targeted Information Retrieval + Solution Evaluation - Response Calibration".
Broad Search: In the initial stage, the agent autonomously chooses and prefers to use generalized keyword which retrieve keywords and return a larger number of candidate answers and their accurate place IDs. The main purpose of this step is to include the correct answer through a generalized search, thereby constraining and narrowing down the broad search space into list of choices.
searches, such as maps waapsstex.seerreee and
Targeted Information Retrieval: After searching for candidate answers, the agent proactively invokes more fine-grained search tools perform deep, targeted search for specific POI. For example, agent may call www.cch.eeaaaallll wecchdetailll with an accurate ID provided by broad-search tools or functions named maps_de .dd stance for targeted information seeking. The use of these tools enables the model to conduct precise searches on the candidate answers, allowing it to further filter out unreasonable options and constrain the search space a smaller of candidates.
functions maps_see wwwssssearch...
Solution Evaluation: After filtering, the model is left with 3 to 4 highly similar candidate answers. At this point, in addition to using deep search tools, the model integrates all the information to perform its final reasoning and selection.
Response Calibration: Eventually, agent call tools validate the candidate answer.
Fig. о shows the frequency of making diverse tool calls when solving map problems. We calculate the frequency of invoked tools tool-call sequence of GPTT-5 and. We segment the action trajectory into 8 relative positions, representing different stages of the entire thought process. It is obvious that the agent prefers to call broad search tools seaas_ttet_ssee ginning of the action trajectory. Then, deep and targeted search tools including naps-search.dete il and google_mape www.aaapssssseeeeeeeeeeeeeeeeaee seearchhplace. begin to emerge. At last, in the validation, more tool calls of maps di stance used for checking the candidate answer.
and
mostly at the be-
A.8.2 ERROR STEPS ANALYSIS
Based segmentation of agent's 's tool-calling trajectory, further investigated the specific step where the agent's error occurred. The results are shown in Fig. 7 We find that agents are most prone to errors at Broad Search, namely when the agent fails retrieve the correct and add it the candidate list, especially when agent is performing keyword searching. Improper keywords composition may lead to an incorrect empty search result hit.
17

---

Published as conference paper ICLR 2026
Tool Calls position Long-range Trajectory of Agent
21.0%
11.6%
13.2%
10.5%
9.85
12.1%
8.3%
13.5%
MADK settll
13.75
14.2%
19.5%
9.55%
11.3%
12.3%
8.9%
11.1%
-2
19.9%
19.2%
13.8%
15.1%
7.7%
0.0%
7.4%
8.0%
Parcentage of
28.3%
googlemap_seom pparceppperppeeeee
15.0%
14.4%
Relative Position Bin (0-7)
Figure 6: Heatmaps for sequence of tool calls made by the GPT-5 when solving map domain problems.
Error Step Analysis Tool Action Trajectories
40
20
20
10
Enor Samples
20-
20
40
Sage 1. Bmad Search
Stape 2: Targeted Information Nntrival
State Solation Ematuatin
Stnge Resports Calbration
Stapes
Figure 7: Distribution of agent errors across the segmented steps of the tool-calling trajectory
18

---

Published as conference paper ICLR 2026
The following exampld" demonstrates how the agent's use of incorrect keyword combinations (either complex too generalized) resulted in the tool s search failing to return hit. In this example, too restritive searching keywords lead to empty response.
function";
"arguments": " "("query": "Yuecheng District, Shaoxing, Scenic spots and stone bridges near Changqiao Zhijie")", "name": "googlemap searrch search places"
b
"type": "function"
"role": "tcol",
"content"; "('tool_resule 'The Basic information for the query: Yuecheng District, Shaoxing, Scenic spots and stone bridges near Changqiao Zhifie with 0 results')
A.9 REPRESENTATIVE BENCHMARK INSTANCES
A.9.1 EXAMPLES
The following examples illustrate instances from various domains in our dataset, showcasing how queries are structured with associated testcases and ground truths (GT). cases where a testcase condition can be decomposed into subquestion:subansser format, the corresponding GT is provided and is not null, ensuring verifiable details. However, if condition lacks an explicit subanswer (e.g., due its nature as a direct factual check without granular breakdown), it is split into multiple independent testcases for thorough inspection and validation.
Model response in Chinese, translated in English
19

---

Published as conference paper ICLR 2026
Bio Domain
Query
A clinical trial aimed to assess the impact of a RET inhibitor on cardiac parameters healthy adults. Which NCT number identifies the trial with these features?
1. The trial utilized a thorough QT/QTe (TQT) design with randomized, double-blind, four-period crossover approach, evaluating two doses of the study drug against placebo and moxifloxacin a positive control.
2. Eligibility required serum potassium 2 3.8 mEg/L, calcium 2 8.5 mg/dL, and magne- sium 2 2.0 mEq/L at screening, along with a BMI of 18 32 32kk-kkkg//m kg/m".
placebo- aijuebo-adjusted QTcF interval changes (AAQTcF) measured via 12-lead ECG over 24-hour period post-dose.
3. The primary endpoint involved modcl-predicted
4. Conducted single site in Tempe, Arizona, trial's primary phase concluded
June 2019, with results first published in Septemher 2025.
Testcase (Condition)
The trial utilized thorough QT/QTc (TQT) design with randomized, double-blind, four- period crossover approach, evaluating two doses moxifloxacin as a positive control.
study drug against placebo
Testcase (Ground truth)
Single-dose, randomized, double-blind, placeho- and positive-contrroled, 4-way crossover; treatments included selpercatinib 320 mg, selpercatinib 640 mg, placeho, and 400 mg mox- ifloxacin (positive control).
Testcase (Condition)
Eligibility required serum potassium 2 3.8 mEq/L, calcium 2 8.5 mg/dL. magnesium 2 2.0 mEq/L. at screening, along with a BMI 18-32 kg/m".
Testcase (Ground truth)
Exclusion K* < 3.8 mFg/L, 2+ < 8.5 mg/dl., Mg 2.0 mEq/L; BMI 18.0 and < 32.0 kg/m².
Testcase (Condition)
The primary endpoint involved model-predicted, placeho-adjusted QTcF interval changes (AAQTeF) measured via 12-lead ECG over a 24-hour period post-dose.
Testcase (Ground .rruh
Primary outcome: placeho-correcte change from baseline in QTcF (AAQTeF) based on model-predicted effect; 12-lead ECG extracted from continuous recordings at pre-dose and to 24 hours post-dose.
Testcase (Condition)
Conducted single site in Tempe Arizona, the trial's primary phase concluded in June 2019, with results first published in September 2025.
Testcase (Ground_truth
Site: Celerion, Tempe, Arizona, United States; Primary completion date: 2019-06-21; Results first posted: 2025-09-18.
GT (NCT number)
NCT05630274
Table 8: Example instance from the Bio domain.
20

---

Published as a conference paper ICLR 2026
Map Domain
Query
+ T 1. X ( 22 2. W
#12.40 H. 3. A 44 121
m X S3
MR #E? (Translation: Zhongshan District of Dalian, find a landmark that meets the following criteria: (1) It administrativly belongs to Zhongshan District (division code 210202): (2) Walking from Qingniwagiao Metro Station this place takes about 31 minutes 59 seconds, with walking distance about 2.40 km;; (3) Driving from Dalian Railway Station to this place cavers about 4.22 km and is estimated to take about 12 minutes 59 seconds, whereas taking Metro Line 5 from Dalian Railway Station to Laodong Park Station and then walking takes about 42 minutes 22 seconds in total, so driving is more than 5 minutes faster. What is the name of this landmark?]
Testcase (Condition)
AE VE IX R T (I (1
|Translaaion: Administratively be-
longs to Zhongshan District (division code 210202).]
Testcase (Ground truth)
nul1
Testcase (Condition)
1 11 / (Translation: Walking from Qingniwaqiao Metro Station this place takes about 31 minutes 59 seconds, with walking distance of about 2.40 km.]
M
Testcase (Ground _tuth) Testcase (Condition)
null
M
THT THEEE 1
M $4 [Translation: Driving from Dalian Railway Station this place covers about 4.22 km and is estimated to take about 12 minutes 59 seconds; in contrast, travelling from Dalian Railway Station by Metro Line 5 to Laodong Park Station and then walking takes about 42 minutes 22 seconds in total, so driving is more than 5 minutes faster.
Testcase (Ground truth)
null
GT (Toponym)
A J /
[Translation: Dalian Sightseeing Tower]
Table 9: Example instance from the Map domain.
21

---

Published as conference paper ICLR 2026
Finance Domain
Query
Identify publicly traded technology company satisfying the following criteria: (1) Head- quarters located in Austin, Texas; (2) Positive net income reported in QI 2025 (quarter ending March 31, 2025), but negative net income in Q2 2025 (quarter ending June 30, 2025); Year-to-date stock price change of September 19, 2025, between 10% and 20%. with beta coefficient exceeding 5.0; (4) Q2 2025 revenue (quarter ending June 30, 2025) between S70 million and $80 million USD.
Testcase (Condition)
Its headquarters is located in Austin, Texas.
Testcase (Ground (trounn__tuuhh)
Austin, Texas
Testcase (Condition)
It reported positive net income in QI 2025 (quarter ending March 31, 2025) but negative net income in Q2 2025 (quarter ending June 30, 2025).
Testcase (Ground truth)
Q1 2025 net income: $580, 693, 000 (positive); Q2 2025 net income: $933, 799, 000 (negative)
Testcase (Condition)
Its year-to-date stock price change of September 19, 2025, between 10% and 20%, and heta coefficient exceeds 5.0.
Testcase (Ground_truth
YTD change as of 2025-09-19:
68%; Beta: 6. 6067
Testcase (Condition)
Its Q2 2025 revenue (quarter ending June 30, 2025) was between $70 million and $80 million USD.
Testcase (Ground truth)
$78, 628, 000
GT (Stock Symbol)
CORZ
Table 10: Example instance from the Finance domain.
Video Domain
Query
On YouTube, there is a video that meets all of the following conditions: 1) Uploaded in September 2025; 2) Duration is approximately minutes and 36 seconds; 3) From chan- nel with over million subseribers; both "Earthquake" and "Baby seene", Which vidco this? Provide URL..
top comment with over 1,000 likes mentions
Testcase (Condition)
Uploaded in September 2025 null
Testcase (Ground truth)
Testcase (Condition)
Duration approximately minutes and 36 seconds; null
Testcase (Ground (ruuh)
Testcase (Condition)
From channel with over 1.7 million subscriber null A top comment with 1,000 likes mentions both "Earthquake" and "Baby scene" null
Testcase (Ground truth)
Testcase (Condition)
Testcase (Ground (Ground_truth) truth)
GT (URL)
https: http://///oo.uuuuuueuueeeeee
Table 11: Example instance from the Video domain.
22

---

Published as conference paper ICLR 2026
Web Domain
Query
a transoceanic venture undertaken by two vessels, the smaller craft commander with a rank-style title. while another man actually held the practical authority for naviga- tion. The sailing schedule slipped because that man went inland to collect debts, and much later regional museum hosted public lecture centered him. Who was he?
Testcase (Condition)
In a Iransoceanic venture undertaken by two vessels Ark and Dove the smaller craft had commander with rank- title Captain Wintour
Testcase (Ground truth)
Testcase (Condition)
Testcase (Ground (truh)
Testcase (Condition)
Historic St. Mary's City (HSMC)
Testcase (Ground truth)
Richard Orchard
GT (Person Name)
Richard Orchard
Table 12: Example instance from the Web domain.
Multi-Domain
Query
Identify the stock symbol of U.S.-listed company fulfilling all criteria: (1) Headquarters in U.S. city renowned for its high density of universities; (2) Farly-year quarterly report indicates minimal revenue and per-share loss slightly less than dollar; (3) Share price more than doubled over mid-2025 season, trailing six-month total return around 80-100% by early August; (4) Develops therapeutics via modality that directly edits DNA sequences for monogenic disorders.
Testcase (Condition)
Headquarters in U.S. city renowned for its high density of universities:
Testcase (Ground truth)
Cambridge, Massachusetts, USA
Testcase (Condition)
Early-year quarterly report indicates minimal revenue and per-shure loss slightly less than one dollar
Testcase (Ground truth)
Q1 2025 revenue N $4.658 million; diluted EPS .20.9
Testcase (Condition)
Share price more than doubled over mid-2025 season, with trailing six-month total return around 80-100% by early August)
Testcase (Gruund.truth)
Share price rose from $1.13 (2025-04-01) to $2.51
(2025-07-31), +1228; 6-month total return as of 2025-08-01 ~ +928
Testcase (Condition)
Develops therapeutics via modality that directly edits DNA sequences for monogenic disorders.
Testcase (Ground truth)
CRISPR gene editing therapcutics (direct DMA sequence editing for monogenic diseases)
GT (Stock Symbol)
EDIT
Table 13: Example instance from the Multi-Domain.
A.9.2 LIST OF TOOLS USED IN THE BENCHMARK
The following presents a comprehensive list of all tools used our benchmark, categorized by their func- tional domain. Each tool is listed with its name and brief description.
23

---

Published as conference paper ICLR 2026
Domain: Map (AMap Services and Google Maps Services)
Tools for geographic information and routing services within China, powered by the AMap server.
Tool Name
Description
maps regeocode
Converts a longitude/latitude coordinate into an administra- tive region address.
maps.geo
Converts structured address into longitude/lattude coor- dinates.
maps_ip_location
Determines the geographic location based on an IP address. Retrieves real-time weather information for specified city. Plans bicycle route between two locations using ad- dresses. Unless you have a specific reason to use coordi- nates, it's recommended use this tool.
maps_weather
maps.bicycling.by.aae adddres
maps haaps_icyyling.bll by coordinates
Plans a bicycle route between two coordinates. Plans walking route between dresses. Unless you have a specific reason to use coordi- nates, it's recommended use this tool.
maps citirectton_walkiiie by address
locations using ad-
wtw.ddrrrcttooowwllllnig... coordinaats
Plans a walking route based on start and end longitude/lati- tude coordinates.
maps direction driving by address
Plans driving route between two locations using ad- dresses. Unless you have a specific reason to use coordi- nates, it's recommended to use this tool. Plans a driving route based on start and end longitude/lati- tude coordinates.
maps.direction.drii by ccooddinates
maps direction transit integrated by address
Plans a public transit route between two locations using addresses. Requires origin and destination city names for cross-city transit.
hranrpccoo.rraassiiiitteerdddddd _oooddinttss
Plans a public transit route based start and end coordi- nates. Requires origin and destination city names, Measures the distance (driving, walking, or straight-line) between two coordinates.
maps.distamm distance
maps search
Searches for Points of Interest (POI) by keyword within specified city.
maps_around_searre
Searches for POIs near a specified coordinate and radius.
maps.search.detail
Retrieves detailed information for POI by its ID.
Tools for interacting with Google Maps for global geographic searches and directions.
Tool Name
Description
googlemap search places google map get place details
Performs a fuzzy search for places on Google Maps. Retrieves detailed information (reviews, hours, etc.) for specific place.
google_man peclllmappgee.ppee get.placc google saggll_map_eeee haap--ettmapddreeeeee
Returns the place ID(s) for places matching a search query. Fetches step-by-step travel directions between two loca- tions.
24

---

Published as conference paper ICLR 2026
Domain:
finance (Financial Data Services)
Tools for accessing stock, market, commodity, and cryptocurrency data, primarily powered by the FMP server.
Tool Name
Description
getpomn notes
Gets detailed information about company-issuce notes and
debt instruments.
incommesstatemet statement get_quote
get_income
Retrieves the income statement for company.
Gets the current stock quote information.
det_quote_change
Gets stock price change over different time periods.
get.afiermarket.quue
Gets aftermarket trading quote information.
get.price_change
Gets price changes for stock based on historical data.
search by symbol
Searches for stocks by ticker symbol.
search. by name
Searches for stocks by company name (English only).
get wwwarrattnnnsssttiieee
Gets analyst ratings snapshot for company.
Gets analyst financial estimates for company.
get_priceetarget_nnes
Gets the latest analyst price target updates.
Gets the latest price target announcements with pagination.
Gets dividend history for a specific company.
get dividends calendar
Gets calendar of upcoming dividend events for all stocks.
get index list
Gets a list of available market indices.
get_index _uote
Gets the current quote for market index.
get bhiggest_gainert
Gets a list of stocks with the biggest percentage gains.
geL.bigget__losse losers
Gets list of stocks with the biggest percentage losses.
get_most_aatvv active get market hours
Gets list of most actively traded stocks by volume.
Gets the current market hours status for specific stock ex- change.
get_commodities_litt
Gets list of available commodities.
set_comoodities_priies
Gets current prices for commodities.
ceistorical.pric....
Gets historical price data for commodity.
get crypto list
Gets list of available cryptocurrencies.
Gets current quotes for cryptocurrencies.
get forex list
Gets list of available forex pairs.
Gets the current quote for a forex pair.
get_ema
Gets Exponential Moving Average (EMA) values for stock.
Domain: Medi Medical//Bology (Biomedical Research Services)
Tools for searching and retrieving data from biomedical databases such PubMed, ClinicalTrials.gv, and My Variant.iinfo..
Tool Name
Description
search
Universal search across all biomedical domains with unified query language.
25

---

Published as conference paper ICLR 2026
Tool Name
Description
fetch
Retrieve detailed information for any biomedical record; auto-detects domain if not provided.
anticlessercche
Searches PubMed/PubTatorS) for research articles and preprints about genes, variants, diseases, or chemicals. Fetches detailed information (abstract, full text) for spe- cific article by its identifier.
article-getter
searcher
Searches ClinicalTrraass gov for clinical studies based conditions, interventions, location, etc.
getter
Fetches comprehensive details for specific clinical trial by its NCT ID.
trial protoooloooteeerr
Fetches core protocol information (title, summary, design, eligibility) for a clinical trial. Fetches publications and references linked a clinical trial. Fetches outcome measures and results data for a clinical trial.
trial references getter
trial outcomes getter
trial Mialllcctionssgette
Fetches contact and location details for sites participating a clinical trial.
searchrr
Searches My -nriintt.innoo for genetic variant database records (frequencies, significance, predictions). Fetches comprehensive details for specific genetic variant by its ID.
variant getter
gene_getter
Get gene information from MyGene.info, including official name, aliases, genomic location and database links.
disease
distase_getter getter
Get disease information from MyDisease. including definition, synonyms, ontology IDs and phenotypes.
drug-getter
Get drug or chemical information from MyChem in- cluding structure, mechanism, indications, trade names and identifiers.
Domain: web (General Web Services)
General- purrose tools for web searching and content parsing.
Tool Name
Description
web_search
Performs general web search for information.
web_parse
Parses a specific webpage or image URL to extract infor- mation based on a user query.
Domain: Video (Media Services)
Tools for searching and retrieving information from YouTube.
26

---

Published as conference paper ICLR 2026
Tool Name
Description
google_search. search images
Searches Google Images for pictures.
google_searche search_videos videos search_videe
Searches Google Videos for video content.
Searches for YouTube videos with advanced filtering op- tions.
w...iidoodettiil
Gets detailed information about specific YouTube vidco.
get channel details
Gets detailed information about a specific YouTube chan- nel.
geL-vide__comentt get_video transscpt
get_video comments
Retrieves comments for specific YouTube video.
Retrieves the transcript for a specific YouTube video.
get_related_ videos get.trending.vides
Gets a list of videos related to a specific YouTube video. Gets list of trending videos on YouTube for specific re- gion.
get video enhanced transcript
Advanced tool for extracting, filtering, and searching within
YouTube video transcripts.
A.10 DISTRIBUTIONS OF QUESTION AND A ANSERR LENGTH
Fig. 8 and Fig. 9 illustrate the distribution of question and ground truth answer (GT) lengths, measured in tokens, for Chinese and English questions.
Both question and answer lengths exhibit skewed distributions, with most samples concentrated in shorter ranges and a long tail extending to larger lengths. This indicates that the benchmark primarily consists of concise inputs and outputs while still including subset of more complex, lengthier cases.
Question Length Distribution (Chinese vs English)
Length Distribution (Chinese English)
Chinese (tokens)
Chinese (tokens)
English (tokens)
English (tokens)
100
Number of Samples
30
Number of Samples
30
400
Length
Length
Figure Question Length Distribution (Chinese vs English).
Figure 9: Ground Truth (GT) Length Distribution (Chinese vs English).
27

---

Published as conference paper ICLR 2026
Domain
English
Chinese
video
multi-domain
map
135
web
100
0
finance
100
0
bio/med
83
0
Total
392
229
Table 18: Language (English / Chinese) distribution across domains in the dataset.
A.11 COMPREHENSIVE EVALUATION AND ANALYTICAL PROTOCOLS
A.11.1 HUMAN STUDY
The Human Evaluation Prompt provides detailed guidelines for assessing the Factual Alignment, Coherence, and Difficult of InfoMosaic-Beern Bench items across three evaluation dimensions.
what you will per item
Task statement (the question users must answer)
Ground Truth
Conditions (intermediate requirements)
Important rules
Use only the given materials (tack/conditiins//())) You can use outside search or prior knowledge.
2. Judge quality of the datasct item, not the general truth of the world.
Dimension Question: Dees the Ground Truth faithfully correspond to the stated conditions Specitically, can each condition be traced to part of the answer, and does the answer rely only information that is covered by the conditions?
Factual Alignment (Answer Condition Consistency)
How to judge: Check whether the conditions answer should:
Ground Truth mapping consistent. The
1. Be derivable by satisfying all listed conditions.
2. Not include extra information beyond the conditions. Not be solvable by ignoring Labels (cheose one):
conditions.
A5 Fully aligned: Every condition is reflected in the final answer; no inconsistent conditions for Ground Truth.
A3 Partially aligned: Answer covers the main intent, but some conditions not describing the ground truth
Al Misaligned: Final answer dees not respect condition (contradi it iiigg, or irrelevant condition ion)..
Dimension
Coherence (Semantic Logical Coherence)
Question: Is the item well-formed as dataset example? Are the task, conditions, and answer mutually consistent, unambiguous, and executable? How to judge: Check the clarity, nen-contradiction and referential coherence among task conditions ground truth.
Labels (choose one):
28

---

Published as conference paper ICLR 2026
Coherent: Task clear; conditions are interpretable and non- conflicting; no unnatural or self-contradictory phrasing.
B3
Minor issues: Small ambiguity, mild redundancy, or slightly awkward phrasing that does not block execution or interpretation Inccherent: Contradictors conditions, ill-formed references (e-g.r undefined entity), or conditions that cannot be executed with the given tools.
Notes:
If single condition directly gives away the answer, mark B1/B0 depending severity and explain (this will affect Difficulty below).
Task Difficulty (Need for Multi-Tool / Multi-Step Reasoning) Question: Would competent agent need multiple tools and/er multiple steps to solve this task using the provided domain tools?
Dimension
How to judge: Focus on necessity (not just the count of calls). If single condition or a single tool suffices, the task is trivial.
Likert score (1 5 ):
cl
Trivial: Solvable via tool
condition; integration needed Easy: Mostly one source plus minimal lookup.
C2
C3
Moderate: Requires combining 2 pieces of evidence or sequential
steps, but straightforward.
C4
Hard: Clear need to aggregate multiple tools/conditions; non-obvious composition.
c5
Very hard: Long-horizon integration across several tools/condLtions;; careful alignment needed.
Heuristics:
web-like single retrieval could at least two independent conditions must be satisfied wffeed/oee heddcmmiined
C 1 C2
C5
(pick based on complex ty).
A.12 HUMAN PERFORMANCE BASELINE STUDY
To address the reviewer request for a cuman-performance baseline, we conducted controlled user study measuring human success encefoaicite:Ojnnex--0 All participants were instructed to solve tasks without any Al tools and only using standard, non-AI information sources (e.g., search engines and domain websites/apps).
random subset of INFOMOSAIC-BENCH, :contentRefer-
A.12.1 TASKS AND SAMPLING
We sampled 10 problems from each domain (Map, Video, Medical/Bio, Finance, Web, Multi-domain), resulting 60 unique problems total. Each problem was solved by different participants, yielding 180 human attempts (30 attempts per domain).
1.1 PARTICIPANTS
We recruited 36 participants with engineering backgrounds: 10 undergraduate students and 26 graduate students (self-reported). No domain expertise beyond general STEM training was required.
1....3 PROTOCOL AND INSTRUCTIONS
Each participant completed 5 problems (total budget < 150 minutes). Each problem had a strict time limit of 30 minutes. Participants required to submit the following fields per problem: (1) Answer, (2) Time
29

---

Published as conference paper ICLR 2026
Human performance
GPT-5 with web-tool
Time/min
map
43.33%
33.33%
8.26
video
36.67%
28.33%
22.23
bio
30.00%
48.33%
20.50
web
13.30%
23.33%
25.33
multi 20.00%
finance
48.33%
22.63
6.00%
45.00%
25.80
overall
27.89%
37.67%
0.79
Table 19: Human baseline performance on 60 randomly sampled AFFMMSAIC--ENCCE problems (10 per domain), each attempted by 3 participants (180 attempts total) compared with performance on GPT-5 with web-tool.
spent, (3) Tools used, and optionally (4) Search keywords. If they could not solve problem, they were instructed to report "N/A" ("None").
Allowed tools. Participants could use standard non-Al tools, including: (1) web search engines (Google/Bing/aiiduu, (2) domain websites or apps (e.g., Google Maps/YouTTube/ikidaaaa and basic databases), (3) calculators and standard browsers without Al features.
Forbidden tools. Participants were explicitly prohibited from using any Al assistants or Al-augmented search/browsing plugins (e.g., ChacF77/frT//Gmmmiii/////rPlraap
A.12.4 SCORING
We scored human answers against the benchmark ground-truth answers. A submission was marked correct only if it matched the unique ground-truth target for the instance (e.g., exact entity/icckr/URL),, consistent with the benchmark's ' strict end-to-end success notion.
A.12.5 RESULTS
Table reports domain-wise accuracy and average time. Overall, humans solved 50/180 attempts correctly (27.8 1)). Map tasks were the easiest and fastest, while multi-domain and web tasks were the hardest and frequently reached the time limit.
A.12.6 DISCUSSION
This human baseline indicates that the benchmark remains challenging even for competent STEM partici- pants equipped with standard non-Al tools. Performance varies substantially by domain: tasks grounded in structured interfaces (e.g., maps) are comparatively more accessible, whereas open-web and multi-domain integration problems are substantially harder under 30-minute time budget. The near-ceiling average time Web and Multi-domain (25+ minutes) suggests that failures are often associated with long-horizon search and cross-soures integration difficulty, rather than immediate lookup gaps.
A.12.7 BENCIMARK EVALUATION PROMPT
Below are the exact evaluation prompts used to extract answers, verify correctness, and assess sub-answers for our benchmark.
30

---

Published as conference paper ICLR 2026
extract answer template
You helpful AI assistant tasked with extracting the final from provided solution.
**Input:**
1. problem statement, prefixed with "===Problem: <problem>",
2. solution to the problem, prefixed with "===Solution:",
**Problen em and Sclution:**
Probbeem (task)
Soluutioo::
(operated text)
** Instrructions::
Carefully analyze the solution and extract the final answer in reply: "The answer <answer extracted> in reply",
the solution does not contain a final answer (e.g., only reasoning, code without execution, or incomplete information), respond with: "The reply doesn't t contain an answer."
Ensure that the extracted answer is exactly presented in the solution. Do not infer or use external knowledge. Do not execute the code yourself. Remember, Never execute the code yourself! Never doing any computation yourself! Just extract and output the existing answer!
eval prompt template
You are a helpful AI assistant. You will use your coding and language skills to verify the answer.
You given:
1. A problem, which is going to start like " em: <problen
2. A ground truth answer, which is going to start like "===Ground truth answer:".
3. reply with the Reply:".
the problem, which going to start like "===
Please do the following:
1. Extract the answer in reply: "The answer is <answer extracted> in reply".
2. Check whether the answer in reply natches the ground truth answer. When comparison
obvious (for example, 3*1lsgrt (6) and 7.348), you may comрare by calculation, allowing a small margin of error.
3. After everything is done, please give each reply comment like the following options:
"The answer is "
"The answer is approximated but should be correct. Correct Answer: <ground
truth answer> Answer extracted: <answer extracted>,
"The answer is incorrect. Correct Answer: <ground truth answer> Answer extracted: <answer extracted>."
"The reply doesn't contain answer,"
Here are the problem, the ground truth answer and the reply:
===Problem: (task)
===Ground truth answer: (ground_truth)
31

---

Published as conference paper ICLR 2026
R==Ry (eperated_text)
testcase www.seesee.eopppp......eee
You helpful Al assistant. You will use your codi and language skilla to veri fy the subanswer.
You are given:
1. A set of subproblems and its corresponding subanswers, which are going to start like Suppoubpoobe and subanswers".
2. reply with the subanswer the subproblem, which going start like "===Reply:",
Please do the following:
1. Matching the subquestions and cempare the subanswer in reply with the ground
truth subanswer.
2. Check whether the subanswer in reply matches the ground truth subanswer. When comparison is not obvious (for example, 3<\aaqrt (6) and 7.348), you may compare by calculation, allowing a small margin of error. Here are some principles:
the ground truth subanswer is numerical value, you may compare the numerical value in the subanswer with the ground truth subanswer, allowing small margin of error.
If the ground truth subanswer is a string, you may compare the string in the subanswer with the ground truth subanswer, case-insensitive, and justify if it is the same as the ground truth subanswer.
3. After everything is done, please give each reply conment like the following format to justify if the subanswer is correct or incorrect: ''json
11
"subquestien": "CORRECT INCORRECT",
"subquestion": "CORRECT INCORRECT"
}
Here the problem, the ground truth and the reply: Suapoobblemms and subanswers:
(task)
==-Reply: (subanswer)
## Output format: You must follow the following format ''json
||
"subquesticn": "CORRECT INCORRECI",
"subquesticn": "CORRECT INCORRECI" }}
32


Reviews:
[{"review_id": "9gpYD3X8aw", "reviewer": "Reviewer_S6ix", "summary": "The authors introduce a benchmark to test multi-source information seeking by tool-augmented LLM agents. It comprises 621 query tasks spanning six domains and provides 77 specialized MCP tools for these domains. Each task requires combining general web search with one or more domain-specific tool calls. Tasks are automatically synthesized via a pipeline, which uses an “organizer-workers” LLM architecture to propose coherent scenarios grounded in verified tool outputs. Evaluation results on 14 LLMs  results show that web search alone yields very low accuracy. Adding domain tools only yields selective gains (improving results in map/video but degrading others) and about 22.4% of failures were traced to incorrect tool selection or usage.", "questions": "- The current evaluation is binary at the task level. Would it be feasible to design a metric that captures partial correctness?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- InfoMosaic-Bench explicitly requires agents to integrate evidence from both web search and diverse tools. No existing benchmark systematically tests this combination. \n- Evaluations cover 77 MCP tools across 6 different domains.\n- Tasks are explicitly designed so that no single tool or search can solve them in one step.", "weaknesses": "- Because all tasks are synthetically generated, the dataset may not accurately reflect the real distribution of human information-seeking queries.\n- Some conclusions may over-generalize. The reported improvement from tool use in map and video tasks, and degradation in medical and financial ones, likely reflects differences in the specific tools provided and the complexity of their parameterization rather than inherent domain effects.\n- The dataset has uneven domain sizes (e.g., 135 map vs. 90 biomedical tasks) and mixes Chinese and English, with varied text lengths and complexity. While this adds realism, it complicates cross-domain comparisons and may introduce language-related bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sWC4gwEIJc", "reviewer": "Reviewer_hZCF", "summary": "The paper presents InfoMosaic-Bench, a benchmark of 621 problems across six domains designed to test whether agents can combine general web search with domain-specific tools exposed via MCP. Tasks are synthesized with a novel pipeline called InfoMosaic-Flow, an organizer–workers pipeline that grounds conditions in verified tool outputs and iteratively prunes web-search shortcuts. Evaluating 14 LLM agents, the authors find: (i) web-only agents underperform, (ii) domain tools help selectively (not uniformly), and (iii) 22.4% of failures stem from tool selection/usage errors.", "questions": "1. How strict is the multi-source requirement post-refinement? Could you quantify the fraction of items where each condition is necessary and show a “single-source solvability” audit on a held-out judge? \n\n2. Human study details. Could you share rating guidelines and per-domain κ? Also, does κ remain high on borderline cases that web-search could partially solve? \n\n3. Latency & cost reporting. Any measurements of wall-clock, tokens, or tool latency per domain—and how those change under domain-tool vs. web-only settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality. First benchmark that systematically evaluates multi-source information seeking with real MCP tools, bridging a gap between web-only search benchmarks and API-centric invocation tests. Organizer–workers InfoMosaic-Flow is a thoughtful synthesis design that enforces cross-source dependencies. \n\n2. Quality. Clear two-stage generation with web-evolving verification and automated + manual quality control; ablations show verification is necessary to remove trivial cases. Well-scoped metrics (Accuracy, Pass Rate) and broad model coverage; analyses include scaling with tool calls and error taxonomies. Human study also supports data quality and reliability). \n\n3. Significance. Results quantify a core limitation of current agents: integrating domain tools with web search; even top closed models underperform on multi-source tasks, signaling an impactful research direction.", "weaknesses": "1. Per-domain scale is modest. With 621 items across 6 domains (e.g., 83 medical, 100 finance/video), per-domain statistics can be noisy. Ideally should expand to a larger dev/test split per domain. \n\n2. Judge dependence. Accuracy uses an LLM judge, which can introduce bias. Lack inter-judge agreement, including rule-based scorers where feasible (e.g., exact match for structured answers), and publish judge prompts. \n\n3. Tool-budget sensitivity. Tool call limit fixed at K=20; conclusions may change with higher budgets or adaptive stopping. Lack sensitivity curves for K and latency/throughput trade-offs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hiAHh5ddLt", "reviewer": "Reviewer_w89X", "summary": "- This paper introduces InfoMosaicBench – an evaluation benchmark to assess the abilities of LLM agents to solve information-seeking tasks by gathering evidence from multiple domains/sources. The paper proposes InfoMosaic-Flow pipeline to synthetically curate their data. \n- Furthermore, the paper benchmarks 14 LLMs in an agentic scaffold with only web-search and report poor task performance, highlighting the importance of domain-specific tools for multi-source information seeking queries. Even when given domain-specific tools, the performance improvements are inconsistent and achieved only for 3/6 domains.", "questions": "Please address each of the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The analyses are very insightful and provide more nuanced insights into various failure modes of agentic systems.\n- The experiments are very comprehensive and the hypotheses are thoroughly tested to empirically validate their claims.", "weaknesses": "- Limited real-world grounding: Are the tasks in this benchmark actually representative of real-world multi-source information-seeking tasks? The examples given in Appendix A.9.1 seem extremely complex (and somewhat obvious that they were synthetically crafted), and I am not very convinced that any human user would actually craft such complicated queries for this task.\n- Understanding the choice of tools: How do authors decide which tools to include/not include for each domain? The provided list in the appendix shows extremely fine-grained tools which are only likely to be available in extremely specialized agents designed for a particular task/domain. Do these tools have any bias on the nature of tasks you synthetically curate from your InfoMosaicFlow pipeline?\n- Consistency of claims: The results in Table 3 do not support the overall claim about agents with web search being incapable of solving these tasks. If that was indeed true, domain-specific tools would yield consistent improvements across all 6 domains which is not the case. Thus, is the poor task performance arising from the absence of domain-specific tools or is it just because of the overall difficulty and complexity of the task?\n- Lack of insights for future work: While the paper provides an analysis of failure modes, there is limited novel insight into further agent design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HrLUcFlxfK", "reviewer": "Reviewer_EoM5", "summary": "This paper introduces InfoMosaic-Bench, a new benchmark for evaluating the ability of AI agents to perform multi-source information seeking. To construct the benchmark, the paper proposes InfoMosaic-Flow, a two-stage (generation and refinement) agentic pipeline that synthesizes 621 tasks across 6 domains (bio, finance, web, maps, video, and multi-domain). The authors conducted extensive experiments on 14 models using a custom agent scaffold.", "questions": "1. What portion of the dataset is in Chinese? This information is not mentioned in the main paper, only alluded to in the appendix.\n\n2. Did you conduct an ablation study where the agent has no access to any tools (including web search)? It seems some of the questions may not require any tool use to answer.\n\n3. During the web search experiments, did you encounter any issues with rate limiting from the search engines?\n\n4. The choice of domains seems somewhat arbitrary. What was the rationale for selecting domains as disparate as biology and finance, alongside more general ones like web and video?\n\n5. The results for the web domain are surprising: Claude-4-Sonnet achieved only 3% accuracy, while GLM-4.5 reached 11%. Have you analyzed the reasons for this performance discrepancy?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper presents an innovative attempt to generate multi-source information retrieval tasks using a two-stage agentic pipeline.\n\n2. The work reflects a substantial effort, encompassing 621 tasks, 6 domains, 77 tools, and a comprehensive evaluation of 14 models.\n\n3. The authors have open-sourced the code, promoting reproducibility.", "weaknesses": "1. The evaluation is limited to a single, custom agent scaffold. The choice of agent architecture could significantly influence performance, and this variable is not explored.\n\n2. I have concerns about the dataset's quality. While the paper mentions quality assurance through human annotation (on a sampled subset), it lacks a human performance baseline. I am particularly skeptical about answer uniqueness and whether questions might have multiple valid answers.\n\n3. The paper appears rushed, which affects its clarity. The writing is often difficult to follow, and the core methodology—the data synthesis pipeline—is not explained in a way that is easy to comprehend. There are also several writing issues:\n\t- L255: \"serious\" should be \"series\".\n\t- L341 (minor): \"Open-sourced Model\" is better described as \"Open-weight Model\".\n\t- L675: Figure 5 shows t-SNE visualization of query in different domains, but finance domain is missing.\n\t- L810: Table 8's example is clearly not from Bio domain as it states.\n\t- L829: Table 9's example is in Chinese with no English translation, while other examples are translated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)