A Benchmark for Deep Information Synthesis
Authors: Debjit Paul
Daniel Murphy
Milan Gritta
Ronald Cardenas
Victor Prokhorov
Lena Sophia Bolliger
Aysim Toker
Roy Miles
Andreea-Maria Oncescu
Jasivan Alex Sivakumar
Philipp Borchert
Ismail Elezi
Meiru Zhang
Ka Yiu Lee
Guchun Zhang
Jun Wang
Gerasimos Lampouras
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Deep Information Synthesis
LLM agents
Deep Research
AI agents
Primary area: datasets and benchmarks

Abstract: Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks do not adequately assess their ability to solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTH, a novel benchmark designed to evaluate agents on realistic, time-consuming problems that combine information gathering, synthesis, and structured reasoning to produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPSYNTH is constructed using a multi-stage data collection pipeline that requires annotators to collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum F1 score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as a crucial benchmark for guiding future research.

Published as a conference paper at ICLR 2026
A BENCHMARK FOR DEEP INFORMATION SYNTHESIS
Debjit Paul', Daniel Murphy?, Milan Gritta', Ronald Cardenas', Victor Prokhorov', Jun Wang', Gerasimos Lampouras'
Dataset Contributors:
Lena Sophia Bolliger', Aysim Toker', Roy Miles', Andreca-Maria Oncescu', Jasivan Alex Sivakumar", Philipp Borchert' Ismail Elezi', Meiru Zhang", Ka Yiu Lee', Guchun Zhang'
Huawei Noah's Ark Lab, UK University Zurich UUniversity of Sheffield UUniversity of Cambridge
"Imperial College London
UCL Centre for Artificial Intelligence
ABSTRACT
Large language model (LLM)-based agents are increasingly used solve complex tasks involving tool use, such as web browsing, code execution, and data analysis. However, current evaluation benchmarks not adequately assess their ability solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval. To address this, we introduce DEEPSYNTII, novel benchmark designed to evaluate agents realistic, time- consuming problems that combine information gathering, synthesis, and structured reasoning produce insights. DEEPSYNTH contains 120 tasks collected across 7 domains and data sources covering 67 countries. DEEPS YNTH is constructed using multi- data collection pipeline that requires annotators collect official data sources, create hypotheses, perform manual analysis, and design tasks with verifiable answers. When evaluated on DEEPSYNTH, 11 state-of-the-art LLMs and deep research agents achieve a maximum Fl score of 8.97 and 17.5 on the LLM-judge metric, underscoring the difficulty of the benchmark. Our analysis reveals that current agents struggle with hallucinations and reasoning over large information spaces, highlighting DEEPSYNTH as crucial benchmark for guiding future research.
INTRODUCTION
Information synthesis involves collecting information from multiple sources and reasoning over it to form coherent insights. While this capability has been central to human decision-maken maning and has driven advances in fields ranging from scientific discovery to policy development (Tricco et al. 2 Sambre & Brône 22022), has traditionally been laborious and cognitively demanding. For example, travel agency from Singapore might want know "Which non-ASEAN countries experienced significant Post-COVID recovery reaching at least 95% of their 2019 visitor arrival levels to Singapore by 2023, and what were the main reasons for travel (business or tourism)?" ?" question that requires identifying ASEAN countries, extracting multiple arrival data from various sources, and analysing them determine the answer (see Figure MD. Recent Large Language Model (LLM)-based agents with capabilities for reasoning, tool use, and interaction across diverse environments have shown promise in complex tasks, such as for hard-to-fine information (Wei et al. (025). interacting with websites (Lu et al., 2025), and planning to navigate the web (Abuelsaad et al. 2024). However,
these developments mostly improve the information -eoomationn-eekking capabilities of agents. remains crucial to evaluate whether such agents solve real-world tasks that require synthesizing information from multiple sources and inferring insights beyond simple fact retrieval.
Despite the substantial promise of LLM-based agents for addressing real-world tasks, most existing benchmarks primarily emphasise shallow fact retrieval tasks (Wei et al. 2024), artificial information- seeking questions (Wei et al. 2025 Mialon et al., 2023), tasks that require information from single, particularly wnol- source like Wikipedia (Wolfson et al., 2025). Furthermore, most agentic benchmarks focus English-language sources (Wei et al. 2025 Mialon al., 2023) and

---

Published as a conference paper at ICLR 2026
Which non-ASEAN countries experienced significant post-COvID recovery reaching least 95% of their 2019 visitor arrival levels to Singapore by 2023, and what were the main reasons for travel (business tourismm?
Answer
Searching "Arrival visitors to Singapore*
"counkry1": "reason1", "country2": "reason2",
Browsing "https:l/..."
12
format output
Browsing "https:!l..."
9.0
process
Browsing "https://dat..aseanii wwt.aaaatssaatttttttoiiititooooo
"origin: any non-ASEAN country"
<
"destination: Singapore*
year: 2019*
ATM
yeaa: 2023
Figure I: A sample task from DEEPSYNTH, illustrating that synthesizing knowledge requires agents to perform multiple steps, including web browsing, gathering information from multiple sources, reasoning over them, and generating the final answer.
overlook the diversity of regional contexts, languages, and information ecosystems, limiting their ability to evaluate agent performance in realistic, globally distributed settings.
To resolve this gap, we introduce DEEPS benchmark, a new benchmark comprising 120 challenging and diverse tasks, aimed to evaluate the ability of agents to browse the entire web, combine information from unstructured and structured sources (paragraphs and tables) across 67 countries, and perform analysis to synthesize new information and insights. DEEPSYNTH tasks annotated with a gold standard, manually annotated reasoning chain, which includes all the intermediate steps, answers and all required supporting evidence. Each task requires agents to navigate an average of 4.2 web pages, and read between 1 to 15 documents and/or tables. These tasks are designed to reflect real-world analysis and insight generation, with an emphasis on the time-intensive nature of processing and integrating information (see Figure IT.
To construct DEEPSYNTH. we asked 16 experts to first curate relevant topics and data sources targeting various countries ($2.2). and subsequently formulate possible hypotheses for each of these data sources. Based on the hypotheses, the experts conducted analyses and derived insights. Finally, drawing on their analyses, they formulated the corresponding questions, answers, and step-by-step reasoning chains. In our experiments, we find that state-of-the-art LLMs struggle significantly on DEEPSYNTII, including recent AI reasoning models such as GPT-5.2-Pro and DeepSeek-RR RR OOeenA 2025b Guo et al. 2025). The best berf-performing model, GPT-5.2-Pro, achieves only an F1 score 8. 8.70, and 6.25 on the stricter EM metric. We also analyse the performance of specialised deep research agents, i.e. dedeperesearchhPpaeeeee OWI (Hu et al. 2025). and observe they successfully solve only three out of 120 tasks, further underscoring the difficulty of our benchmark. Our analysis reveals that (i) these agents frequently commit navigation and synthesis errors, and (ii) their performance drops sharply when tasks require synthesising information from under-represented sources, e.g. data pertaining to the African region.
2025a), smolagents(Roucher et al. 2025), ) and
To summarize, our main contributions are:
1. We release DEEPSYNTII, new benchmark for agents that contains 120 real-world and time-consuming information synthesis tasks.
'Our data and code for DeepSythn Bench are publicly available
2

---

Published as a conference paper at ICLR 2026
2. We show that DEEPSYNTH poses significant challenge for state-of-the-aar agents, revealing key limitations in their capabilities. The best best-performing agent achieves only an F1 score of 8.97 points, leaving substantial room for improvement.
3. We conduct in-depth analysis explain how DEEPSYNTH YNTH is challenging and demonstrate why current agents cannot yet be considered reliable systems for information synthesis.
2 THE DEEPSYNTH BENCHMARK
DEEPSYNTH is benchmark designed to evaluate agents realistic, time-consuming tasks that require planning, information gathering, and synthesis from the web. Specifically, DEEPSYN YNTII evaluates agents on their ability navigate multiple websites, extract information from both structured and unstructured sources, and reason effectively to produce correct solutions. It consists of 120 tasks that are carefully designed and annotated by experts. Each task (see Figure is formulated to yield concise output in the form of JSON object dictionary, with key-value pairs organised in a tabular style, thereby enabling straightforward and reliable verification. Solving these tasks requires agents to formulate plans, decompose problems into sub-steps, select and use appropriate external tools (e.g.. document processors, code interpreters), and integrate intermediate results into final solution.
We now describe the process of constructing DEEPSYNTH. We first outline the criteria for our tasks, then describe our data collection pipeline, and conclude with an analysis of the collected data.
2.1 CRITERIA FOR DEEPSYNTH TASKS
Motivated by prior benchmarks (Mialon et al. 2023 Yoran et al. 2024; Wei et al. 2025 Phan al., 2025), ; the design of DEEPSYNTH tasks is driven primarily by criteria that promote the future development of Agents' information seeking and synthesis capabilities towards practical and grounded goals. Specifically, our criteria consist of:
a) Multi-source Information Synthesis: Tasks should require agents to identify connections across multiple data sources to combine information from them to produce coherent solution. More specifically, tasks are designed such that agents must not only fetch relevant information but also perform subsequent operations on it (see Table IT.
b) Inspired by the Real World:
instructed to draw inspiration from real-world situations. The tasks are designed so that any resulting insights would conceivably be able shape the decisions and actions of an individual or a group people, such as political scientists, policy makers, travel agents, etc.
Experts
c) Verifiable Answers: A task that has closed-form answer, which can be automatically verified and is stable over time, making it suitable for reproducible evaluation. While the answers our tasks may be better suited to open-form answers, properly argued and grounded in citations, we necessarily restrict them to maintain their verifiability.
d) Diversity: Our benchmark is designed to span a wide range of tasks, requiring agents to gather and reason over information across 67 countries and distinct domains. Beyond geographic and topical diversity, the tasks also encompass temporal analyses, comparative evaluations across groups categories, and relational reasoning, ensuring that agents tested on a variety of reasoning modes.
c) Robustness Against Memorisation: Similar to Mialon et al. (0033) we ensured that the tasks are explicitly constructed mitigate data contamination and prevent superficial memorisation. The gold-standard answers are intentionally built to be non-retrievable through verbatim lookup known pre-training corpora or direct web search, compelling systems to plan and perform multi-step reasoning to derive the correct output.
2.2 DATA COLLECTION
A common practice in designing deep agentic benchmarks is to start with fact and then craft question from it, making the answer difficult to locate (Wei et al. 2025). Since our goal was to ensure answers non-retrievable, adopted different approach. Building DEEPSYNTH involved four
3

---

Published as a conference paper at ICLR 2026
Data Source Identification
c. Hypothesis Valldation
Curate
Analysis
Search
Demographic
Knowledge (optional)
Topics + Data sources
Topics Hypotheses
Knownledge Symthnaal
Data sources
Source: London underground tap-in tsp-out deta for 2024
Validation: Detect anomailes in the data and verify their through searches official records.
TTl.go.uk
Hypotheses Gathering
d. Task Formulation
Refinement
Exploration
Annotation
Toplon
Heporhooos
Data Search
Topica
Hypotheses
Task: Question Agentic Staps Antweer
Hypothesis Certain stations may experience significant fuctuations ridership due special events (e.g. concerts, sports matches).
Task: Identify the Landon Undergraund station that exhibited the greatest variance in daily tap-out counts plausible
2024 determine factor accounting for this variability.
Figure 2: An overview of our data collection process for building the DREPSYNTH benchmark
key steps: (a) identifying data sources, (b) gathering hypotheses, (c) performing analyses, and (d) formulating tasks (see Figurel 2
Data Source Identification. In this step (see Figure 2 left), engaged 16 human expert to propose a diverse set of data sources and topics, drawing on their expertise, demographic backgrounds, and interests. Given the complexity of the annotation process and the need for efficient coordination, participation was restricted to individuals with whom we maintained direct communication, along with the paper's s core authors. We collected 223 data sources across domains We excluded data
finance, environment, science, education, transportation, political/sociopolliiin sources that originated from untrustworthy or non-official websites, including those requiring user authentication, well sources containing information that contradicted other verified references. Our objective was to curate tasks that are useful to individuals or groups; therefore, we filtered data sources to retain only those from which useful, verifiable insights could be drawn. For example, included official statistical reports on "the gender gap in labour force participation rates in Australia", "computer and digital literacy rates in Sri Lanka"" and "air quality and pneumonia-relatt deaths across regions in the UK", since such data enables clear downstream reasoning tasks (e.g., analyzing temporal trends, comparing across regions, or correlating with policy interventions).
Hypothesis Generation. We then asked annotators to formulate or two plaussbbe insights that could be inferred from the selected data sources (see Figure Dbottom left). The objective of this step to elicit hypotheses that are both insightful and practically valuable, encouraging reasoning beyond surface-level fact retrieval (see $(.I(e)) For example, one such hypothesis was: "Is there linear relationship between air quality and pneumonia-related deaths across regions the UK?". Data sources that did not meet the criteria of usefulness and insighifulness (see (b. were subsequently filtered out After this step, retained a total of 155 data sources, each paired with its corresponding set of hypotheses.
Hypothesis Validation. this step, annotators were tasked with conducting a detailed analysis of each data source to assess the validity of the proposed hypotheses (see Figure 2 top right). The objective was twofold: (i) to verify whether the data supported the hypotheses, and (iI) derive tasks with verifiable answers (see 2.1 (c). Hypotheses that failed to meet the verifiability criterion were
Details about annotators are provided in Appendix A.2
Please note this step involves degree of subjectivity. and relied the domain knowledge and judgment of annotators ensure the quality of the retained data sources and hypotheses.
4

---

Published as a conference paper at ICLR 2026
Statistic
Value
Total Tasks
120
Avg.
Question tokens
78.49
Avg.
Intermediate steps
7:54
browsing
Web pages per Task
4.2
Avg.
Annotation Time
5.5 hours
Web Search
Synthesis Operations
%
Trend Detection
20.93%
Diverse fietypa reading
Average
11.05%
Correlation
6.98%
Ranking
19.77%
Anomaly Detection
6.98%
Muks Modallty
Counting and Comparing
33.72%
Filtering
0.58%
Percentage Questions
Table 1: DREPSYNTH statistics across tasks.
Figure Percentage of tasks per capabilities re- quired to solve DEEPSYNTH.
refined or discarded. Following this validation and filtering process, we retained 130 data sources, each paired with its corresponding, verified hypothesis.
Task Formulation. Finally, annotators were asked to formulate task questions along with inter- mediate steps, supporting evidence and corresponding answers. We note that the intermediate steps indicate only one possible reasoning path or planning from question answer, and that no model or agent necessarily needs to imitate that path. Since DEEPS YNTH tasks often rely on multiple pieces of supporting evidence and reference various documents or tables, annotators were instructed to provide the URLs where the data sources can be accessed. Additionally, they were asked to include brief explanation of how the task can be solved, specifying any tools, code snippets, or mathematical formulas used in the solution. We provide more examples and additional statistics in $A.3
Data Validation. All questions went through a second annotation stage, where another annotator independently answered the question. Only tasks where the answers from both annotators were identical were retained in the dataset, leaving finally 120 challenging information synthesis tasks.
2.3
DATA STATISTICS
Table I summarises the key statistics of our benchmark. Tasks in DEEPSYNTH are highly detailed, with an average length of 78.49 tokens, an average of 7.54 intermediate reasoning steps and requiring navigation through an average of 4.2 web pages to reach solution. Additionally, on average, formulating each task (from data source identification to task formulation) took the annotators approximately 5.5 hours. This number highlights the challenge in creating such tasks. Overall, all these numbers underscore the inherent complexity and challenge of the benchmark. Moreover, the tasks encompass diverse range of analytical and reasoning skills, including correlation analysis, anomaly detection, and identification of causal or linear relationships - as reflected in Table Table presents the regions covered by our benchmark, along with the percentage of tasks corresponding to each region. Notably, the benchmark comprises a higher proportion of tasks from Europe and Asia, with some tasks spanning multiple countries and regions. Figure B provides an overview the capabilities required by agents to solve the benchmark and their prevalence in tasks. In particular, we observe that web search and browsing are the most critical skills for retrieving the correct information.
EVALUATION SETUP
Models. We use DEEPSYNTH to benchmark five state foate--f-thaee the models: (a) o4-mini (OpenAl 2025c), (b) GPT-4.1 OOpenA 2024). (c) GPT-5 (OpenAl 2025b), (d) Semini--25--Pra (Comanici et al. (202)) and (e) DeepSeek-RR (Guo et al. 2025). For Gemmn-2,.-PPrr we use "dynamic thinking". where the model decides how much think. GPT-5 was evaluated using "high reasoning effort".
5

---

Published as a conference paper at ICLR 2026
Model
Fl Score
Precision
Recall
Exact Match
LLM Judge Score
LLM Baselines
o4-mini (2025-08)
3.05
2.33
4.39
0.0
0.0
GPT-4.1 (2025-08)
3.46
2.86
4.39
0.0
0,0
o3 (2025-08)
3.29
2.85
3.90
0.0
GPT-5.1 (2025-08)
3.83
2.98
5.37
0.0
0.0
Gemini-Pro-2.55 (2025-08)
6.25
4.71
9.27
0.0
5.0
GPT-5.2-PPr (2026-02)
8.70
8.45
8.96
6.25
6.67
DeepSeek-R1- -a-Seeck-RI-hattt (2025-08)
3.23
2.75
3.90
1.67
2.5
DeepSeck-RR R1 Reeeosone (2026-02)
2.80
2.73
2,87
2.50
6.67
Prameworkkbassed Agents
o3 03-deep- reeepreseaachhee (2025-08)
8.97
7.73
10.69
2.50
17.5
Smolagent (GPT-4.1)
3.75
3.27
4.39
2.50
7.5
Smolagent (GPT-5)
6.42
6.34
6.50
1.67
2.5
OWL (GPT-4.1)
4.62
6.52
1.67
12.5
Table Performance comparison on the DEEPSYNTI benchmark (Pass@ 1). F1, Precision, Recall and Exact Match measure the quality of model predictions. LLM Judge (%) reports the average precision. Models with are models or framework which are closed, while are open-source.
We also investigate the performance three state-of-the-art (deep research) agentic frameworks: (a) 03-deep-research (OpenAl 2025a); (b) smolagents (Roucher al. 2025), which is minimalist framework focused on simplicity and rapid prototyping. It uses a standard ReAct loop (Yao et al. 2023) and its primary distinguishing feature is that it expresses all actions, such tool use, code, which is parsed out of the response and executed Wang et al. (2024); (c) OWL (Hu et al.) (025). which employs role-playing strategy where planner and executor collaboratively solve tasks, optionally augmented with smaller, more specialised 'workers'. Both OWL and smolagents have been open-sourced. The details of their tool capabilities are listed in Table 10 All models were prompted using the same instructions, provided in Appendix A.5
Metrics. All tasks require models to generate outputs in a JSON format (or lists JSON objects). Our strictest metric is exact match, meaning that all keys and values must be correct. For partial evaluation, we check how many key-value pairs are correct (out of the total pairs) and report precision, recall and F1-score. As additional 'soft' metric, we follow Wolfson al. (2025) and leverage the LLM-as-a-judge (with an identical prompt, see Fig. [Op reporting average precision. This serves two purposes: 1) for small string differences (with semantic equivalence), this method will reward answers beyond exact match, 2) in case of numerical answers, small margin (1% to 5.5% difference) can still be considered correct, hence providing more granular and permissible scores.
RESULTS
4.1
MAIN RESULTS
Table 2 shows the performance of SOTA models DEEPSYNTH. We first evaluate the parametric knowledge and reasoning capabilities of LLMs. The results show that GPT-5,2-PPr achieves the highest score of 8.70, and both GPT-5.2-Pro and DeepScek-RI-Reacce Rea-sooneer achieve the highest LLM Judge score of 6.67, indicating substantial room for improvement. Interestingly, the performance gap between reasoning models (e.g. Gemini-2.5-. 2.5-Poo. GPT-5.1, DeepSeek-R1) and general-purpose LLMMs on F1 score. This finding suggests that the key bottleneck lies not
GPT-4.1) is relatively small in reasoning ability alone, but the availability of the necessary information for reasoning. We investigate this observation in greater depth in Analysis; see $5 Further, under the strict exact-match metric, we observe that almost all models obtain a score of zero, indicating that none can solve single task perfectly. The poor performance of base LLMs indicates that internal retrieval of parametric knowledge is insufficient, showing that these tasks are robust against memorisation (see criteria 2.T) This also highlights the need augment these models with external tools.

---

Published as a conference paper at ICLR 2026
04 14-2214
69P-411
P1 beace Mireoo
2agg GFFLL sAgaoo)
Smolejanis (GPS4.1)
GPT3.1
Sampling (Bests))
Mijoriy
Gemiri
25,0
22.5
ChepSere-Chat
gro
154
23
17.5
Dopp Rassunte
judge
12.5
al-dwp--oontt movech
Smooageett 12PT4
7.5
7.5
ONL 12P44
13
Scoce
Bestgg3
Beetg
Beetgg
(a) Pass@1 performance on DEEPSYNTH-Dev.
(h) Best-of-N performance on DEEPSYNTH-Dew,
Figure 4: Performance comparison on the DEEPS YNTH-Dev benchmark. (a) F1 Score measures the quality of model predictions. LLM-Judge (GPT-4.1) reports the average precision as judged by 1. Light bars denote LLM baselines; dark bars denote tramework-based agents. (b) Best-of-N LLM Judge Scores comparing GPT-4.1 standalone vs. Smolagents (GPT-4.1) across N (1,3,5) and and Self-Consistency @5 (majority voting).
To investigate this further, we evaluated our benchmark using three agentic frameworks that integrate external tools, including simulated web browsing, web search, and code interpreter. We find that o3 -eeppreseerch, which incorporates web search and an executable code interpreter, outperforms the base o3 model by 5.68 F1 score and 2.50 EM. Furthermore, smolagents and OWL achieve some gains, with improvements of 0.29 and 1.95 Fl points and 2.5 and 1.67 EM points respectively, over the base G.T-4.1. Overall, we observe that all systems perform poorly. These findings emphasise that effectively solving tasks in DEEPSYNTH requires enhanced tool-use capabilities. Interestingly, find that low precision indicates that all models frequently produce incorrect or extraneous answers.
presents results the DEEPSYNTH-Dev subset. Among standalone LLMss, GPT-5.2-Pro achieves the highest F1 score (15.6), while Gemini-Pro-3 leads on the LLM-Judge metric (15.0), suggesting produces semantically reasonable outputs that strict matching penalizes. Among agents, o3-deep-research attains the highest LLM-Judge score (20.0), reinforcing that tool augmentation benefits synthesis-heavy tasks. We observe consistent gap between L.MM- Judge and Fl scores across all models. Our manual evaluation suggests that this discrepancy primarily arises from failures to produce numerically precise or structurally exact outputs.
DEEPSYNTH-Des Results. Figure 4a
Best-of-N and Ceff-Consistency Analysis. Figure 4b examines whether multiple attempts improve task completion on DEEPSYNTH-Dee Under Best@5 55 Smolagents (GPT-4.1) reaches 25.0% LLM- Judge accuracy compared to 17.5% for GPT-4.1, suggesting that tool-use introduces beneficial variance across runs. However, self-consistency (majority voting at N=5) yields only 5% accuracy for both systems, with low average consistency scores (0.27), indicating that correct answers rarely emerge as the majority prediction. The stark contrast between @e and self-consistency (27.5 27.5% vs. 5% for Smolagents) demonstrates that current agents exhibit high output variance on DEEPSYNTH tasks. Occasional runs succeed, but models lack the reliability needed for consistent, correct answers.
Ablation Study. To assess the role different capabilities on DEEPS YNTHI, we perform an ablation study. As shown in Table (top), performance shows consistent declines across all metrics when any capability removed, with the largest drop (1.81 Fl points) observed when search excluded. While the overall changes are modest due the low baseline performance, these trends indicate that document processing, code execution, and search each contribute to task success, highlighting the multifaceted challenges posed by DEEPSYNTH
7

---

Published as a conference paper at ICLR 2026
Performance Metries
Model
F1
Precision
Recall
EM
OWL (Full)
5.41
6.52
1.67
Tool Ablation
Web Browsing Toolkit
4.80
4.20
5.60
1.67
Search Toolkit
3.60
4.61
Document Processing Toolkit
4.90
4.50
5.4
1.67
Code Execution Toolkit
4.82
4.30
5.50
0.0
Reasoning Chain Ablation
GPT-4.1
3.46
2.86
4.39
0.0
Intermediate Step
9.36
8.76
10.05
5.0
Smolagent (GPT-4.1)
3.75
4.39
2.50
Intermediaae Step
10.50
8.96
12.70
10.0
Table 3: Ablation Study. Tool Ablation: Comparing the benefits of using different tools DEEP-
SYNTH. Reasoning Chain Ablation: Studying the role of planning given the intermediate steps.
Models
24 00
25.00
Madeis
C371-8.1
.12
21 228
Smellgeet (GFT-L))
21
Gemini-Pru
Caminl-PPr--2.3
63
18.92
23-deep-renearchh
74.14
Score
Score
03
1.20
12.43
10
2.209
2.53
10
10.20
5.37
4,21
4.13
1.422
2111
2.27
2,00
2.00
Detectur
Countiig
betertion
Intermediate Soeps
13:13
18:103
Operation Types
Figure 5: F1-scores across intermediate steps.
Figure 6: F1-scores synthesis operations.
5 ANALYSIS
In order to understand the challenges of solving DEEPSYNTII questions, analyse performance across different data collection strategies, followed by qualitative analysis of model errors.
RQ::
How do models perform the number of intermediate steps increases? We break down the models' performance based on the number of intermediate steps entailed by DEEPSYNTH tasks. Figure presents the performance breakdown, highlighting that all models struggle as the number of intermediate steps increases, which can be considered an indicator of the task's complexity. Notably, the agentic frameworks (o3-deep research and smolagents + GPT-5) perform better for 11-15 intermediate steps, while they are on par with other LLMs for smaller numbers intermediate answers. Given that tasks in DEEPSYNTH require average of 7.54 intermediate steps, these results provide insights into why the benchmark is so challenging.
Does providing agents with intermediate steps improve their performance? We evaluate how agents perform when they are provided with the ground truth intermediate reasoning steps (i.e. planning) without revealing the intermediate answers. As shown in Table 3 model performance improves substantially under this setting, with GPT-4.1 and smolagents + GPT-4.1 showing large gains. Both EM and Fl scores increase, indicating that models appear to lack planning capabilities.
RQ2:
Which synthesis operations more challenging? To further assess the models' analytical capabilities, we examine their performance on different synthesis operations when intermediate steps provided alongside the task questions (see TableT III. p the results across various operation types, revealing substantial variation in task-specific performance. More specifically, we observe that o3 model achieves the highest F1 score in anomaly detection (26.51 %), significantly
RQ2:
8

---

Published as a conference paper at ICLR 2026
Region
Tasks
GPT-4.1
03-deep-research
Gemini-2.5-Pro
smolagents
Africa
8.3%
0.0
0.0
0.0
0.0
North America
11.7%
4.65
8.00
12.00
8.33
South America
5%
0.0
25.00
0.0
0.00
Asia
29.2%
3,36
12.70
6,50
11.88
Oceanic 38.3%
Europe
3,45
10.83
4.91
5.28
10.8%
8.96
14.43
6.67
24.00
Others 12.5%
3.12
6.45
12.12
0.0
Table 5: Multi-Regionas Analysis: Agent performance across region-specific tasks (F1 score).
NOTE: A question may span multiple regions. "Others" contains tasks without regional association.
outperforming the other agents, while Gemini-2.5-Pro and smolagents + GPT-4.1 exhibit moderate gains over GPT-4.1 across most task categories. Trend detection and ranking also demonstrate relatively strong performance for Gemini-2.5-PP5 and o3, indicating that these models effectively capture certain structured patterns. contrast, none of the models exhibit measurable performance on filtering tasks, which may partly reflect the limited number of filtering tasks in the benchmark (see Table I. Overall, these findings suggest that, although some agents can successfully identify anomalous structured patterns, significant improvements required for tasks involving arithmetic, comparative reasoning, or complex multi-step analysis.
Error Cause
No. of instances
Navigation Error
15
No answer
Technical Issue
Synthesis Error
16
Table 4: Error analysis for OWL (GPT-4.1). Navigation and synthesis errors are the most prominent.
What types of errors do models commonly make? To better understand the challenges in solving DEEPSYNTH, we manually analysed a random subset of 32 tasks in which OWI. + GPT-4.1 made errors We focus on OWL because, as an open-source framework. enables detailed examination of execution traces and interactions between agents and tools. We categorize errors into four types, with their frequencies summarized in Table (1) Navigation errors when the agent fails to locate or access the correct source of information, such navigating to the wrong web page, document, or section; (2) No Answer when the agent does not respond or fails to generate any output; (3) Technical Issue errors caused by system limitations, software bugs, or tool malfunctions that prevent task completion, independent of reasoning or navigation; and (4) Synthesis Error - when the agent reaches incorrect conclusion despite accessing the correct information, due to flaws in logical reasoning, interpretation, or multi-step analytical processes.
RQ::
This analysis is multi-label, single instance may exhibit multiple error types. The majority of errors arrors-15/32 due to navigation and 16/32 due to caatoninn-highhlee that DEEPSYNTH presents significant challenges
for state-of-the-an open-source models. Figure illustrates failure of OWL, in which the correct URL was found, but the agent fails to interact correctly with the website and its database interface.
RQ₃: How do agents perform on tasks from different regions? We observe that 03-deep research exhibits the most consistent cross-regional capability, particularly in the high-volume areas such as Europe and Asia. Notably, all models fail on Africa-related tasks, achieving an Fl score of 0.0. These findings highlight the presence of strong geographical biases current models and suggest that their performance is not globally uniform, likely reflecting imbalances in the distribution and coverage
*Subset chosen due to the time and cost of manually analysing all outputs. Two annotators who involved in the original data annotation conducted this analysis.

---

Published as a conference paper at ICLR 2026
Real World
Multi
Information Synthesis
Multi-Part
Dataset
Regional
Answers
GAIA MMillon et al. 2023)
partial
x
partial
x
AssistantBench Yoran et al.
2024
x
partial
BrowseComp (Wei et al. 2025
X
X
x
x
HIE (Phan et al. 2025
partial
partial
partial
DEEPSYNTH
v
Table Comparison of datasets on various reasoning and retrieval capabilities.
of their training data. Since DEEPSYNTH contains a diverse set of tasks from multiple regions, naturally increases the overall difficulty of the benchmark.
RELATED WORK
As LLM-based agents improve in reasoning, tool usage, and interaction actoss diverse environments, researchers have sought to evaluate L.LMs on questions that require multi-hop reasoning skills (Wu
et al.
2025
Wolfson al., 2025), code generation (Jimenez et al. 2024 Chan al. 2024 Ouyang
Starace et al.). information seeking (Wei et al. 2025 Yoran et al. 2024 and even general assistance capabilities (Mialon et al. 2023). Table 6 summarises the key differences among popular agentic benchmarks; most of these correspond to the criteria we considered in $ 2.1 for DEEPSYNTH tasks' design. distinctive feature of DEEPS comprises multiple components- e.g.. JSON object containing event causes (strings), percentages (floats), dates or years (integers)-witf) wwth explicit logical links (e.g., key-value pairs). This structure makes the benchmark particularly challenging, it requires agents to retrieve, reason, and integrate heterogeneous information correctly.
et al.
2025
its multi-part answers, where each response
Several existing benchmarks share partial overlap with DEEPSYNTH but lack systematic evaluation of information synthesis. For instance, GAIA (Mallon et al. 2023) requires planning and information seeking but involves limited synthesis and less realistic tasks. BrowseComp (Wei al. 2025) is an informatiionseeking benchmark comprising challenging, invertedly constructed questions that require persistent, multi-hop web navigation uncover hard-to-find facts; in contrast, DEEPSYNTH moves beyond retrieval to systematically evaluate information synthesis through multi-part, structured answers. Assistan[Bench (Yoran al. 2024) addresses real-world gaps and includes limited multi- part answers, but omits other essential aspects. Humanity's 's Last Exam (Phan et al. (2025) offers precise, unambiguous, and non-searehable questions, yet these are often obscure and detached from real-world contexts. In contrast, DEEPS YNTH is, to the best of our knowledge, the first benchmark to systematically evaluate information synthesis across realistic, multi-step tasks.
CONCLUSION
We presented DEEPSYNTH bench, new benchmark comprising 120 challenging and diverse tasks across 67 countries. By combining planning, tool use, and multi-step reasoning, DEEPSYNTH aims evaluate the ability of agents to move beyond shallow retrieval and engage in goal-directed, information-ride problem solving. DEEPSYNTH was inspired by real-world problems, and its tasks were designed to be strictly verifiable, geopolitically diverse, and robust against memorisation. Our experiments demonstrated the difficulty of our benchmark, with both state-of-the-at L.MMs and specialized deep research agents struggling to solve any significant number of tasks. The best of the former (remini-Pro-2.5) achieved an Fl score of only 6.25 with no task reaching perfect score, while the best of the latter (o3-deep-resee deep-reseaach)) reached 8.97. These results help establish that there is substantial room for improvement on multi-source information synthesis, and we hope DEEPS Ynth will inspire future work, starting with improving navigation and synthesis, and addressing the significant geopolitical biases observed.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles agentic systems. arXiv preprint arXiv:2407.13022 2024.
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Milan Gritta, Debjit Paul, Xiaoguang Li, Lifeng Shang, Jun Wang, and Gerasimos Lampouras. Process evaluation for agentic systems. In Findings of the Association for Computational Linguistics: EACL 2026. Association for Computational Linguistics, March 2026.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081 (000081):6333666 2025.
Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Bowei Xia, Tao Sun, Ziyu Ye, Zhaoxuan Jin, Yingru Li, Qiguang Chen, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, and Guohao Li. OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation, 2025. URL BRLhttts::// httptp:://axxiv..oo www.aaaxvvvvo.o. abs/////////// 2505 23885
Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL /tttps://operreiiii
net/ fermm id=VIE ByNCM66
Jinu Lee and Julia Hockenmaier. Evaluating step-by-step reasoning traces: survey, In Chris- tos Christodoulopoolls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2025, pp. 1789-1814, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN www.aattttoolooorrr/2222555
7. doi: 10.18653/v1/2025 http://////22255..nnnnnneeeemm/ findings-emnlp.94 URL
findings-emni 94/
Zimu Lu, Yunqiao Yang. Houxing Ren, Haotian Hou, Han Xiao, Ke Wang. Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. WebGen-Bened Evaluating LLMs on Generating Interactive and Functional Websites from Scratch. arXiv preprint arXiv:2505.03733, 2025.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: A benchmark for General AI Assistants. The Twelfth International Conference on Learning Reprexentations, 2023.
Philipp Mondorf and Barbara Plank. Beyond accuracy: evaluating the reasoning behavior of large language models-a survey. arXiv preprint arXiv:2404.0/869, 2024.
OpenAl. GPT-4o System
Card,
2024.
URL
http::/cch.openai.coom
gpt 40- sa..syyttm--car..d.f
OpenAl. Deep Research System Card, 2025a.
URL
https:/cchh.ppenai.comm
Seep-researee www.sssstteeemmm card. fdd
OpenAl. GPT-5 System Card, 2025b.
URL
https://cch.openai.comm
gpt 10t-5--ystten www.syttttccacda.......
OpenAl OpenAI and o4-mini System Card, 2025c. http::///opeeaaaccooommmm 03-04-mi 33-o4-mini-sye -ts.mmmmmnisssttttaraar...

---

Published as a conference paper at ICLR 2026
Anne Ouyang, Simon Guo, Simran Arora, Alex L Zhang. William Hu, Christopher Ré, and Azalia Mirhoseini. Kernelbench: Can lims write efficient gpu kernels? arXiv preprint arXiv:2502.10517, 2025.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. REFINER: Reasoning feedback on intermediate representations. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1100-1126, St. Julian's s. Malta, March 2024a. Association for Computational Linguistics. doi: 10. 12555/ 67. URL / wrgp://ooooyr//22222//// www.:2244..1111ng...... 67/
Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings, Making reasoning matter: Measuring and improving faithfulness of chain-of of-thh reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 15012-15033 Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10. 00.18653/11/22222 org/2024, findi ngs emn 882/
/10.8//////222/nnnnnnnnmm/m///// findings-emnlp.822
URL https: //ap::///cll antho logy
Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's 's last exam. arXiv preprint arXiv:2501. 14249, 2025.
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kau- nismäki. 'smolagents': a smol library to build great agentic systems. httpp:://tttttii com/ 2025.
mugginngfacce/mmolagggg
Paul Sambre and Geert Brône. Gilles fauconnier & mark turner, " the way we think: conceptual blend- ing and the mind's hidden complexities". 2002. URL httth::///apppp semanticschol.ss.
org/Corppi Corp/Crrusllln ID: : 152136175
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. PaperBench: Evaluating Al's Ability to Replicate Al Research. In Forty-second International Conference on Machine Leurning.
Andrea C Tricco, Jennifer Tetzlaff, and David Moher. The art and science of knowledge synthesis. Journal of clinical epidemiology, 64(1):1 2011.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In Forty-first International Conference on Machine Learning, 2024.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality large language models. arXiv preprint arXiv:24/ 1.04668, 2024.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung. Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. /2516, 2025.
Tomer Wolfson, Harsh Trivedi, Mor Geva, Yoav Goldberg, Dan Roth, Tushar Khot, Ashish Sabharwal, and Reut Tsarfaty. MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents. Transactions of the Association for Computational Linguistics, 2025.
Jian WU, Linyi Yang, Dongyuan Li, Yuliang Ji, Manabu Okumura, and Yue Zhang. MMQA: Evaluating LLMs with multi-table multi-hop complex questions. In The Thirteenth International Conference Learning Representations, 2025. (tRpst::: :ttttp::://ppeeeeee /ttp://ppppe/....c//////e. net forum?
id Gd= XykcCaaa
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
12

---

Published as a conference paper at ICLR 2026
Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Methods in Natural Language Processing, pp. 8938-8966 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.1 (00.5/////222.emm///p.. 505. URL https://aclanthooooyyoooot
Empirical
ma ina.in... 505/
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MORE RESULTS AND ANALYSIS
This section collects some additional results and analysis. Specifically:
Table 7
shows how long was required for state-of-the-art LLMs and specialized deep research agents to run DEEPSYNTH bench.
Table 8 presents the intermediate step accuracy and error propagation the DEEPSYNTH Dev tasks.
Table 9 summarises the cost and output token characteristics of the models evaluated our experi- ments. For models that produce structured multi-stage reasoning traces, we report both reasoning- token ranges and final completion-toke teteetioontokennnnne ranges. Costs correspond to the total API price per full run of DEEPSYNTH task.
Table presents brief comparison across the Agentic Framework tool capabilities of the specialized
deep research agents apply to DEEPSYNTI bench (OpenAl 2025a Roucher et al. 2025 Hu
et al.
2025).
Table 11
provides definitions and examples the key information synthesis operations in DEEP- SYNTH'S tasks.
Table 12 collects Fl-score, Precision, Recall and EM scores highlight the role of planning intermediate steps on LLM models.
Table 13 presents the performance comparison the DEEPSYNTH-Dev benchmark (Pass 1).
Finally, Sesure[showe an example run using the OWL framework, and illustrates errors when trying to collect and reason about data.
Model
Avg. Time (sec.)
04-mini (2025-08)
20.8
GPT-4.1 (2025-08)
7.52
GPT-5 (2025-08)
83.41
Gemini-Pro-2.55 (2025-08)
34.10
DeepSeek-RI (2025-08)
5.2
o3-deep-research (2025-08)
645.39
Smolagent (GPT-4.1)
35.84
OWL (GPT-4.1)
1025.5
Table 7: Average Time per instance to run our benchmark
Process Evaluation Recently, several works have shown that LLMs make mistakes in their interme- diate reasoning steps (Paul et al. 2024a Mondorf & Plank 2024 Lee & Hockenmaier 2025) and can be unfaithful to their own reasoning (Paul et 2024b). Gritta et al. (2026) argued that outcome-only metrics are insufficient for critical applications and proposed compliance checklists verify that agents follow recommended protocols. Motivated by these findings, we evaluate intermediate step accuracy by requiring models emit structured outputs after each decomposition sub-step. While end-to-end F1 scores capture overall performance, they obscure where in the reasoning chain failures originate and whether errors early stages propagate to corrupt downstream computation. We evaluate three models: GPT-4.1, DeepSeek-R 1, and GPT-5.2 on 40 DEEPSYNTH tasks, scoring each intermediate output against corresponding gold answer using entity-normalie
F1 with format normalization to handle structural variation between predicted and gold ISON outputs. GPT-4.1 I and DeepSeek-RI operate without web access as eestruction-folloimin models, while GPT-5.2 operates with web search enabled via the Responses API.
Table 8 presents per-step Fl and error propagation rates. All three models exhibit a steep accuracy decay: retrieval steps (steps 1-2) achieve 2-12% F1, indicating partial but incomplete data acquisition, while computation and reasoning steps (steps 3+) collapse to near zero. Error propagation is near- total- step fails (F1 < 50), the subsequent step also fails 91-100% of the time, with recovery
14

---

Published as a conference paper at ICLR 2026
DeepSeek-RI
GPT-4.1
GPT-5.2
Avg. Prop. (%)
Step
10.0
Step 2
12.4
9.8
2.6
97.0
Step 3
3,9
3.3
0.5
100.0
Step 4
1.4
2,4
0.0
100.0
Step
0.0
100.0
Step
0.0
100.0
Final Answer
20.1
18.5
16.7
Table 8: Intermediate step accuracy and error propagation. Per-step FI (%) on 40 DEEPSYNTII tasks. Prop. denotes errоr propagation rate: the percentage of cases where failure at step & (FI < 50) also results in failure at step k++1. Steps with < 3 are omitte.. GPT-4. and DeepSeek-R2 R1 operate without web access; GPT-5.2 operates with web search enabled. All models show steep accuracy decay and near-total error propagation.
rates below 10%. This confirms that the low end-to-end F1 scores reported in Table are driven primarily by early retrieval failures that cascade irrecoverably through the reasoning chain.
notable divergence emerges between models. Among instruction followiing models, DeepSeek-RI achieves the highest intermediate accuracy at early steps (11.2% at step 1 vs. 10.0% for GPT-4.1) and the highest final-answer F1 (20. 1%), suggesting that its extended chain-of-thought reasoning provides modest advantage in producing more accurate intermediate data. However, GPT-5.2 presents strikingly different profile: despite having access to web search, it achieves the lowest intermediate step accuracy (4.1% % step 1) while maintaining comparable final F1 (16.7%). Qualitative analysis of GPT-5.2's intermediate outputs reveals the cause this apparent paradox. Rather than producing approximate values, GPT-5.2 performs genuine web search: correctly identifying target databases, API endpoints, indicator codes, and download URLs, but reports structured failures when its execution environment cannot process the retrieved sources (e.g., JavaSeript-renndrre tables, binary x files, or bulk data downloads requiring interactive queries).
Model
Cost (USD)
Output Tokens (Min / Max)
deepseek-RI
$0.041
687.4 . (avg)
GPT-4
$0.432
188 (min) / 383 (max)
GPT-5
$5.52
1600-15872 (reasoning). 201-425 (completion) 448-5760 (reasoning), 40-392 (completion)
04-mini
$1.39
o3 deep-research
$184.61
448-5760 (reasoning), 40-392 (completion)
Gemini-Pro-2.5
$11.78
12995 (avg)
Table 9: Model cost and output token ranges.
Framework
Code Interpreter
Web Search
API Calls
Document Processing Browser Simulation
03-deep-reeearch (2025-08) Smolagent (GPT:5)
OWL
indicates capability present; x indicates capability not available. Web Browser functionality is included within Web Search capabilities for applicable frameworks.
Table 10: Agentic Framework Tool Capabilities Comparison. Note:
15

---

Published as a conference paper at ICLR 2026
Operation
Description
Example
Trend Detection
Identifying patterns, directions, changes in data over time or across contexts.
Climate events the last decade to detect trend of increasing wild- fires.
Average
Determining representative value that summarises numerical data col- lected from multiple sources.
Synthesising daily temperatures from multiple weather stations to re- port the average regional tempera-
Correlation
Measuring relationships or associa- tions between two more variables.
Syathesising data from health stud- ies to find correlations between ercise frequency cholesterol els.
Ranking
Ordering items or facts based on spe- cific criteria or importance.
Compiling product reviews from dif- ferent websites and ranking products by overall rating.
Anomaly Delection
Identifying data points or patterns that significantly deviate from the norm.
Synthesising sensor data from mul- tiple factories to detect unusual ma- chine behaviour.
Counting and Compar- ing
Quantifying occurrences and com- paring values sources cate-
Counting positive vs negative men- tions of a policy in articles and comparing the proportions.
gories.
Filtering
Selecting relevant information based on criteria, quality. or thresholds.
Selecting only recent when synthe- sising research on climate change.
Table 11: Key operations in information synthesis, their definitions, and examples of application.
Performance Metries
Model
F1
Precision
Recall
EM
3.29
2.85
3.90
0.0
Intermediate Step
12.87
11.38
14.81
7.50
Gemini-Pro-2.5
6.25
4.71
9.27
0.0
Intermediate Step
10.40
8.36
13.76
7.50
Table 12: Analysis. Studying the role of planning/nnerrmediate steps.
A.2 DATACARD AND ANNOTATION GUIDELINES
DEEPSYNTH currently spans 67 unique countries, including Afghanistan, Algeria, Australia, Austria, Belgium, Bhutan, Brazil, Brunci, Cambodia, Cameroon, Canada, Chile, China, Czech Republic, Den- mark, Estonia, Fiji. Finland, France, Germany, Ghana, Grecce, Greenland, Hungary, Iceland, India, Indonesia, Italy, Japan, Laos, Latvia, Lebanon, Liechtenstein, Lithuania, Luxembourg, Malaysia, Maldives, Myanmar, Nepal, Netherlands, New Zealand, Nigeria, Norway, Pakistan, Peru, Philippines, Poland, Portugal, Qatar, Singapore, Slovakia. South Africa, South Korea, Spain, Sri Lanka, Sweden, Switzerland, Taiwan, Tajikistan, Thailand, Tunisia, Turkmenistan, United Kingdom, United States, Uzbekistan, Vietnam, and Zimbabwe.
Annotator Demographics. We provide additional information that may be relevant for analysing this dataset. Building DEEPSYNTI required the work of expert annotators, who devised the task questions and their answers, and who independently annotated the questions assess their non- ambiguity. We have 81.25% of the annotator PhD holders. Both come from the following population:
1. Age:
(a) 18 25 12%
(b) 26 35: 68%
(c) 36 45 18%
16

---

Published as a conference paper at ICLR 2026
Model
Score
Precision
Recall
Exact Match
LLM Judge Score (GITT-4.1)
LLM Baselines
04-mini (2025-12) 12)
3.26
2.63
4.29
0.0
2.5
GPT-4.1 (2025-12)
1.81
1.57
2.14
0.0
7.5
o3 (2025-12) A
6.33
5.68
7.14
1.25
10.0
GPT-5.1 (2025-12)
6.18
6.72
5.71
1.25
12.5
Semini--Pr-22. (2025-12)
4.98
7.14
0.0
5.0
Geminni-ro-33 (2025-12)
8.59
7.53
10.0
2.5
15.0
GPT-5.2 (2025-12)
15.64
18.45
13.57
2.5
5.0
DeepSeek- Chat (2025-12)
2.11
2,08
2.14
0.0
5.0
DeepSeck-Reasonee (2025-12)
5.03
4.49
5.71
1.25
7.5
Framework-based Agents
oe-deep--esearch (2025-08)
9.88
7.55
14.29
7,50
20.0
Smolagent (GPT-4.1)
6.33
5.68
7.14
7.14
7.5
OWL (GPT-4.1)
4.11
3.95
4.29
0.0
12.5
Table 13: Performance comparison the DEEPSYNTH-DE benchmark (Pass @ 1). F1, Precision, Recall and Exact Match measure the quality of model predictions. LLM Judge (% reports the average precision. Models with are models or framework which are closed, while f open- source.
2. Gender: 25% Female, 75% Male
3. Nationality: India, Greece, Luxembourg, Slovakia, UK China, Peru, Romania, Turkey, Kosovo, Germany
4. Academic Background:
(a) Bachelor's Degree: 6.25%
(b) Master's Degree: 12.5 %
(c) PhD 81.25%
The guidelines that given the annotators presented in Figures 7 and 8 The goal of this benchmark to evaluate the capability of state-of-the-art L.LM-based agents to perform information synthesis and web-based navigation across diverse real-world sources. Accordingly, focuses on capturing variation in web content across regions rather than enforcing uniformly balanced annotator demographic. While annotators were drawn from 1l countries continents, the tasks themselves cover 42 countries, and the associated webpages span a broad range of regional domains. As the benchmark evaluates an agent's ability to search, retrieve, and synthesise information from heterogeneous sources, the demographic composition of annotators does influence the underlying skill being measured.
design
three
A.3 EXAMPLES
In this section (see Table 14]. we present some representative examples from DEEPSYNTH bench. We omit some information, e.g. the reasoning trace and intermediate steps, to prevent task leakage.
A.4 MORE DETAILS ABOUT EVALUATION
F1 and LLM-Judge Metries. F1 in our benchmark is computed using exact string and numeric matching across all fields of the required JSON output. This makes it very strict: even minor deviations (a missing key, slightly different string form, or a small numerical mismatch) reduce the score sharply. In contrast, the LLM-judge is a soft metric designed to capture partial semantic correctness. It (a) rewards outputs that semantically equivalent despite surface-leve differences (e.g.. "U.S." vs. "United States"), and (b) tolerates small numerical deviations (approximately 1%-5.5%), providing a more graded signal of correctness than F1.
17

---

Published as a conference paper at ICLR 2026
Deep Information Synthesis
Deep Information Synthesis Bench
Overview
Welcomet in this task, you will help us create/generate deep research (benchmark) questions that require multi-step reasoning, tool use, or non-obvious cranections-thingg large languegge models (LLMs) struggle with today. These questions help us study how well LLMs can reseerch, plan, and reeson like human experts.
What is Information Sythesis?
Information synthesis invalves collacting information from multiple sources and reasoning over to form conerent insights
What is Novel Insight / Researchy?
Non-obvious connection between two sets of data. correlation between two things]
intuitively -olear-, Quantifying Comparing causes/etfacts events/patterns
Quantifying connection
quantifled (yet).
Primary Checklist Deep Research Questions
Requires planning and multi-step reasoning.
Involves multiple tools/actions, such as:
Search
Calsuator
Web navigation
Code executor
Image/video understanding, etc.
Answer not directly found on the Internet.
Individual facts exist online, but the connection insight not explicitly stated.
Ideally, the enswer provides novel insight into topic.
The answer clear, unambiguous, verfiable, unique (specific variable set).
Ensure conflicting official data sources exist that could different vaild
Figure 7: Annotation Guidelines
Why EM and LLM Scores? Exact Match (EM) is well-suited for our benchmark because 95% of the answers are numeric and all keys correspond to unambiguous factual fields. In this setting, strict matching provides reliable signal of correctness: either the model produces the correct value for each key or it does not. EM is also deterministic and stable, avoiding the variability or hallucinatiion-related errors that can arise with LLM-based judges.
To complement this strict measure, we additionally use an LLM-based judge that evaluates softer, semantic aspects of the output. This score captures cases where the reasoning sound and the answers are approximately correct but differ slightly in phrasing or small numerical deviations. Together, EM and the LLM score offer balanced evaluation: one measures exact factual accuracy. while the other captures approximate correctness and reasoning fidelity.
Evaluation Example Below illustrate how EM, Fl, and the LLM score behave under different model outputs.
Ground truth: ("India": 4.5, "China": 7.8, "U.S." 10.5)
Model output:
[IIndia": 3.6. "China": 8.7, "U.S.": 10.5)
Scores: EM = 0.0; Fl 33.3; LLM Score = 1.0
Model 2 output: ("India": 3.6. "United States": 10.5, "China": 8.7}
Scores: EM 0.0; F1 0.0; LLM Score 1.0
Model 3 output: ("India": 4.5, "U.S.": 10.5, "China": 7.8)
Scores: EM 1.0; F1 100; LLM Score 1.0
18

---

Published as a conference paper at ICLR 2026
Your Annotation
Instructions for Providing Answers:
eline object
Each
Format:
answers
object
Columns/ Keys; / Flaids Requirod:
Unique
Question
Reasoning Path (inctude any code used)
Answer
Data File
State which data sources were used, download the data locally, and name the downloaded file using the correspending unique ID.
Code inclusior:
code used, include vartatim insidde the Reasoning Path field the exaact commands and scripts are preserved.
Sample (single item)
"ouestion": "Compare gender
video games
changed 2804?",
"reasoningg_aati"" "1) Search Qutry: characters in gane
2) bude used: calculate the growth rate def growth():: ...
3) Netes: used ortictal statisticc site (Please DOVOT ronget report the mosce webstte)"
"answen": "X
158",
"eata_rile": "stea-it
Sample CSV Excel template (columns)
unique_id d
quastion
reasoning_patt
answer
url
Compare gender
JSON (includes codee
game:
Immppp//gamee..co
video games and how chenged
game: 15%6
httpe///gemeY.com"
Figure 8: Annotation Guidelines
According to Eurostat, among the Baltic countries, compare bi-annual data on natural gas prices for household consumers (excluding taxes and levies).
Determine, for each country, the initial bi-annusl semester (e-g., 2020-51 or (2020-52) during which natural gas prices (in euro per kloowst hourl demonstrated a sianificant upward shift, specificaly reaching or exceeding 180% increase from their 2019, Semester 2, baseline Ipre-Cavidl.
Searching 'Eurostat natural pricess household bi-annual
Browsing (url-1)
Finding the data table to download it
scroil down))
source
(url-2)
Error 4G41
Figure 9: Example using OWL, illustrating errors when trying collect and reason about data. The agent finds the right URL but fails to query the right data from the website's database interface. second attempt tries to download data file from an incorrect URL, resulting in not found error.
19

---

Published as a conference paper at ICLR 2026
Questions
Metadata
In the 2021 Canadian federal elections, compare the last opinion polling results by Abacus Data, EKOS Rescarch, Mainstrect Research, Forum Research, Research Co. and Nanos Research before the elec- tions. How accurate is cach of these polls as predictor for the actual number of votes for each party (rounded to two decimals), you should include the 6 largest parties, and 'others'. Rank each polling company by mean-squared error. Your answer should be a ISON ohject with the keys being the polling company names. The values should he the MSE (rounded to one decimal). The companies should be sorted in increasing value of MSE.
Type : Compare. Rank Region : America Domain: Sociopolitics
As Swiss Federal Statisties Officer, am studying goods transport performance by railway. From 2008 to 2024, what is the average annual change rate for domestic transport. import, export, and transit? Provide the answer as JSON object where each key is the transport type and each value the percentage change (rounded one decimal). According to ASEAN stats, between 2016-2023, which ASEAN countries' exports in eelcommunication, computer and information services had negative correlation with the total nominal GDP of all ASEAN countries combined? The final answer should be presented as a JSON: keys should be these ASEAN countries, and values should be the pearson correlation value(s) rounded to two decimals.
Type: Change Region: Europe
Domain: Transportation
Type: : Correlation
Region: Southeast Asian Domain: Economics
analysing the relative importance of airports for Qatar Airways. Using crowd-sourced data from the OpenSky Network for December 2020, can you identify which are the five most important airports for Qatar Airways flights as measured by their pagerank, using the default damping factor suggested in the paper introducing the google search engine. Your output should he ISON object with the keys being the IATA airport codes, and values being Pagerank value rounded to three decimals.
Type: : Rank Region: Middle East, Domain: Transportation
analyzing New Zealand migration data from 2001 to 2020 to identify anomalies in migration patterns. For each year, count the number of months in which the net migration was negative (< O). Please return the results in JSON format, where each key is year and the value is the number of months with negative net migration.
Type: Counting, Compare Region: New Zealand Domain: Socioeconomics
What are the tokenizer-level compression ratios (measured as bytes per token) for the following sentence: " Deep Insight Benchmark is an open-source benchmark that evaluates agents' and
Type : Task Specific Quantity
Region: None Domain: Computer Science
ability to solve tasks requiring analysis of multi-region real-world data. ", when tokenized using the tokenizers of Llama (meta- 2oots-llam//lama-2--bbffo
Qwen (Qwen/Qwen3-4B Base) models? Return the results as a ISON object, where each key is the model name and the value is the compression ratio (rounded to two decimal places).
and Apple Mtpple/aatVL.MML...
"Llama" Llloaa
"Qwen":floe :float
"Apple":ftu floaa
Table 14: Examples of questions.
Model 4 output:
["India": 100.7, "U.S." 100.6, "China": 7.8} Scores: EM= = 0.0; Fl =1=333 33.3; LLM Score = 0.0
A.5 PROMPTS
In this section, we provide the instructions and prompts used across all models; see Figures and and
20

---

Published as a conference paper at ICLR 2026
Judge whether the following [response] to [question] is correct not based the precise and unambiguous [correct_answer] below.
[question]: (question) [response]: (response)
Your judgment must be in the format and criteria specified below:
The final exact answer extracted from the [response]. Put the extracted answer as 'None' if there no exact final answer to extract from the response.
cerracte__fina_eee
(correct_answer) [correct_answer]
final answer length: Provide the overall number of unique answers that appear in [response], not just the correct ones. Be sure to provide a number, not an estimate!
reasoning: Explain why the extracted_ laaalffnllaaasaaell is correct or incorrect based on [correct_answet are meaningful differences between [correct_answer serracted_fina_aneee
focusing only on if there
and the Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer]] focus only whether the answers match.
correct: Answer 'yes' if extracted siatted_final__e final_answel matches the [correct_answer] given above, or is within small margin of error for numerical problems, a margin of 1 to 5.5 percentage points is acceptable. Answer 'no' otherwise, i.e. if there is any inconsistency, ambiguity, non-equivalency, or if the extracted answer is incorrect.
matches the [correct_answer, given above. Answer '0' otherwise, i.e. if there is any inconsistency, ambiguity, non-equivalency. or if the extracted answer is incorrect. In the case where [correct_answer] is a number percentage, then answer with the following formula to compute the normalized similarity score:
precision: Answer 'I if eatracted_finalaensee
(abs ([correct_answer) max (abs (I ([correct_ansver)) abs abal
extracted_final_anee
final precision: Extract the precision score from above, just the final score (number).
overlapping answers: List all the answers in [response] that also appear in [correct_answer]] You can consider answer from [response] to match with an answer in [correct_answer] if it is equivalent or is within small margin of error for numerical problems, to 5.5 percentage points is acceptable. List all of the [response] answer] with each answer delimited by 1 If the number of overlapping answers is zero, output 'NULL'.
margin of
answer appearing in
Figure 10: The prompt for the LLMdas-a-judge from Wolfson et al. 2025
21

---

Published as a conference paper at ICLR 2026
INSTRUCTIONS:
You are a professional researcher preparing structured,
data-driven answer.
Your task is to analyze and answer.
Answers are not report. Often time answers are numeric and need JSON outputs.
You will be given a research task by a user. Your task is generate accurate answer.
Focus on data-rich insights.
GUIDELINES:
**Answer Format**
Be analytical, avoid generalities, and ensure that each section supports data-backed.
Prioritize reliable, up-to-date official sources such as Wikipedia, government websites, etc.
Every question will already contain the format of the output. Please follow that.
Exact match is the metric.
To help extract the answer, before writing down the final answer, please using split token <Answer>;
It is of utmost importance that you follow the answer format mentioned in the question.
If the question says that the model needs to output the answer JSON. Please follow that.
If the question says that the model needs to output list of JSON. Please follow that.
2. Incomplete answers**
Do NOT give incomplete answers.
There is always an answer, and the answer often needs some deep reasoning.
3. **Iagguge***
Please only answer in English
Figure 11: System prompt provided to the model, outlining instructions, answer formatting guidelines, and language requirements to ensure structured, data-driven responses.
22


Reviews:
[{"review_id": "xCk0gwFcS8", "reviewer": "Reviewer_Pai7", "summary": "This paper introduces DeepSynth, a benchmark that evaluates agent on realistic and time consuming problems. DeepSynth covers 120 tasks across 7 domains and data sources covering 42 countries.  Each task requires agents to navigate ~4.2 web pages, and read between 1 to 15 context documents. The benchmark has been proven to be challenging even by state-of-the-art models like GPT-5 and Gemini-Pro-2.5 and agentic systems like o3-deep-research and smolagent, showing DeepSynth can serve as an arena of evaluating the capabilities of agentic systems.", "questions": "- Is EM reliable as a metric? The correlation between EM score and LLM Judge Score are not strong. Is there some examples where the EM is low but the LLM Judge Score is high & EM high but LLM Judge score low?\n- Is the model used for LLM Judge Score GPT-4.1 (the same in Wolfson et al. (2025))? In this case, is it likely that it is biased towards its own generation? How do we interpret that Smolagent (GPT-4.1) has higher LLM Judge Score than Smolagent (GPT-5), but F1 scores show otherwise.\n- Thank you for providing the time estimation! What are the approximate cost of compute for running different agents on this benchmark?\n- Is there an inter-annotator agreement rate for hypothesis validation?\n- Is the human performance 100 among experts since DeepSynth only include tasks where answers from both annotators were identical?\n- With only 120 tasks, what are the confidence intervals on the\nreported scores?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The construction process of DeepSynth is rigorous. It consists of five steps: (1) data source identification, (2) hypothesis generation, (3) hypothesis validation, (4) task formulation, and (5) data validation. All steps is done by expert annotators. The demographic information of the annotators & average time consumption are also reported in this paper.\n- DeepSynth is challenging for many state-of-the-art models and systems, showing huge gap between human capabilities and current models.\n- The analysis section gives interesting insights, and also provides insights on how the agents fail, whether the groudtruth intermediate steps will help, and how the agent behaves across tasks from different regions.", "weaknesses": "- I understand the benchmark is challenging. However, the evaluation metrics are hard to interpret (especially EM and LLM Judge Score, see the questions section). The value of EM is almost all 0s and LLM Judge Score might favor their own model family (GPT-4.1).\n- (minor) The annotator demographic might also be biased (75% male, 81.25% PhD).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T2V1eup4Jm", "reviewer": "Reviewer_kYyA", "summary": "The paper presents DEEPSYNTH, a benchmark of 120 tasks for evaluating LLM agents on multi-source information synthesis across 7 domains and 42 countries. It uses a multi-stage expert-driven pipeline for task creation and shows SOTA agents (e.g., GPT-5, o3-deep-research) achieve low F1 scores (max 8.97), exposing gaps in reasoning and tool use.", "questions": "- Can you provide inter-annotator agreement metrics (e.g., Cohen's kappa) for hypothesis validation and task formulation to address subjectivity concerns.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Multi-stage pipeline ensures non-memorizable, diverse tasks.\n- Covers 9 models/agents with metrics like F1/EM and ablations.\n- Breaks down performance by steps/operations.", "weaknesses": "- Lacks full code/prompts, data promised post-acceptance.\n- Annotator selection biased; uneven regional coverage.\n- Overlaps with prior benchmarks without direct comparisons.\n- Low baselines may reflect poor prompting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXNSIDbyYm", "reviewer": "Reviewer_hwhP", "summary": "This paper proposes DEEPSYNTH, a high-quality, well-designed benchmark comprising 120 challenging and diverse information synthesis tasks across 42 countries and 7 domains. The authors clearly articulate the motivation, methodology, data collection and curation process, followed by a detailed analysis of the dataset characteristics. The paper presents comprehensive experiments evaluating popular SOTA models and specialized agent frameworks, with detailed performance analysis revealing significant limitations in current systems.\n\nOverall, this is a solid paper with rigorous benchmark design and high-quality implementation. The presentation effectively explains the design choices and underlying rationale. The work addresses an important gap in agent evaluation by focusing on information synthesis rather than simple fact retrieval. Recommend accept.", "questions": "- Regarding the inherently challenging nature of the benchmark, what are the authors' thoughts on improving the applicability so that less powerful models can also derive meaningful evaluation signals from the benchmark? Would incorporating graduated difficulty levels be beneficial?\n- Although one design principle emphasizes 'robustness against memorization,' once the benchmark becomes public, won't it inevitably face contamination issues? How do the authors plan to prevent models from over-optimizing specifically for this benchmark, which could cause unfairness when comparing against other models that haven't been exposed to these tasks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Excellent observation regarding the lack of real-world tasks that require synthesizing information from multiple sources, with strong motivation for designing such a benchmark that is well-suited for advancing current LLM and agent systems which are becoming increasingly powerful and saturating existing benchmarks.\n- Dataset collection and curation process is exceptionally well-designed, executed, and presented through a rigorous 4-stage pipeline (data source identification, hypothesis generation, validation, task formulation), enabling readers and users to easily understand the characteristics and quality standards of the benchmark.\n- The experiments with SOTA models are comprehensively designed and the analysis and ablations are thoroughly conducted, unveiling the significant incapabilities of current models (best F1 score only 8.97) and underscoring the substantial value and challenge posed by this benchmark.\n- The findings derived from the dataset creation process and experimental analysis are insightful and well-supported with quantitative evidence.", "weaknesses": "- Overly difficult questions might limit the practical applicability of the benchmark, making evaluations of less powerful models even more challenging. For example, if less powerful models all achieve nearly zero performance, it becomes difficult to obtain meaningful signals to differentiate and evaluate which model performs better. In comparison, benchmarks with graduated difficulty levels generally suffer less from this limitation.\n- Since the benchmark requires gathering information from the dynamic Internet that constantly changes, some websites may become unavailable over time (e.g., pages/tables expire or are restructured). This could make certain tasks inaccessible or make long-term maintenance of the benchmark particularly challenging, potentially affecting reproducibility and consistency of evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oaj7MjU1m5", "reviewer": "Reviewer_fzgH", "summary": "This paper introduces DeepSynth, a very challenging benchmark for evaluating LLM agents. DeepSynth consists of 120 diverse tasks created by 16 experts, where each task requires an agent to navigate through about 4 web pages and read up to 15 documents and tables. The tasks are designed to reflect real-world analysis demand, and cover knowledge across 42 countries. The authors evaluated 5 state-of-the-art models (o4-mini, GPT-4.1, GPT-5, Gemini-2.5-Pro and DeepSeek-R1), as well as 3 state-of-the-art agentic frameworks (o3-deep-research, smolagents and OWL) on DeepSynth. It is observed that this benchmark is challenging for all these models and frameworks, with the best F1 score being only 8.97 points. The authors also conducted a bunch of ablation studies to understand the challenges in DeepSynth. They found models perform worse as the number of intermediate steps increases, and most failures are caused by either navigation error or synthesis error, which suggests future directions for developing LLM agents.", "questions": "1. In Figure 5, the space between types should be larger for better readability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. There lack good benchmarks for developing LLM agents. The DeepSynth benchmark in this paper will help accelerate the research and development of LLM agents.\n2. The DeepSynth benchmark is designed to reflect real-world demand. The data curation looks reasonable and solid, though the dataset scale of 120 tasks may be a little bit small.\n3. The benchmark is challenging enough even for state-of-the-art models and agentic frameworks. It reveals failures in existing LLM agents and shows substantial room for future improvement.", "weaknesses": "1. Information synthesis may not be an intuitive title for the community. To my understanding, what you meant by information synthesis is closely related to multi-step reasoning or agentic tasks. Maybe you can consider agentic tasks as your title?\n2. There is a substantial gap between the F1 score and LLM judge score. In my opinion, this indicates that your ground truth answer may have different semantic forms, and string matching doesn’t work well there. Could you evaluate this issue with experiments, and provide some guidelines for those who will use your benchmark?\n3. There isn’t a human baseline for this benchmark. A human baseline will let us understand whether the benchmark is solvable for average humans or require specific domain knowledge. I understand this may not be completed during rebuttal since each task requires 5.5 hours for humans on average, but it’s nice to have human baseline results later.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)