Demystifying Deep Search: A Holistic Evaluation with Hint-free Multi-Hop Questions and Factorised Metrics
Authors: Maojia Song
Liu Renhang
Xinyu Wang
Yong Jiang
Pengjun Xie
Fei Huang
Soujanya Poria
Jingren Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Deep Search Agent
Primary area: datasets and benchmarks
TL;DR: We propose a new evaluation paradigm for deep search that identifies specific LLM failure sources, introduces challenging hint-free datasets with holistic evaluation, and offers a strong baseline incorporating memory and verification.

Abstract: RAG (Retrieval-Augmented Generation) systems and web agents are increasingly evaluated on multi-hop deep search tasks, yet current practice suffers from two major limitations. First, most benchmarks leak the reasoning path in the question text, allowing models to follow surface cues rather than discover reasoning chains autonomously. Second, evaluation is typically reduced to a single pass rate, which collapses diverse behaviors into one score and obscures whether failures stem from inadequate search, poor knowledge use, or inappropriate refusal. To address these issues, we present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox that ensures full traceability of model actions, and a holistic evaluation framework that separates search sufficiency, knowledge utilization, and refusal behavior. Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures: models struggle with knowledge utilization despite having sufficient evidence and demonstrate near-absent appropriate refusal when evidence is lacking. These patterns expose a fundamental gap—today's systems excel at executing given reasoning paths but fail when required to discover them. We develop an agentic workflow EvidenceLoop that explicitly targets the challenges our benchmark identifies, incorporating verification loops and systematic evidence tracking that improve both search and synthesis capabilities. This baseline demonstrates that WebDetective's diagnostic framework can guide concrete architectural improvements, establishing our benchmark as a critical tool for developing genuinely autonomous reasoning systems rather than pattern-following agents.

DEMYSTIFYING DEEP SEARCH: A HOLISTIC EVALU- ATION WITH HINT-FREE MULTI-HOP QUESTIONS AND FACTORISED METRICS
Maojia Song"
Renhang Liu'
Singapore University of Technology and Design (SUTD) maojia songenymail. sutd. edu.sg
Nanyang Technological University (NTU) renhang: liu@ntu. eewientu.ddu.. edd.. sg
Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang Tongyi Lab, Alibaba Group {xinyu. wxy, jianqyong. iy, xiepengjjn.xpii f. E.huang/@llibbba-innee
Soujanya Porial
Jingren Zhou
Nanyang Technological University (NTU) poriaantu..... adu.....t....uu....
Tongyi Lab, Alibaba Group jingren, achou@alibabal inc. com
ABSTRACT
RAG Reetrievll-Augmentee Generation) systems and web agents are increasingly evaluated multi-hop deep search tasks, yet current practice suffers from major limitations. First, most benchmarks leak the reasoning path in the question text, allowing models follow surface cues rather than discover reasoning chains autonomously. Second, evaluation is typically reduced to single pass rate, which collapses diverse behaviours into one score and obscures whether failures stem from inadequate search, poor knowledge use, inappropriate refusal. To address these issues, we present WebDeveetive, benchmark hint-free multi-hop ques- tions paired with controlled Wikipedia sandbox that ensures full traceability of model actions, and a holistic evaluation framework that separates search suffi- ciency, knowledge utilisation, and refusal behaviour. Our evaluation of 25 state- of-the-art models reveals systematic weaknesses across all architectures: models struggle with knowledge utilisation despite having sufficient evidence and demon- strate near-absent appropriate refusal when evidence is lacking. These patterns expose fundamental gap-today's 's systems excel executing given reasoning paths but fail when required to discover them. We develop agentic workflow EvidenceLoop that explicitly targets the challenges our benchmark identifies, in- corporating verification loops and systematic evidence tracking that improve both search and synthesis capabilities. This baseline demonstrates that WebDetective's 's diagnostic framework can guide concrete architectural improvements, establishing our benchmark as a critical tool for developing genuinely autonomous reasoning systems rather than pattern-fol foolowwig agents.
INTRODUCTION
Web autooomuus systems that navigate and extract information online-exee extend language models beyond parametric knowledge by combining search with internal reasoning. aggregating evidence, and synthesising answers Nakano et al. (202I); Ferrag al. (2025). A core evaluation setting for these systems is deep search: multi-step discovery under noise, hypothesis generation, and the persistent "I can't find it" barrier faced even by expert searchers. Recent work improves agent designs [i et al.) (2025b)c); : [un et (2025)) Diang et al. (2025) Li al. al. (2025a) and proposes
new benchmarks Du al. 2025
/Wong al. (2025)
"qqual Contributions.
Corresponding author.

---

PATH-HINTING (eg.. Hotpol (3A)
SPPEIFICATION-MIIINGG (egg. BroweseCompp
HINT-FREE
(Curs: WebDelechhe
the busband of sispmather boother of Kane Comes?
Which radio presenter at 5&d former loctnalar, at least ytart senior to his wito who was 2007 Labor candidata?
Dusetione ensunes be hindd atrelghttorwerr Wed rigeent needs to find exidensos via oafh aoocotion
Cornes? ?
Kane
Cornee
Search Biratee (Himed
Open Web
"Rado Presenter SAA & Focttaelerr her" (Hinnted by GI
Beother
Birther
wib Hirt
OF
Chad Cores
Unknown
Chad Cornee
Chad
Chad
= -00 people
Fitor
Steproofha
20 years
Explore
Stepmuther
Slepmather
(Minted by C)
Wil Hini
Nicoles Cornes
Filler
Unknown
Niooles Cornee
Nlotles Comes
Search Husband
2007 Labor (Mintedd by Q)
web Hitt
Graham Comes
Graham Cames
Unknown
Graham
Figure 1: Comparison of different question formulations in multi-hop deep search. Left: Path- Hinting (PH) benchmarks such as HotpoiQA embed the reasoning path directly in the question text, effectively reducing reasoning to execution. Middle: Dpecification-Hiniing (SH) benchmarks such as BrowseComp obscure the target entity behind multiple attributes, testing filtering rather than autonomous exploration. Right: Our Hint- Free (HF) formulation in WebDetective removes both path and specification hints, requiring agents to autonomously discover reasoning chains within a controlled Wikipedia sandbox.
However, identify largely overlooked issue: many deep-search benchmarks contain hinting that steers models toward the answer, reducing the need for genuine autonomous reasoning. Such hints bypass a key capability of web agents: discovering which connections matter, forming hypotheses about promising reasoning trajectories, and adaptively exploring without guidance.
As shown in Figure 1 classical multi-hop QA datasets (e.g., HotpotQA Yang et al. 20)8 exhibit
Path- Path-Hinting (PH), where the question narrates the reasoning chain (e.g., "Who is the husband of the stepmother of the brother of Kane Cornes?"), turning the task into executing a prescribed path. Newer benchmarks (e.g., BrowseComp Wei et al. (2225) and WebShaper Tao et al. (2025) introduce Specification-Hintine (SH), describing the target via distinctive attributes. While less explicit, SH often becomes constraint filtering rather than exploratory search. In both cases, agents receive scaffolding uncommon in real use, limiting evaluation of true autonomy.
A second limitation is that evaluations typically report only aggregate pass rates, collapsing distinct failure modes. An agent that searches well but fails to synthesise evidence differs fundamentally from one that stops early, over-relies on parametric knowledge, refuses when it should continue. When evidence is insufficient, models should recognise and signal uncertainty rather than guess. Without separating these behaviours, diagnosis and progress are difficult.
We introduce WehDetective, a benchmark that co-designs hint-free questions and a controlled en- vironment for fine-grained diagnostics. We construct Hint-Free (HF) Multi-Hop questions that avoid both path narration and attribute fingerprints while retaining a unique answer (e.g.. "Who is the father of Kane Cornes?"). Questions are paired with a controlled Wikipedia sandbox that in- cludes reference reasoning path. To prevent shortcutting, each node in the chain is masked so the next entity is reachable only through its predecessor. This supports comparison between arbitrary valid strategies and canonical shortest search-only path, while disentangling search sufficiency, synthesis, and final success. We also propose an agentic baseline workflow, EvidenceL.oop with explicit context retention, memory management, and verification steps.
Using our diagnostic framework, we evaluate 25 representative models and observe a consistent pattern: without hints, even strong systems succeed only a limited subset, and their failures differ from what conventional benchmarks suggest. Many retrieve relevant evidence but fail to assemble coherent reasoning chain; others abandon search prematurely or produce unsupported answers despite clear knowledge gaps, indicating weak calibrated refusal. These results highlight the need to evaluate path discovery rather than path execution.
2

---

Our contributions
WebDetective, the first benchmark systematically remove both path and specification hinting while enabling controlled, fine-grained diagnostics.
An evaluation framework that separates search sufficiency, information synthesis, and re- fusal behaviour to pinpoint failure modes in deep-search agents.
EvidenceL.oop. an agentic workflow with context retention, memory management, and verification, demonstrating how the benchmark can guide concrete system improvements.
2
THE WEBDETECTIVE BENCHMARK
2.1 HINT-FREE MULTI-HOP QUESTION ANSWERING
Given a question and a knowledge corpus C, an agent must find the answer a* by discovering and composing a sequence evidence pieces & e., from C. Each e; is atomic fact on the page of entity that links (semantically
relationally) to *++1, forming reasoning chain Po 7 V1 - + Uns where Up is the starting entity (typically mentioned in q) and Un yields a". The reasoning function Rimee 3 + a" specifies how these facts are composed -vii logical inference, relation transitivity, or domain rules. We call any information embedded in g that reveals Riune fingerprints Un a hint h.
We define two prevalent hint types in existing multi-hop QA benchmarks Yang et al. (0T8h)) Chen al. (2019); Wei et al. (2025):
Path- Wath-Hinting (PH): The question directly encodes the reasoning chain, with /Pи Encode (func)) revealing its structure. For example, "Who is the husband of the stepmother of the brother of Kane Comnes"" decomposes into: find brother - stepmother - husband. The agent no longer discovers the path, but executes the specified hрH-
Specification- Hiittinnnnnnngggg (SH): The question hides the target behind constraints, with hSH = {si, 82..... } narrowing the search space to unique entity. For instance, "Which radio pre- senter at 5AA, former footballer, at least 20 years senior to his wife who was 2007 La- bor candidate?" yields hSh = (radio presenter 5AA, former footballer, 20+ years senior wife, wife was 2007 Labor candidate), collectively fingerprinting Graham Cornes; the task reduces to constraint satisfaction (match all s,) rather than discovering which connections matter for reasoning.
In contrast, we propose Hint-Free (HF) Multi-Hop QA, where h. = 0. Questions contain only the essential query, without path narration or attribute fingerprints-for example, "Who is the father of Kane Cornes?" Answering requires discovering evidence pieces (Kane Cornes has brother Chad Comees, 62 (Chad's stepmother is Nicole Cornes), e3 (Nicole's husband is Graham Cornes), and composing them via familial reasoning to derive a" = Graham Cornes. Crucially, the agent must independently uncover both evidence chain and the reasoning function Rrunes capturing the fundamental ability to transform simple information need into autonomously constructed reasoning structure.
2.2 THE Co- PRINCIPLE
While hint-free questions eliminate linguistic scaffolding, we observe that question design alone is insufficient to ensure genuine multi-hop reasoning. open corpora or live web environments, even well-designed hint-free questions permit shortcuts that bypass the intended reasoning chain. Con- sider our example "Who the father Kane?" in Wikipedia or web search, direct co-occurrences of "Kane" and "Graham Cornes" may exist in unrelated contexts, intermediate entities like "Chad Cornes" could be found through direct search, allowing agents to bypass the intended reasoning path. Moreover, because both answers and intermediates are usually accessible through multiple paths, it becomes impossible to tell whether an agent truly discovered the reasoning chain or simply relied on shorteuts prior knowledge.
is discoverable
agent visits page(u;-)
3

---

This masking eliminates corpus- shorteuts agents cannot find the answer through spurious co-occurrences or unrelated contexts-wiinn still accommodating diverse valid reasoning strategies. The ground-truth chain serves reference path (the shortest sufficient route using the corpus alone), but our evaluation framework does not enforce strict adherence to this path. To compre- hensively assess whether an agent's search is sufficient and efficient, we employ multi-faceted evaluation:
(I) Shortest Reference Chain Coverage. We first check whether all hops in the reference chain were visited. For any missing evidence, we probe the model's parametric knowledge with targeted queries (e.g., "Kane Coree has brother ?"). If visited evidence plus verified parametric knowl- edge covers the complete chain, the instance achieves knowledge sufficiency.
(2) Alternative Path Recognition. Agents may discover valid reasoning routes different from the reference chain. We collect all evidence the agent actually visited during search and feed this clean context to an LLM judge. If the judge can correctly answer the question from this visited evidence alone, the instance also achieves knowledge sufficiency-rggardless of whether the reference chain was followed. Consider one real example from our benchmark: "On which compilation album does the song 'Loneliness of a Middle Distance Runner' appear?" The reference path follows: Novel (The Loneliness of the Long-Distance Runner) - Single (Jonathan David) + Compilation (Push Barman to Open Old Wounds), tracing through the single that featured the song B-side. However, an agent might instead traverse Novel + Belle and Sebastian (the band mentioned as adapting the title) the band's discography page listing "Push Barman to Open Old Wounds" as a B-sides compilation-rrachiin the same answer through the artist's discography rather than the specific single.
(3) Efficient Parametric Knowledge Use. Our Search Score further credits agents that efficiently combine search with parametric knowledge. When an agent correctly answers the question using fewer (or equal) hops than the reference path while actively performing search, this demonstrates in- telligent leveraging internal knowledge shorteut the reasoning process, and receives additional credit.
Returning to our rumning example, "Who is the father of Kane Cornes?", the reference path is Kane + Chad (brother) > Nicole (stepmother) - Graham (husband). An agent achieves knowledge suf- ficiency by: (a) visiting all four pages, (b) visiting partial pages while parametric probes confirm missing knowledge, or (c) discovering alternative evidence that supports the correct answer. An agent that correctly answers using only two hops (c.g.. Kane + Chad, then leveraging parametric knowledge about Chad's family) receives Search Score credit for efficient reasoning. This design eliminates unearned shortcuts while rewarding genuine multi-hop reasoning through any valid strat- egy.
Lastly, while we instantiate this benchmark using Wikipedia as a controlled testbed, the co-design methodology is not inherently tied to this corpus. The framework generalises to any domain satisfy- ing three conditions: (1) a text corpus with factual content, (2) an entity-level graph or link structure over that corpus, and (3) the ability mask entity mentions along chosen paths. Given an appropri- ate dataset meeting these requirements, the same evaluation pipeline- path construction, selective masking, and factorised metrics-can be directly applied news archives, scientific repositories, enterprise knowledge bases, or curated web snapshots, extending the diagnostic value beyond the current Wikipedia instantiation.
2.3 BEYOND PASS RATE: DIAGNOSTIC EVALUATION FRAMEWORK
Traditional multi-hop QA evaluation reduces performance to single pass rate, obscuring distinct failure modes: agent that searches exhaustively but fails to synthesise evidence is fundamen- tally different from one that refuses prematurely or hallucinates from parametric knowledge. Our sandbox, with guaranteed unique reasoning paths, enables precise diagnosis by separating knowl- edge sufficiency (whether the agent has obtained the necessary evidence through search or memory) from generation quality (whether it can synthesise a correct answer or appropriately refuse). This decomposition reveals that similar pass rates can mask very different underlying capabilities. See A Appendix A for details.
4

---

Knowledge Discovery Metries. We assess whether agents acquire necessary information through two complementary metrics. Knowledge Sufficiency determines if an agent possesses all required evidence & (ei,..., Ca for answering either from search parametric knowledge. We track which evidence the agent discovered through search by monitoring visited pages in our sandbox. For any missing evidence 4 & Etound, we probe the model's parametric knowledge with targeted queries (e.g., "Kane Cornes has brother _?"). The Search Score extends this by crediting models that efficiently combine partial search with parametric
recogniinn that if an entity discovered through search has meaningful relationship to the answer stored in parametric memory, this represents legitimate reasoning that demonstrates efficient knowledge utilization.
Generation Quality Metrics. Conditioned on knowledge sufficiency, we classify cases as knowledge-sufficee (S) or insufficient (I), and attempted (AA) or refused (N). This yields two key metrics. Good Refusal (GR) evaluates abstention when evidence is lacking, with FLGR balancing recall (avoiding hallucination) and precision (refusing only when justified). Knowledge Utilisation (KU) measures synthesis when evidence is present, with Flgu balancing recall (using available ev- idence) and precision (answers grounded rather than speculative). These complementary Fl scores capture refusal discipline and evidence integration.
We define a unified Generation Score as
FlGR + Flgu
GenScore = 2
-KnowledgeScore,
where Knowledge Score is the fraction of instances with knowledge sufficiency (all evidence gath- ered). The sufficiency weighting prevents gaming by agents that simply refuse all questions, ensur- ing models are rewarded only when they both acquire the necessary evidence and handle it appro- thiouh correct synthesis or justified refusal.
Knowledge Degradation Analysis. Even when models achieve knowledge sufficiency, they often fail to generate correct answers, revealing a gap between evidence possession and synthesis. To diagnose this gap, we design two tests. The Knowledge Forget test captures cases where models fail to apply parametric knowledge in full-context reasoning, despite succeeding on isolated probes. The Lead-astray test captures failures caused by noisy search contexts irrelevant pages, failed at- tempts, or exploration clutter-that prevent models from synthesising answers they could otherwise produce from clean evidence. Together, these metrics go beyond simple pass rates by pinpoint- ing whether errors stem from inadequate search, over-confident hallucination, weak synthesis, or degraded reasoning under noise, thereby clarifying the capabilitics required for robust multi-hop reasoning.
2.4 DATASET CONSTRUCTION
We build WebDetective through a pipeline that transforms single-hop Wikipedia QA pairs into ver- ified multi-hop reasoning chains, ensuring every hop is necessary. See Appendix B and Appendix C for details.
Source Data and Chain Discovery. Starting from Wikipedia QA pairs where each question references starting entity vo and has an answer Un, we remove the direct link up - u, and use BFS on the hyperlink graph to find the shortest alternative path Up -> u1 -> - Un, For each edge (we, 1)+ we extract the sentence e, in the page of v; that links v and 0,41, forming the evidence chain E = (en..
Verification of Necessity. Since many graph paths are irrelevant, we apply three automated checks with strong LM (Qwen-3-235B-A2D))
(1) Parametric Inaccessibility: LM(g) & Un. ensuring the answer is not retrievable from parametric memory alone. (2) Evidence Sufficiency; LM(q. E) = Uns confirming the full chain supports the answer. (3) Evidence Necessity: LM(q. E (ee)) # Un for all e, verifying no hop is redundant.
Human Validation. Surviving questions are manually checked to ensure all evidence is required, reasoning is logically sound, and no hints are embedded in the wording. The final benchmark contains 200 validated questions with varied hop counts and types.
Domain Coverage. The final dataset spans diverse domains to ensure broad evaluation coverage: Film, Television & Theatre (20.0%), History, Politics & Military (18.0%). Geography & Landmarks
5

---

(16.0%), Music (13.5%). Sports (11.5%), Biography & Education (8.5%), Literature & Pop Culture (6.5%), and Science & Technology (6.0%). This distribution ensures WebDetective is not skewed toward any single relation type but covers reasoning patterns essential for evaluating multi-hop ca- pabilities.
3 A BASELINE ATTEMPT: THE EVIDENCELOOP FRAMEWORK
To address the unique challenges posed by hint-free multi-hop reasoning, we develop Evi- denceLoop, an agentic workflow baseline that explicitly incorporates context retention, memory management, and verification steps to maintain reasoning coherence across extended search trajec- tories. Unlike standard ReAct implementations that can lose track of evidence across many search iterations, our workflow introduces structured mechanisms for tracking discovered entities, main- taining evidence chains, and verifying reasoning paths before answer generation.
Iterative Refinement with Fallback. Our framework balances exploration breadth with compu- tational efficiency through Rmax iterations. In each round r, N solver agents explore different reasoning paths in parallel, each operating with up to B actions and guided by an aggregated con- text C" (C 0). Their outputs are refined through two stages: (I) an extraction agent distills key findings, entity references, and promising paths; and (2) an aggregation agent synthesises them into Cm+11 retaining valuable evidence while discarding noise. This process allows early iterations to explore broadly- sportss connections, geographic links, family ties ties-while progressively focusing on the most promising directions. no conclusive answer is found after Rymax rounds, a final fall- back stage consolidates all evidence into (thnnal and invokes synthesis-only solver, distinguishing failures due to insufficient exploration from those due to poor evidence composition.
Evidence Memory System. Iterative refinement is supported by persistent memory M that records all evidence retrieved during search. Each piece of evidence is assigned a unique Evidence ID (EID) and stored with its full content. Agents receive both summaries and EID references (e.g.. "Kane has brother Chad [EID-042]"), enabling them to reason over concise contexts while retaining the ability to fetch full documents prevents agents from being overwhelmed by long documents or lossy compressions: they work with focused summaries but never lose access the complete evidence trail, EIDs also enable systematic traceability, as later modules can verify claims against their original sources.
demand through retrieve action. This dual representation
To ensure evidence-gee evidence-grounded reasoning, any proposed answer must be decomposed Cm. ). each linked to an EID. A verification agent V checks that (1) each claim entailed by the source content, (2) the claims collectively support the proposed answer, and (3) the answer directly addresses the question. Verification occurs during execution: rejected pro- posals return specific feedback, allowing solvers repair reasoning within the remaining budget B, while accepted answers immediately terminate exploration. This mechanism prevents hallucinations from propagating, enforces tight evidence grounding, and improves efficiency by halting search as soon verification succeeds.
Verification.
into atomic claims (ci,...,
Positioning Relative to Existing Frameworks. EvidenceLoop a diagnostically-derived work- flow designed to address the specific failure modes revealed by WebDetective. Table summarisee the key architectural differences from standard approaches. We provide a more detailed descrip- tion of the EvidenceLoop architecture, including its controller configuration, memory modules, and verification procedures in Appendix
Table I: Architectural comparison of EvidenceLoop with existing agentic frameworks (Standard
ReAct and ReAct with Reflection).
Aspect
Standard ReAct
ReAct + Self-Refecttonn
EvidenceLoog (Ours)
Controller Loop
Single Mhonghh-tool-bbeeraatt
Same occasional refiection reflection messages
Two-phase: explocation then verification
Memory
dialogue history
Structured buffer: fentity. snippet, source) tuples
Verification
Offen none
Critiques not tied to evidence
Explicit loop using only evidence baffer
Brendth Ieerations
Single trajectory Implicit, rarely triggered
Single tmjectory
Parallel imjectories aggrege- cycles
Refusal Bebaviour
Loosely specified
Expliciily tied verification: incomplete + refuse

---

Table 2: Comparison of 25 state-of-the-art models with ReAct-style tool capabilities. Metrics cover Knowledge Discovery (Knowledge Sufficiency, Search Score), Generation Quality (Genera- tion Score, Good Refusal FI, Knowledge Utilisation FI), Knowledge Degradation (Forget, Lead- astray), and @ Bold values denote best results: higher is better for Knowledge Discovery, Generation Quality, and Pass @ I, while lower indicates greater robustness for Knowledge Degrada- tion. EvidenceLoop uses DeepSeek-RI as its base model; shaded rows highlight this comparison pair.
Knowlelgg Discores
Gencention Quality
Knowlolge Deginddtion
Provider
Uli..
Prss@1
Sulf. (%)
Soare
Score (%)
(%)
F1(
(%9
(%)
(5)
201ch
23.59
6.00
12.80
12.00
18:15
17.35
21.91
21.00
OpenAl
(1.00)
78,00
15:20
3.20
48.97
24.20
24.29
53.50
(i4 531
9.37
50.00
GPTS Opecny
79.00
$8.00
23.21
3.50
40.55
17.22
32.91
50.50
1-hmn Antonm JIN
M6-01
35.50
301
Anthoopic
Chaab--quu-41
78.00
76.30
25.57
46.54
31.00
44.50
Google
Gemiii2
Google
Depelitd 105
19:00
64.50
40.56
16.35
XAI
God. Gol-4xi
2005
74.00
77.50
34.71
37.63
56.19
23:65
27.70
50.50
Althaha
1558
reserieed Teats
2015
$3.50
$7.50
4.29
0.00
15.69
43.93
41.12
18.50
Bysnioonee
Dedhon 16-Thin
64.00
18.11
21.10
Ziepa
GLM-45-W6
luz
16211 (ila)
$5.50
63.50
60.50
12.31
26.39
34.79
35.09
11.97
48.18
4155
1500
Miwisdun
Kim- Aim-K240) K244
Kim--2-004 K2-0--2-- posteasrr 2009
Moortaa Al 100
54.30
53.00
$5,00
59.00
13.17
9:22
28:99
16.36
20.89
19.31
49005
43.12
36.00
33.96
24.00
23.50
DespSeck
61.00
1360
27:47
16.34
44.23
48.72
De BeepSank-V V.
Tommrnn Cxre 2006
55.00
58.50
16.31
16.49
22.25
24:33
50.85
24.50
4 EXPERIMENTS
We evaluate 25 state-of-the-art models with ReAct-style tool use, including those from OpenAI, Anthropic, Google, xAI, Alibaba, Bynce, Zhipu AI, Moonshot AI, and High-Flyer. All models interleave reasoning, search, and observations in our controlled Wikipedia sandbox on WehDetec- tive with 200 hint-free multi-hop questions (2-4 hops), under limits of 40 tool calls and a 32K-token context. Unless noted, decoding uses temperature=0. (1) Knowledge Score, knowledge sufficiency; (2) Search Score, retrieval effectiveness; (3) Gener- ation Score, weighted Fl of Good Refusal and Knowledge Utilization; (4) Good Refussal FI; (5) Knowledge Utilization F/; and (6) Pass @ /. We additionally analyse Forget and Lead-astray to probe knowledge degradation (Section 4.2.2p. For our EvidenceLoop. we use DeepSeek-RR as the base model and set breadth=3, itteraiionn3 Comprehensive results appear in Table 2
top-p=0. 95. We report six metries:
4.1 MAIN RESULTS
Frontier models are far from saturating the task. Even the strongest systems reach only ~50% Pass@1 1 on benchmark: o3-Pro tops out at 56.0%, while GPT-5 and Grok-4 both achieve 50.5%; Claude-Opus-4.1 is at 44.5%, and many others fall well below 40%. This illustrates the challenging nature of our benchmark, WebDetective.
Search, generation, and final accuracy are decoupled. High retricval does not translate propor- tionally into better synthesis or Pass@) 1. For example, GPT-5 attains an 80.0% Search Score but only 23.21 % Generation Score and 50.5% Pass 1; o3-Pro similarly has 78.0 Search but 20.86 Gen- eration (56.0% Pass @1 1). Conversely, Grok-4 achieves the highest Generation Score (34.71) with 77.5 Search and 50.5% % Pass@1, while Qwen3-235B-Thine Generation and 21.5% % Pass 1. These gaps indicate that information synthesis, not just retrieval, a key bottleneck.
posts 72.0% Search yet just 11.15%
Refusal ability is underdeveloped. Good-refusal performance is generally low: the best ob- serve is 53.95% F1 (Doubao- 1.6-Flash). Many frontier models underperform markedly-e.. e.g., GPT- 5 (8.89%) o3-Pro (9.37%), and M4-Mini (19.75%) and even strong generalists like Claude-Opus-
7

---

Table 3: Emergent model profiles from metric interplay analysis. Models cluster into distinct behavioural profiles based on their knowledge sufficiency, refusal behaviour, and information syn- thesis, revealing characteristic strengths and failure modes across the spectrum of frontier systems.
Profile
Metric Patrorn
Passe1
Example Models
Fallure Mode
Knowledge
Refusal
Utilization
Powerful but Overconfident
High
Low
High
50-566%
GPT-S, a3-Pro, 03
Hallucination from overconfiencce
Well-Caltbrated Elite
High
Med
High
44-51%
Grok-4, Clande-quus-411
Miner: unneorseary cantion
Sywthesi Bettlenek
High
Low
Low
18-22%
Qwen3-2358, Tongyi-DR
Cannet compose multi-hop ressoning
Consrvvtive Midille
Mod
Med
Med
29-39%
Chande Sonmet4,,, GLM-455
Under-utilzees capubiliiies
Confused
Low
Low
20-2256
o4-Mini, DeepSeek-R1
Poor synthesis poor calibration
Self Self-Aware Weakness
Low
High
Low
13-18%
Douban variants, Gemini Flash
Comprehensive inability (appropriate)
Ideal (Unachieved)
High
High
High
None
None optimal behavior
4.1 remain modest (28.57%). This highlights weak calibrated abstention when evidence is insuffi cient.
EvidenceLoop demonstrates targeted improvements. Comparing Evidence its base model DeepSeek-R I reveals meaningful gains from our diagnostically-derinn design: Pass@ im- proves from 20.0% to 25.0% (+25% relative), Generation Score from 10.57% to 12.61% % (+19% relative), and most notably, Knowledge Utilization F1 from 15.55% to 23.79% (+53% relative). The substantial improvement in Knowledge Utilization demonstrates that our structured evidence buffer and verification loop directly address the synthesis bottleneck that WebDetective identifies as crit- ical failure mode. While absolute gains are moderate, these improvements validate that the failure modes exposed by our diagnostic framework can guide concrete architectural improvements. We conduct additional experiments across different base models on EvidenceLoop and investigate the effectiveness of each component in Appendix E
4.2
ANALYSIS
4.2.1
UNDERSTANDING MODEL FAILURE MODES THROUGH METRIC PATTERNS
To better understand the diverse failure modes in multi-hop reasoning, we analyze the interplay between our three core metrics: Knowledge Sufficiency (ability to gather evidence), Good Refusal Fl (calibration of uncertainty), and Knowledge Utilization Fl (synthesis capability). Rather than examining metries in isolation, we investigate how their combinations reveal distinct behavioral profiles.
We categorize performance using empirically-derivee deriieed thresholds: Knowledge Sufficiency (High: 70%, Medium: 60-70%, Low: < 60%). Good Refusal F1 (High: 40%, Medium: 25-40%, Low: < 25%), and Knowledge Utilization Fl (High: > 45% Medium: 25-45%, Low: < 25%). Analyzing all 23 models, observe that they cluster into six distinct profiles based these metric combinations, with certain theoretically plausible patterns notably absent from the empirical data.
We present the laxonomy Table 3 The Powerful but Overconfident profile (GPT-5, 03-Pro, 03) achieves the highest pass rates (50-56%) through strong evidence gathering and synthesis, but exhibits dangerous overconfidence with refusal rates below 10% despite 21-30% knowledge insuf- ficiency. These models prefer hallucination over admission of uncertainty. In contrast, the well- calibrated Elite (Grok-4, Staude-Opus-4, achieves similar knowledge sufficiency and utilisation but maintains moderate refusal rates (29-38%). demonstrating that strong capabilities do not need to preclude epistemic awareness, although this calibration costs approximately 5-6% in pass rate.
The Synthesis Bottleneck profile reveals a critical failure mode: models like Qwen3-235B.
Thinking achieve high knowledge sufficiency (72.5%) but catastrophically fail at synthesis (< 25% utilisation). Despite possessing evidence, they cannot compose multi-hop reasoning chains, yet their low refusal rates indicate unawareness of this limitation. The Conservative Middle models Slaude-Sonnet-4-TIe
1.M--.5-Inner exhibit consistent mediocrity across all metrics, suggest- ing excessive caution- their moderate utilisation (31-44%) despite reasonable knowledge gathering (63-68%) indicates they refuse even when capable of answering.
8

---

At the lower performance tiers, observe striking divergence in self- Self-Aware of Weakness models (Doubao variants, Semini-2.5-Flash-TTT
appropriately refuse in 40-54% of cases, correctly recognising their limitations in both search and synthesis. Conversely, Weak and Confused models (o4-Mini, DeepSeek-R) exhibit similar capability limitations but fail to recognise them, attempting answers despite 16-18% utilisation rates.
Our analysis reveals three distinct failure modes in the multi-hop reasoning pipeline. Search Searchfailure affects 21-46% of attempts even in top models, indicating that evidence discovery remains chal- lenging. Synthesis failure is more severe-even with sufficient knowledge, utilisation rates peak at 56%, suggesting that composing multi--hpp reasoning chains remains fundamental bottleneck. Cal- ibration failure manifests bidirectionally: vop-performing models are systematically overconfident (refusing < 10% despite significant insufficiency), while weaker models may over-refuse or, worse, lack any calibration signal. Notably, no model in our evaluation achieves both high utilisation and high refusal-a perfectly calibrated model would excel synthesis while maintaining appropriate uncertainty, but current architectures appear to force tradeoff where strong synthesis capability invariably leads to overconfidence. This suggests a fundamental tension between competence and epistemic humility in existing architectures.
The emergence of these distinct profiles suggests that improving multi-hop reasoning requires tar- geted interventions. Models in the Synthesis Bottleneck category need architectural improvements to reasoning composition, not better search. Overconfident models need calibration mechanisms that don't 't sacrifice performance. Most importantly, the absence of any model achieving high perfor- mance across all three metrics-even Grok-4 and (laude-Opus44.1 1, the best-balanced models, only reaches 50.5% and 44.5% pass demonstemssss that robust multi-hop reasoning remains an open challenge, with synthesis capability being the universal limiting factor.
4.2.2 KNOWLEDGE DEGRADATION IN SYNTHESIS
Even when models achieve knowledge sufficiency (KS(d) = 1), they often fail to generate the correct answer. We call this knowledge degradation: evidence is present in context, yet models forget, ignore, or misuse it during synthesis. To analyse this effect, we focus on two diagnostics, Forget and Lead-astray, which reveal two distinct synthesis failures: models either fail to recall known knowledge (Forget) or become disrupted by noisy search contexts (Lead-astray). Appendix shows the formal definition.
Knowledge degradation patterns. From Table 2 models with lower Forget and Lead-astray rates consistently achieve stronger Knowledge Utilization, which in turn drives higher Generation Score and Pass @ I. High performers such Grok-4 (Forget 23.6 23.65%, Lead-astray 27.70%) and o3- Pro (Forget 21 13%,, Lead-astray 25.35% maintain the best Knowledge Utilization (56.19% and 49.40%), translating into the top Generation Scores (34.71% % and 20.86%) and competitive Pass @ (50.5% and 56.0%). GPT-5 shows a similar profile with very low Forget (17.72%) and strong Knowl- edge Utilization (49. (49.58%) In contrast, models with severe knowledge degradation see downstream metrics collapse. GPT-OSS- 20B, with Forget at 100%, attains only 10.73% Knowledge Utiliza- tion and 2.75% Generation Score: Swen3-30B-Thinkimg (Forget 79.65%) similarly drops 13.16% Knowledge Utilization and 7.25% Generation. Mid-tier models such as Gemini-2.. 2.5-Faas --aash--Think and Tongyi-DeepResearchh follow the same pattern. Overall, the results show that stable evidence retention not raw retrieval- is the key determinant of deep-search reasoning performance.
Forgetting dominates misdirection. Averaging across all models, the mean difference Forget Lead-astray is +10.5%% points. This gap indicates that, on WebDetective. failures after achieving knowledge sufficiency are more often due to not using synthesis) than to being led astray by spurious context. In other words, the principal bottleneck lies in evidence integration at answer time rather than in resisting distractors.
evidence (forgetting during
4.2.3 ROBUSTNESS TO TEST-TIME SCALING
To assess the robustness of our benchmark, we examine test-time scaling (TTS) along two axes. First, we scale context length for strong ReAct model (Claude-Opus-4.1) to test whether larger budgets improve performance. Second, we vary breadth and iteration counts in EvidenceLoop to probe whether extensive exploration exploit WebDetective. These analyses test whether

---

Claude Opus 4. Scaling (Context Length)
EvidenceLoop Scaling (Breadth-Reeation) Iterationn
Score 150
Searth Score
00-
Gamaration Score
Peseg:
20-
20000
30000
9303
53383
300002
120006
24
2-1
32
Contect Length
Configuration (breadit-itertior)
Figure 2: Test-time scaling (TTS) on WebDetective. Left: Increasing Claude-Opus-4.1 I's con- text length boosts Search Score and Pass@ 1 at shorter ranges, but both plateau beyond 32k to- kens, while Generation Score shows only modest gains. Right: EvidenceLoop remains stable across breadth-iteration settings, with Pass@ improving under moderate configurations (c.g., 1-2, 3-2). Generation Score changes little, highlighting synthesis- not search as the main bottleneck under TTS.
WebDetective can be artificially boosted by TTS or instead faithfully reflect underlying system capabilities.
enlarging the context window from to 32K tokens brings negligible gains: Generation Score plateaus at about 34%, Pass about 50%, and Search Score increases by less than 1 1%. For EvvdenceLLoog expanding the controller from breadth=1, iteration=2 to breadth=3, iteration=2 raises Search Score slightly (45% -> 46%, +1%), leaves Generation Score unchanged at 21%, and improves Pass @ I from 49% to 56% (+7%). These results indicate that our benchmark is robust to naïve test-time scaling. Neither larger context budgets shallow exploration suffice to "hack" WebDetective; achieving further gains requires genuine advances in model reasoning and knowledge utilisation.
In Figure 2 we observe two main trends. For Claude-Opus
5 CONCLUSION
We introduce WEBDETECTIVE, a benchmark for evaluating web agents on hint-free multi-hop deep search within controlled Wikipedia sandbox. Unlike prior datasets that embed reasoning paths (PH) or entity fingerprints (SH), our design enforces autonomous discovery of reasoning chains while enabling fine-grained attribution of failure modes. Crucially, while the sandbox climinates corpus-based shortcuts, it accommodates multiple valid reasoning strategies- bine partial search with parametric knowledge, and our metrics credit any path achieving knowl- edge sufficiency. Evaluation of 25 state-of-the-an models reveals consistent weaknesses: systems often retrieve sufficient evidence but fail utilise it effectively, and appropriate refusals remain nearly absent. Our proposed agentic workflow EvidenceLoop. designed to target the specific failure modes our diagnosties reveal, demonstrates that explicit verification and systematic evi- dence tracking can partially close this gap inpe-oniii Knowledge Utilization by 53% over its base pondel-underscoring that performance cannot be trivially improved by test-time scaling alone. The co-design methodology generalises beyond Wikipedia to any domain with factual content, entity graphs, and masking capability, establishing WebDetective as a framework for developing genuinely autonomous reasoning systems.
may com-
10

---

ETHICS STATEMENT
We have read and will adhere to the ICLR Code of Ethics and the ICLR Code of Conduct. Our research introduces WebDetective, framework for hint-free multi-hop question answering and ev- idenceLoop, agentic workflow baseline. The methods used in our study well-established for academic research. These environments do not contain any personally identifiable information (PII) sensitive real-world data. Our work did not involve human subjects, crowd-sourcing, the scraping of private data; therefore, Institutional Review Board (IRB) approval was not required.
We acknowledge that research autonomous agents carries potential dual-use risks. To mitigate these, our experiments are intentionally confined to benign, closed-world tasks such as online shop- ping and household activities within simulated settings. We followed good scholarly practice by reporting our methods and results transparently and citing prior work accurately. The authors de- clare no competing interests or external sponsorships that could have influenced the outcomes of this research.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. All essential details for repro- ducing results provided within this paper. The WebDetective benchmark design, includ- ing the hint-free question formulation principles, the co-designed evaluation system with selec- tive entity masking, and the two-level diagnostic evaluation framework, is thoroughly detailed in the methodology sections. The complete WebDetective dataset statistics, design methodology, and human validation procedures are comprehensively described in the dataset construction sections. Our experimental setup, including the specific language models evaluated (GPT-5, O3-Pro, Claude-Opus-4.1, Gemini-2.5-Pro. Grok-4, @wen3-235BB-hinndee the controlled Wikipedia sandbox configuration, knowledge sufficiency probing methodology, and evaluation protocols, fully documented in the experimental sections. The diagnostic metrics for- malization, including Knowledge Score, Generation Score, Good Refusal (GR), Knowledge Utili- sation (KU), and knowledge degradation tests (Forget and Lead-astray) are rigorously defined the evaluation framework sections. To facilitate full replication of our benchmark construction pipeline and agent evaluation experiments, we will release our complete codebase, the controlled Wikipedia sandbox environment, a hint-free question dataset with evidence chains, and evaluation scripts as supplementary material.
co-
and others),
ACKNOWLEDGMENT
This work was supported in part by A*STAR SERC CRF funding to C.T. This work is also sup- ported by the National Research Foundation, Singapore, under its National Large Language Models Funding Initiative (AISG Award No: A1SG-AMIP-20044-0055), NTU SUG project #025628- 00001.
REFERENCES
Anthropic. Introducing claude 4. httta://www.aatrrrpp...c///e/eep 2025. Accessed: 2025.
May
ByteDance Seed Team. Doubao 1.6 thinking and flash models. https:/ /tapp::///ee..... seed. bytedance, com/ 2025. Part of the Doubao/Sced model family. Accessed: 2025.
Jifan Chen, Shih-ting Lin, and Greg Durrett. Multi-hop question answering via reasoning chains. arXiv:1910.026110 2019.
DeepSeek-AI, Wenfeng Liang, al. arXiv:2412.19437. 2024.
Deepseck-v3 technical report.
arXiv preprint
DeepSeek-AL, Wenfeng Liang, et al. Deepscek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.

---

Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang. and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents, 2025. URL https::/aaxii..ott
Jinyuan Fang, Zaiqiao Meng, and Craig MacDonald. Trace the evidence: Constructing knowledge- grounded reasoning chains for retrieval-augmented generation. In EMNLP Findings, 2024.
Mohamed Amine Ferrag, Norbert Tihanyi, and Merouane Debbah. From 1lm reasoning to au- tonomous ai agents: A comprehensive review, 2025. URL htttp::/eexxv.orr///00
2504.19678
Google DeepMind. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. hnttpe://aaxx/v.orr///aa/////// 06261, March 2025. Accessed: 2025.
Jie He et al. Mintqa: A multi- question answering benchmark for evaluating Iims on new and tail knowledge. arXiv:2412./7032, 2024.
Xanh Ho et al. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. In COLING, 2020.
Soycong Jeong et al. Adaptive-rag: Learning to adapt retrieval -arevalld large language models through question complexity. In NAACI, 2024.
Pengcheng Jiang, Xueqiang Xu, Jiacheng Lin, Jinfeng Xiao, Zifeng Wang, Jimeng Sun, and Jiawei Han. s3: You don't 't need that much data to train a search agent via rl, 2025. URL https:
//arxiv.org/ abs/ 2500. 14146
Carlos E Jimenez et al. Swe-bench: Can language models resolve real-world github issues? In ICLR, 2024.
Tom Kwiatkowski et al. Natural questions: A benchmark for question answering research. TACL, 2019.
Kuan Li, Zhongwang Zhang. Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baix- uan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang. Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor: Navigating super- human reasoning for web agent, 2025a.
http///axvvvv/r////e/ abs/ 02592
Ronghan Li et al. Different paths to the same destination: Diversifying llms generation for multi-hop open-domain question answering. Knowledge-Baad Systems, 309, 2024a.
Ruosen Li, Zimu Wang, Son Quoc Tran, Lei Xia, and Xinya Du. MEQA: A benchmark for multi-hop event-centric question answering with explanations. In The Thirty-eight Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Truck, 2024b. URL
http://ppaaaarreeeeeeeeeemoooota net. wtwee//oooommmmdnn//aaaaea
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and
Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models, 2025b. URL
https://aaat wwwaariiv..gg/aaaee abs/ abs//501..5366
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and
Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability, 2025c. URL
Zijian Li et al. Graph neural network enhanced retrieval for question answering of llms.
arXiv:2406.0 25t::2400600577. 2024c.
Hao Liu al. Hoprag: Multi-hop reasoning for logic-aware retrieval-augmeentdd generation.
arXiv:2502. 12442, 2025.
Xiao Liu et al. Agentbench: Evaluating llms as agents. arXiv:2308.03668. 2023.
Moonshot AI. Kimi k2: Open agentic intelligence technical report. /tos://gitee htap:://ggthin com/
July 2025. Accessed: 2025.
12

---

Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted juestiinnansserte with human feedback. arXiv preprint a1X1:2211..0333222000 2021.
Cheng Niu et al. Ragtruth: A hallucination corpus for developing guardrails in rag systems. arXiv preprint, 2024.
Agada Joseph Oche, Ademola Glory Folashade, Tirthankar Ghosal, and Arpan Biswas. A systematic review of key retrieval-auggmentdd generation (rag) systems: Progress, gaps, and future directions, 2025. URL /ttttpss///arxiv.or///////////// /
18910
OpenAl. Introducing swe-bench verified, 2024.
OpenAl. Introducing htttaa/s/.ppeeaa.cc/////e/eeertr Au- gust 2025a. Accessed: 2025.
OpenAl. apt-oss--20b & gpt-oss-20b model card. arXiv preprint arXiv:2508. 2025b.
OpenAl.
Openai
o3 and 04-mini system
card.
https: //cdn.
openai com coenai.com/pdee http:/dpff2222cc1/// ooemaa/.com//of///22///// 02dc
-3-and- 04 04- 04mod-o4-mii--- mini -im-mmiiinisstemm-e--m-m-- card. April 2025c.
Accessed: 2025.
OpenAl.
Introducing
openai 04-mini.
bttps://opende com/indxx/
Lntroducing--3-aaa -0n-----mad----mimii April 2025d. Accessed: 2025.
Fabio Petroni et al. Kilt: A benchmark for knowledge intensive language tasks. In NAACL, 2021.
Es Shahul et al. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint, 2024.
Aditi Singh et al. Agentic retrieval-augmented generation:
A survey on agentic rag.
Stt1:25001.91366 2025.
Hao Sun, Zile Qiao, Jiayan Guo. Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of 1lms without searching, 2025. URL httph:ttpapariiii org/ abs hasp//bb/2255.... 04588
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Web- shaper: Agentically data synthesizing via information-seeking formalization, 2025. URL
Tongyi DeepResearch Team. Tongyi deepresearch: The leading open-source deep research agent. NLP Wtta--//eee/eeeeeeerettee 2025. 30B-parameter agentic model for long-horizon research tasks.
https:///iithin ctw.ttttoooooooooooooooommooo A1
Harsh Trivedi et al. Musique: Multihop questions via single-hop question composition. TACL, 2022.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.125516 2025.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. Constructing datasets for multi-hop read- ing comprehension across documents. In TACL, 2018.
Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao. Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, Wenhao Huang, Yang Wang, and Ke Wang. Widesearch: Benchmarking agentic broad info-seeking, 2025. htt.tppp///aaiivoooog//////////p 07999
xAI. Grok 4: The most intelligent model in the world. https://x.ai/nne 2025. Accessed: 2025.
July
13

---

Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang. Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking Ilm agents on consequential real world tasks, 2025.
http:///wwwivv...aaaao////o 14161
An Yang, Binyuan Hui, al. Qwen3 technical report. arXiv preprint arXiv:2505.09385. 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio. William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018a. URLp:tts:://aat
httt:p:://axxiv.or//aaaa /1009.00... 110190960000000000
Zhilin Yang et al. Hotpotqa: dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018b.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https::/aaxxis wtg/abs/
2406. 12045
Zhipu Al Team. Glm-4.5: Reasoning, coding, and agentic abilities. August 2025. Technical report available at https://.ai/bloggll-4.55
Shuyan Zhou et al. Webarena: A realistic web environment for building autonomous agents.
arXiv:2307. 13854, 2023.
Andrew Zhu, Alyssa Hwang, Liam Dugan, and Chris Callison-Burch. Fanoutqa: A multi-hop, multi-document question answering benchmark for large language models. ACL, 2024.
14

---

Appendix
Table of Contents
A Formal Metries Definition
16
B Dataset Construction
19
Dataset Statistics
20
Statistical Considerations
21
E
The EvidenceLoop framework
21
E.1
Core Architecture: lterative Refinement with Fallback
22
E.2
Evidence Memory System
22
E.3
Verification: Ensuring Evidence-Grounded Reasoning
................................
23
Ablations on Architecture Design
................................
23
Generalization Across Models
F
Failure Case Studies
24
G
Related Work
25
G. 1
Multi-Hop Question Answering Benchmarks
25
Setrivall-uugentt Generation and Agents
................................
25
Evaluation Frameworks
25
LLMs Usage
26
15

---

A FORMAL METRICS DEFINITION
Traditional multi-hop QA evaluation reduces agent performance to a single pass rate, obscuring the diverse failure modes that occur in complex reasoning tasks. An agent that searches exhaustively but fails to synthesise evidence exhibits fundamentally different limitations than one that refuses pre- maturely or hallucinates from parametric knowledge. Our co-designed sandbox, with its guaranteed unique reasoning paths, enables unprecedented diagnostic precision in distinguishing these failure modes.
We introduce a two-level evaluation framework that separates knowledge sufficiency from generation quality. First, we assess whether an agent possesses the requisite knowledge- either through suc- cessful search parametric memory answer the question. Second, conditioned on knowledge sufficiency, we evaluate the agent's ss ability to either correctly synthesise an answer or appropriately refuse when evidence is insufficient.
Knowledge Sufficiency Assessment: We assess whether an agent possesses- either through search parametric knowledge- all information necessary to answer the question. Given the required evidence chain & = (e1,..., ),), we first identify which evidence the agent discovered through search by tracking visited pages in sandbox. For any missing evidence 6 $ found, we then test whether the agent can access this information parametrically.
Specifically, for each missing piece of evidence e,, we construct focused probe query p; that tests for that specific knowledge. For instance, if the agent never visited Kane's page and thus missed discovering that "Kane Cornes has brother Chad Cornes," we probe with: "Kane Cornes has brother We define Probe(p:) as a function that submits probe to the base model and returns whether the model's ss response matches the expected answer for evidence e,
For instance d with evidence chain of length nd. we define:
{
if ei Efound (found via search)
{
k:
if e $ &fonnd A Probe(p:) = correct otherwise
The instance is knowledge sufficient if: (d) II
We define the overall Knowledge Score as the fraction of instances where the agent achieves knowl- edge sufficiency:
S
KnowledgeScore
(I)
N
This metric directly measures search effectiveness a low KnowledgeScore indicates the agent fails to discover necessary evidence through exploration, regardless of its ability to synthesize answers.
Search Score: While our masking mechanism enforces the canonical reasoning path vo 01 + Un. we observe that models may leverage alternative valid reasoning strategies. Specifically, if an entity U, (reachable through search from vp) has a meaningful relationship to the answer Un stored in the model's 's parametric knowledge. the model can combine partial search with memory to reach the correct answer. This represents a legitimate form of reasoning that demonstrates efficient use of both search and parametric knowledge.
To capture this capability, we define SearchScore that credits models for finding correct answers through any valid combination of search and parametric knowledge, provided their search efficiency meets or exceeds the ground truth path:
SearchScore = KnowledgeScore CC
N
where C = (d D correct(d) searched(d) A hops(d) hops A KS(d) 0) represents instances where the model:
Produces the correct answer despite not having complete knowledge sufficiency through the canonical path
16

---

Actually performs web search (not relying solely parametric knowledge)
Uses no more search hops than the ground truth path length
Effectively combines discovered entities with parametric knowledge
The requirement that searched d) = true ensures we only reward genuine search-memory combina- tion strategies, not pure parametric recall. This metric recognizes that effective multi-hop reasoning isn't solely about following predetermined paths, but about efficiently discovering and leveraging available information through intelligent combination of partial search with existing knowledge. The hop constraint ensures models aren't it simply performing exhaustive search, but are discovering meaningful connections that enable efficient reasoning.
Generation Quality Assesement: Given the knowledge sufficiency assessment, we evaluate gen- eration quality through conditional framework that captures the fundamental tension multi-hop QA: agents must synthesize answers when they have sufficient evidence while appropriately refusing when they don't.
Let D = {d1,...,dy dN) denote the evaluation dataset with N instances. We partition 'D along two dimensions:
Knowledge dimension:
S = {d D: KS(()) =} (knowledge sufficient instances)
(3)
I D D (knowledge insufficient instances)
Response dimension:
A = {dl E D agent attempts answer)
(5)
N {d E D agent refuses)
(6)
where attempts further partitioned into = A.u.A. with A. denoting correct answers (match- ing ground truth) and Aw denoting wrong answers. Note that AUN= D.
The intersection of these dimensions creates critical regions that reveal different agent capabilities and failure modes:
Knowledge sufficient, answers correctly (SnAA)): The ideal scenario the agent possesses all evidence and successfully synthesizes the correct answer. This demonstrates knowledge utilization, the ability to compose multi-hop hop reasoning without forgetting intermediate steps or being disrupted by irrelevant information.
Knowledge sufficient, answers wrongly (S n:w)): synthesis failure failure-despite having all necessary evidence, the agent produces an incorrect answer. This reveals breakdowns in reasoning composition, where evidence possession doesn't 't translate to correct synthesis.
Knowledge sufficient, refuses (SNW): Over-caution the agent has sufficient evidence but refuses to answer. This represents failure to recognize that the evidence chain is complete, missing opportunities to provide helpful answers.
Knowledge insufficient, refuses (I NW: The second ideal scenario scenario-good refusal. The agent lacks critical evidence and appropriately declines to answer, demonstrating epistemic awareness and avoiding hallucination.
Knowledge insufficient, attempts answer (I A): The most problematic behavior the agent lacks evidence but attempts an answer anyway (whether correct by luck or wrong), typically through hallucination, guessing, or over-reliance on partial information.
This visualization reveals that generation quality isn't monolithic agent might excel at refusing when uncertain but fail to synthesize known information, vice versa. For instance, overly con- servative agent might achieve perfect good refusal by refusing everything (large A region), while an overly confident agent might attempt every question (large A region) leading to frequent halluci- nations in the zone.
To capture these complementary capabilities, we define two core metrics:
17

---

Good Refusal (GR) measures the agent's ability appropriately abstain when lacking evidence. It evaluates N overlap with Z- high recall indicates the agent successfully avoids hallucination by refusing most knowledggeinsufficiennt bleeding unnecessarily into S).
insufficient cases, while high precision ensures refusals are justified (not
[ NI П
Wnz [
RecalloR PrecisionGR
RecallGR
PrecisionGR
FLGR
RecallGR PrecisionGR
Knowledge Utilization (KU) assesses the agent's ability to synthesize correct answers when evi- dence is available. It examines Ag within S-high recall means the agent leverages available ev- idence effectively, while high precision indicates that attempts are typically grounded in sufficient knowledge.
|Ae ns! |S|
Precisiongu A NS [A
FIKu 2 Recallku: Precisionku
Recallku
(8)
Recallku + Precisiongu
Importantly, these metries are non-competing well-designed system. An ideal agent achieves high FI (refusing when and only when knowledge insufficient) while maintaining high Fu (correctly answering when evidence is available). To capture both capabilities while preventing gaming, we define unified Generation Score:
one shouldn't 't decrease the other in
GenScore = FLGR + FlKU
S
N
The |S|/N weighting (KnowledgeScore) is crucial for preventing metric exploitation: without it, models could game the evaluation by adopting a degenerate ceategy--erforree minimal search and refusing nearly questions. Such a model would achieve high FIGR (correctly refusing the many knowledge-inee sesufdgg-inuuffciien cases) while contributing nothing usefuul yet still obtain a substantial GenScore. This creates perverse incentive where models might optimize for conservative refusal rather than improving search capabilities. The weighting ensures that models cannot exploit the evaluation structure-thr must demonstrate effective evidence discovery to achieve competitive scores, aligning the metric incentives with the actual goal of multi-hop reasoning systems.
Unlike simple pass rates, our metrics provide actionable insights: low KnowledgeScore indicates inadequate search strategies, low GR scores reveal over-confident hallucination, and low KU scores expose synthesis failures despite having evidence. This diagnostic precision, enabled by our co- designed evaluation environment, illuminates the specific capabilities required for robust multi-hop reasoning.
Knowledge Forget Test. We LM(q, Eround) where &ounnd Evisited n EGT represents evidence from ground-truth URLs that the model actually visited. When this fails despite KS(d) = 1, it reveals knowledge forget: the model cannot leverage its parametric knowledge to fill missing pieces when answering the full question, even though it correctly answers individual probes Probe(p:) for each missing evidence e; E Ecr \ Etound-
Lead-astray Test. When LM(q, Eronnd) succeeds but the model fails in its actual search trajectory, we identify lead-astray: the model can synthesize the answer from clean evidence but disrupted by the accumulated search context (failed attempts, irrelevant pages, exploration noise).
Formally, for the set of knowledge-sufficient instances (d D KS(d) incorrect(d)) where the model fails despite having all necessary knowledge:
ForgetRate (d E S* LLM(ga, Eddmna) * a;}l
18*
LeadAstrayRate S LM( Ed El a A actual output(() * ajHl
S*
These metrics decompose knowledge-suffiicen failures: ForgetRate identifies when models not integrate parametric knowledge with partial search results, while LeadAstrayRate reveals when noisy search trajectories corrupt otherwise successful reasoning.
18

---

B DATASET CONSTRUCTION
To instantiate our hint-free multi-hop QA benchmark, develop systematic pipeline that trans- forms single-hop Wikipedia QA pairs into verified multi-hop reasoning chains while ensuring each hop is necessary for answering.
Source Data and Chain Discovery. We begin with a corpus of Wikipedia-based QA pairs where each question targets specific paragraph a Wikipedia page (the starting entity v)) and has an answer that is another Wikipedia entity (e,). These seed questions are designed to be unambiguous and simple, avoiding any linguistic hints about reasoning paths. To construct multi-hop chains, we first block the direct connection between and Un. then perform breadth-firs search (BFS) to find the shortest alternative path up > U1 + - Un through Wikipedia's hyperlink graph. For each edge (va, U++1 1) in the discovered path, we extract the sentence e; from vy Wikipedia page that contains the hyperlink to 1,+1, forming the evidence chain {er, 62, ......... )
Verification of Reasoning Necessity. Not all discovered paths constitute valid answers to the question most arbitrary paths from to to Un through Wikipedia's link graph are completely un- related to what the question asks. For instance, path connecting two people through their uni- versities and shared colleagues is irrelevant for a question asking about family relationships. We implement three-stage verification process using strong language model (Qwen-3-2355 in implementation). denoted as LM(-) which takes text input and generates an answer:
1. Parametric Inaccessibility: We verify that LM(4) * Un ensuring the answer cannot be directly retrieved from the model's parametric memory without evidence.
2. Evidence Sufficiency: We confirm that LM(g, E) = Un. validating that the complete evi- dence chain enables correct answer generation.
3. Evidence Necessity: For each evidence piece e,, we verify that LM(g, \ {ei}) & Un- ensuring every hop in the chain is required for reasoning. This ablation test eliminates questions where evidence pieces are redundant where shortcuts exist.
Human Validation and Dataset Statistics. Questions passing automated verification undergo hu- man annotation by researchers with NL.P expertise. Each question is independently reviewed following structured protocol:
1. A Annotation Protocol: For each question, annotators receive the question evidence chain E fei..., Cn and answer 2/2- They verify three criteria:
Necessity: Whether the question can be answered without the evidence chain using only general knowledge
Sufficiency: Whether the evidence chain logically derives the answer without requiring external information
No hints: Whether the question avoids linguistic cues that reveal intermediate reason- ing steps
2. Validation Process: Each question requires 2-3 minutes of review. Annotators trace through the reasoning chain step-by-step, attempting to answer the question both with and without the evidence to ensure all pieces are necessary. Questions where intermediate en- tities could be guessed from the question phrasing or where the evidence chain has logical gaps are rejected.
3. Dataset Filtering: Of approximately 450 veriffee questions reviewed, 200 ques- tions (~44%) pass human validation. Common rejection reasons include: evidence chains not targeting the questions, evidence chains with missing logical connections, and questions containing subtle hints about the reasoning path (e.g., mentioning attributes that implicitly identify intermediate entities).
This manual verification process, totaling approximately 20 hours of annotation effort, ensures final dataset contains only questions that genuinely require discovering and composing the complete multi-hop reasoning chain.
19

---

mask wiki 200 Dataset Analysis
Quistion Type Uxstribunion
Endiy Count pen Question Distributiion
Evidence Count Distri ucon
122-
1005
68-
60-
34.0%
13.96
23
20
Tgats
Othyr
2
Queettin and Ansase Lenghn Cindr
Hop Eitebbttion
Swarch Usedd Court Dittritution
97-
quaeticc uerge
Reswer Leugth
120
9)-
100
03
00
CD-
23
13
20-
120
300
400
Charsctor Count
Hego by analysla analgolsstting setting
Marber of Quorits Laod
Figure 3: Dataset statistics for WebDetective benchmark. The figure shows: (a) Distribution of question types, (b) Number of entities per question, (c) Evidence count distribution, (d) Question and answer length in characters, (e) Hop length distribution by analysis setting, and (f) Search query usage patterns. The dataset exhibits controlled complexity with predominantly 2-3 hop questions while maintaining challenging longer chains.
C DATASET STATISTICS
Our final WebDetective benchmark comprises 200 hint-free multi-hop questions, carefully curated through our verification pipeline. Figure presents a comprehensive analysis of the dataset charac- teristics.
Question Complexity. The dataset exhibits controlled complexity suitable for diagnostic evalua- tion. Questions require 2 to 6 hops of reasoning (mean: 2.85 hops), with the distribution heavily weighted toward 2-hop (55%) and 3-hop (31%) questions, while maintaining a challenging subset of 4+ hop questions (14%). This distribution balances tractability with sufficient complexity to stress- test multi-hop reasoning capabilities. Each question involves 3 to 8 Wikipedia entities (mean: 3.73), with the modal question requiring exactly entities to form the complete reasoning chain.
Question Types and Domains. The dataset spans diverse question types, with "What" questions comprising 34% of the dataset, "Who" questions 27%, and "Which" questions 13.5%, ensuring broad coverage of information-seeking patterns. Questions concise (mean: 71.4 characters) with typically short answers (mean: 28.6 characters), reflecting natural information needs without verbose specifications that might hint reasoning paths.
Evidence Requirements. The evidence distribution aligns with hop counts, with most questions re- quiring 2-3 pieces of evidence (52.5% and 31 % respectively). This controlled evidence requirement
20

---

enables precise diagnosis of where reasoning fails whether at initial discovery, intermediate steps, or final synthesis.
The dataset's careful balance of complexity, diversity, and diagnostic precision makes it suitable for evaluating the full speetrum of multi-hop reasoning capabilities, from basic 2-hop familial re- lationships to complex 5-hop chains requiring sustained context retention across multiple search iterations.
Domain Coverage. The dataset spans diverse topical domains to ensure broad evaluation coverage, as shown in Table 4 This distribution ensures WebDetective is not skewed toward any single re- lation type (such familial ties) but instead covers reasoning patterns essential for comprehensive evaluation.
Table 4: Domain distribution in the WebDetective benchmark.
Domain
Percentage
Film, Television Theatre
20.0%
History, Politics & Military
18.0%
Geography, Locations & Landmarks
16.0%
Music & Audio Production
13.5%
Sports & Competitions
11.5%
Biography, Genealogy & Education
8.5%
Literature, Comics & Pop Culture
6.5%
Science, Technology & Transport
6.0%
D STATISTICAL CONSIDERATIONS
Quality-Focuset Construction Pipeline. The relatively modest dataset size (200 questions) is direct consequence of our strict construction and filtering pipeline. For each candidate question- answer pair (q, a") with evidence chain & {ei,..., J, we enforce three conditions: (1) Para- metric Inaccessibility: the verifier LM cannot produce a" from alone; (2) Evidence Sufficiency: LM(g, E- = a and (3) Evidence Necessity: for every i {i...., n), we probe with the incomplete set (g,E)) E {e:}) and verify failure. The majority of automatically generated candidates are discarded during this rigorous process, followed by manual verification.
Confidence Intervals. Our binomial confidence analysis shows that with n 200, we achieve 95% confidence interval of +6.9% (assuming 50% pass rate). While this sufficient for identifying the large-scale failure patterns our benchmark targets-the gap between Knowledge Score (79%) and Generation Score (23%) for top models, the near-universal failure of refusal capabilities (< 30% Good Refusal FI), and the synthesis bottleneck (Knowledge Utilization peaks substantial effects that do not require larger samples to detect reliably.
Diagnostic Purpose. Our primary aim with WebDetective is to provide a diagnostic benchmark that exposes fundamental failure modes rather than ranking systems with marginal differences. The patterns we identify consistent across all 25 evaluated systems, demonstrating that even with our 2-3 hop concentration (86% of questions). the benchmark remains far from saturated: the highest Pass @ 1 is only 56% (o3-Pro), and state-of-the-art agents frequently fail synthesize 2-3 pieces of evidence without hints.
E THE EVIDENCELOOP FRAMEWORK
The hint-free nature of our benchmark exposes fundamental limitations in current multi-hop reason- ing approaches. Without linguistic scaffolding, agents must autonomously discover which connec- tions matter among thousands of facts challenge that, our results show, causes even state-of- the-art models to achieve only 50% accuracy. To better understand these challenges and establish a baseline for future work, design agentic workflow that explicitly targets the unique difficulties our benchmark reveals: the need for broad exploration without context explosion, evidence retention across long trajectories, and synthesis from accumulated but noisy search contexts.
21

---

E.1 CORE ARCHITECTURE: ITERATIVE REFINEMENT WITH FALLBACK
Our framework attempts to balance exploration breadth with computational feasibility through Rmax iterations. Illustrated in Figure each iteration launches N parallel solver agents (Ai, A'r that explore different reasoning paths simultaneously. Each agent A receives the question q. an aggregated context C" from previous iterations (with C initially), and executes up B actions.
After each iteration, we employ a two-stage refinement process:
1. An extraction agent processes the reasoning contexts from all N parallel agents to identify key findings, evidence references, and promising paths
2. An aggregation agent synthesizes these extracted insights into refined context C*+1 for the next round, preserving valuable discoveries while discarding exploration noise
This iterative refinement addresses a core challenge our benchmark exposes: early rounds might explore many directions- sports connections, geographic locations, family relations but the extraction-aggrreationn pipeline identifies which paths warrant deeper exploration, preventing the context explosion that causes single-pass approaches to fail while avoiding premature path com- mitment. no conclusive answer emerges after Rmax iterations, final aggregation agent consol- idates all discovered evidence into comprehensive context Cftinal, This context is then passed to a synthesis-oni solver that attempts to derive the answer purely from the accumulated evidence without additional search tetions-effectivllyy testing whether the failure stems from insufficient exploration or poor evidence composition.
Question
Verifier Agent
Itentton
Question: father Hew Gomen?
abowe curation with zocks
Clain::
Think thek about who Kona Conner...
Menway
Chadts nuthor. who (D
Teut call Search Mare Camnes)
Please vertly above
Apent
Agent
Gewrthieeeemingg Tradss
Tool Rroporms Kara mos (Monsory KDh mnj
Nutrova (leemory
Extractor Agent
Extractor Agent
Propese:
Arover clisass Comes Brahan Comes
(Wemary
Kary Connal has truiner... (Mhmnory 1D: (ij
Search Summmrtes Experinces
Verfication
Verbbattos Resutt
Aggregatior Agent
Comsont
Figure 4: Overview of the EvidenceLoop framework. The system employs parallel solver and extractor agents that perform search and reasoning to generate proposals with supporting claims. These proposals then verified through memory retrieval, with the aggregated context fed into subsequent iterations until verification succeeds or a termination condition is met.
E.2 EVIDENCE MEMORY SYSTEM
Enabling this iterative refinement is our Evidence Memory System M. When any agent performs search visits page, the system: 1) Stores complete results in persistent memory; 2) Assigns unique Evidence IDs (EIDs) for reference; and 3) Returns both full content and EID to the agent.
The EID system serves multiple critical functions in our framework. First, during extraction and aggregation between iterations, the extraction and aggregation agent produces summaries that pre- serve EID references alongside extracted facts-for for example, "Kane has brother Chad EEID0022, Chad's stepmother is Nicole [EID-089]*. This allows subsequent solver agents to receive concise, actionable summaries while retaining the ability retrieve full evidence demand through the
22

---

retrieve action external tool, which takes an EID and returns the complete original content from memory. Second, these EIDs enable systematic verification (detailed in Appendix E.3 where verification agents trace claims back to their original sources and validate reasoning chains against the actual evidence.
The memory system transforms how evidence flows through iterations. Rather than forcing agents work with either overwhelming full documents or lossy compressions, agents can work with focused summaries while maintaining access complete evidence through EID-based retrieval. This design ensures that even as contexts become more refined across iterations, agents never lose access to the complete evidence trail that supports their reasoning. allowing them dive deep into specific evidence when needed for detailed analysis or verification.
E.3 VERIFICATION: ENSURING EVIDENCE-GROUNDEE REASONING
The verification mechanism prevents premature hallucinated answers from propagating through our system. When any solver agent Aţ proposes answer, it must decompose the answer into atomic claims {cr, C,, Cm where each claim e explicitly linked to an EID from the memory system c.g.. "Kane has brother Chad [EID-042]". No unsupported claims are permitted.
The verification agent V evaluates each proposal:
Vgg. answer, (cy, { + (YES, NO(fedhack))
For each claim-evidence pair, the verifier retrieves the full content from M via the EID and validates: (I) whether the source genuinely entails the claimed fact, (2) whether the claims collectively derive the answer, and (3) whether the answer correctly addresses the original question.
Verification occurs during solver execution. Rejections provide specific feedback back to the solver, allowing immediate gap-filling within the remaining action budget B. while acceptance imme- diately terminates all iterations. This ensures both evidence grounding and computational effi- ciency-solvers can correct incomplete reasoning in real-time while avoiding unnecessary explo- ration once the answer is verified.
E.4 ABLATIONS ON ARCHITECTURE DESIGN
To quantify the contributions the memory and verification components, we conduct ablation stud- ies using DeepSeek-RL. -R--2k-----52 the base model. As shown in Table memory-driven working- context structuring is the main force behind alleviating knowledge-related deficiencies, while ver- ification plays complementary role by strengthening precision and ensuring robust utilization of retrieved knowledge.
Impact of Memory. Removing the structured evidence buffer (w/o Memory) leads the most substantial performance drop, with the Search Score decreasing by 8.5% (71.5% - 63.0%). This underscores that memory-driven working-context structuring is the primary contributor mitigating knowledge-related shortages and enabling effective knowledge discovery.
Impact of Verification. Eliminating the explicit verification loop (w/o Verification) results moderate but consistent reduction in both Search Score (-2.9%) and Knowledge Utilization F! (-1.54%). This indicates that verification provides valuable additional support. improving retrieval quality and enhancing the stability of information synthesis.
Table 5: Ablation study on EvidenceLoop Search Score and Knowledge Utilization Fl are re- ported, with emphasis on the memory and verification modules, identified earlier key failure points.
Model Variant
Search Score (%)
Knowledge Util. F1 (%)
EvidenceL.oog Loop (DeepSeek-RP) -0528)
71.50
25.00
wfo Memory
63.00
24.38
w/o Verification
23.46
23

---

Table 6: Performance comparison across different base models. EvidenceLoop consistently out- performs the standard baselines on DeepSeek SaapSeee--1-O000 1-----1-0222 and GLM-4 4----455-ir particularly in Knowledge Utilization and overall Pass@) accuracy. Relative improvements are indicated in paren- theses.
Model & Method
Knol. Suff
Search Sc.
Knol. Util F1
Pass@1
Backbone: DeepSeck-RI-0528
Baseline
61.0
65.5
16.36
20.5
EvidenceLoop
68.0
71.5
25.00
28,5
A
+7.0
+6.0
+53% (rel.)
+39% (rel.)
Backbone: GLM-4.5-Air
Baseline
55.5
60.5
17.97
19.0
EvidenceLoop
62.0
63.5
19.70
23.5
A +6.5
+3.0
+10% (rel.)
+24% (rel.)
E.5 GENERALIZATION ACROSS BASE MODELS
To assess whether the benefits of EvidenceLoop rely on specific model characteristics or gen- eralize across different backbones, extended our evaluation to two distinct architectures: DeepSeek-R1-0522 and GLM-4. 5-Air. -0-4-. This analysis specifically tests the framework's 's ro- bustness against context degradation and limited search coverage inherent in standard ReAct-style baselines.
EvidenceL delivers consistent improvements in knowledge discovery and synthesis across both architectures. Specifically, we observe:
As detailed in Table 6
Search Coverage: Search Scores improve by +3.0 to +6.0 points, indicating more ef- fective exploration of the information space.
Knowledge Utilization: The structured evidence buffer and verification loop yield sub- stantial gains utilization, with relative improvements in FL Score ranging from 10% to
53%.
Task Performance: These gains translate directly downstream accuracy, boosting Pass@1 1 by +4.5 to +8.0 points absolute (+24% to +39% relative).
Most notably, the DeepSeek-R1-0522 backbone achieves a 53% relative improvement in Knowl- edge Utilization F1 (16.36% + 25.0%). This result confirms that EvidenceL.oo effectively miti- gates the synthesis bottleneck identified by WebDetective, transforming retrieved raw context into actionable evidence through its iterative explore-verify mechanism.
FAILURE CASE STUDIES
We identify four recurring failure patterns through qualitative analysis:
Premature Search Termination: After small amount of failed searches, models enter a "learned helplessness" state where they immediately conclude the answer is unavailable and refuse further exploration. Even explicit prompting like "please continue searching" fails to restart the search process, with models insisting no more information exists in the sources. This occurs despite obvious next steps being available- instance, finding Chad Kanc's brother but refusing to visit Chad's page directly, instead declaring the search futile.
Context-Induced Instruction Degradation: As search context accumulates, models pro- gressively lose their ability to follow basic instructions. In shorter context, they proper proper <answer> tags and structured reasoning, but gradually degrade dropping tags intermittently. then completely abandoning format requirements, producing stream- of-consciousness text without capitalization proper syntax.
24

---

Models lose track of discoveries across search iterations, repeatedly searching for entities they ve already found or failing to maintain entity rela- tionships discovered earlier. They cannot distinguish between "not found due to masking" versus "haven't searched yet," leading to redundant searches or premature abandonment of viable paths. The accumulated context becomes noise rather than useful evidence.
Evidence Tracking Failure:
Redundant Search Loops: Models frequently re-visit or re-search pages they'v ve already explored, especially after intermediate reasoning steps. For example, after visiting Kane's page, finding Chad, visiting Chad's page, then spending time reasoning about the relation- ships, the model might search for "Kane Cornes" again or revisit Kane's Wikipedia page, essentially restarting from scratch. While not technically incorrect, this pattern wastes tion budget and rapidly fills context with duplicate information, accelerating context degra- dation and reducing the effective search depth the model can achieve before hitting token limits or action budgets.
G RELATED WORK
G.1 MULTI-HOP QUESTION ANSWERING BENCHMARKS
Multi-hop QA benchmarks evaluate models' ability to compose information across multiple rea- soning steps. Early datasets like HotpotQA Yang al. (2018b) and WikiHop Welbl et al. (2018) established foundational evaluation frameworks but suffer from systematic biases. Recent bench- marks have expanded coverage: FanOutQA Zhu et al. (2024) addresses multi-document reasoning, MINTQA He al., (2024) targets long-tail knowledge with 28K+ questions, and MEQA Li et al. 2024b) focuses on event-centric reasoning chains. However, these benchmarks embed hints that fundamentally alter the reasoning task.
We identify two categories of hints prevalent in existing benchmarks. Path-hinting occurs when questions linguistically encode reasoning chains (e.g., "What dance academy did the starring actress from The Glory of Tang Dynasty graduate from?"". reducing the task to executing pre-specified steps. Specification-hiniine provides excessive constraints that make answers discoverable through constraint satisfaction rather than reasoning (c.g.. combining "East German team," "founded 1966." "player born in 90s"). Unlike MuSiQue Trivedi et al. (2022) or 2WikiMuliHopPAA Ho al. (2020). which contain implicit structural hints, WebDetective introduces genuinely hint-free questions re- quiring autonomous reasoning path discovery.
G.2 RETRIEVAL-AUGMENT
GENERATION AND AGENTS
The evolution from static RAG pipelines to agentic architectures represents fundamental shift in how LLMs interact with external knowledge Singh et al. (225) (cche et al. (0025). While traditional RAG systems like TRACE Fang et al. (2024) achieve improvements through knowledge- grounded reasoning chains, they operate within predetermined patterns. Agentic RAG systems employ adap- tive strategies: Adaptive-RAse RAG Jeong et al. (2024) adjusts retrieval depth based question complex- ity, while graph-based approaches like GNN-Ret Li et al. (2024c) and HopRAG Liu et al. (2025) leverage graph neural networks for multi-hop hop reasoning, achieving 10% accuracy improvements on benchmarks like 2 2WikiMQA.
-oooedgg-grouudedd
Recent advances in 2025 emphasize diverse reasoning paths. DP-CoT et (2024)) addresses single-path limitations through passage-level and sentence-level evidence generation. However, our evaluation reveals these advances fail to overcome hint-free challenges: median Generation Scores of 20% across tested models indicate current architectures cannot effectively discover reasoning chains without linguistic scaffolding.
G.3 EVALUATION FRAMEWORKS
Traditional metrics like exact match and Fl scores collapse diverse failure modes into single values, obscuring why models fail Kwiatkowski et al. (2019); Petroni et al. (2021). Recent frameworks attempt more nuanced evaluation: RAGAS Shahul al. (2024) provides reference-free RAG met- rics, while RAGTruth Niu et al. (2024) enables hallucination analysis. For agents, AgentBench Liu
et al. (2023) evaluates across eight environments, tau-bench Yao al. (2024) addresses multi-turn
25

---

interactions, and ThemagentCompany Xu et al. (2025 introduces workplace tasks with simulated colleagues.
Web-based benchmarks have evolved significantly. WebArena Zhou et al. (2023) provides realistic web environments requiring long- long-horizon planning but lacks controlled evaluation for precise failure attribution. SWE-bench Jiimene al. (0024) evaluates code generation
GitHub issues, with SWE-bench Verified OpenAl (2024) addressing underspecified problems. While these benchmarks
test complex capabilities, they don't I address the specific challenge of verifying multi-hop reasoning paths.
Our diagnostic framework decomposes evaluation into knowledge sufficiency (whether agents pos- sess required evidence) and conditional generation quality (symthesis given sufficient knowledge). This separation reveals that models achieve 79% knowledge sufficiency but only 23% generation scores, indicating synthesis and relevance determination- not search- as primary bottlenecks.
H
LLMs USAGE
LLMs were used to polish the writing.
26


Reviews:
[{"review_id": "s5RplktZpQ", "reviewer": "Reviewer_ZvCi", "summary": "This paper presents WebDetective, a benchmark for evaluating web agents and RAG systems on hint-free multi-hop deep search tasks. It identifies two core issues in existing benchmarks: hint leakage in question design (Path- or Specification-Hinting) and single-metric evaluation that masks distinct failure modes. WebDetective co-designs hint-free questions with a controlled Wikipedia sandbox enforcing sequential reasoning through selective entity masking, and introduces a factorized diagnostic framework separating knowledge sufficiency, generation quality, and knowledge degradation. Evaluations on 25 models show that most retrieve sufficient evidence but fail in reasoning composition and calibrated refusal.", "questions": "(1) How do you ensure that human validators and verifying LLMs do not inadvertently rely on their own parametric knowledge when assessing whether questions are answerable only through the provided evidence chain?\n(2) Could you clarify how EvidenceLoop differs conceptually and architecturally from prior ReAct or self-reflection frameworks? Is its improvement mainly from verification loops or memory structuring?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) The paper identifies a genuine gap in current deep-search benchmarks—most rely on linguistic hints that reduce reasoning to execution. The distinction between Path-Hinting, Specification-Hinting, and the proposed Hint-Free formulation is sharp and well grounded.\n\n(2) The co-designed Wikipedia sandbox with selective entity masking is both creative and rigorous, ensuring that agents must discover reasoning paths rather than shortcut them. This design provides strong interpretability and enables precise failure attribution.\n\n(3) Evaluating 25 state-of-the-art models provides broad coverage and meaningful insights. The taxonomy of behavioral profiles (e.g., Powerful but Overconfident, Synthesis Bottleneck) is particularly informative for understanding system limitations.", "weaknesses": "(1) The proposed EvidenceLoop baseline achieves modest improvements and is not clearly positioned relative to existing ReAct-style frameworks, making it less convincing as a substantive modeling advance.\n\n(2) The controlled sandbox enforces a single reasoning path, which improves interpretability but departs from realistic web conditions where multiple reasoning routes may coexist. This trade-off limits ecological validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SsNDEgG8W5", "reviewer": "Reviewer_fQW4", "summary": "The paper introduces WebDetective, a benchmark for evaluating web agents on hint-free multi-hop question answering tasks. It also introduces a co-design evaluation environment to enable rigorous diagnostics. The evaluation contains multiple frontier systems and is comprehensive.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The writing is clear and easy to follow.\n\nThe problem formulation is novel, and the observation that existing benchmarks contain unrealistic hints and specifications is meaningful.\n\nThe diagnostic evaluation framework can help target different weaknesses of different systems, meaningfully characterize different systems under different categories, and provide insightful/actionable guidance to the development of future deep research systems.", "weaknesses": "1. The controlled Wikipedia sandbox, while enabling precise evaluation, creates artificial constraints that may not reflect real-world multi-hop reasoning scenarios. In practice, information is rarely hidden behind such strict sequential dependencies—multiple valid paths typically exist, and partial information can be retrieved from various sources. With an analysis of how performance correlates between sandbox and open-web settings, it could further enhance the validity and value of the benchmark. \n\n2. The 200 questions are relatively small for establishing robust benchmark conclusions. It would be helpful to enhance the statistical rigor by including significance tests.\nThe heavy concentration on 2-3 hop questions (86%) limits insights about longer reasoning chains. Besides, the questions are synthetically constructed by blocking direct paths in existing QA pairs, which may not reflect naturally occurring multi-hop reasoning patterns. \n\n3. The paper lacks analysis of domain diversity beyond question types, leaving unclear whether the benchmark targets diverse reasoning patterns or is dominated by specific domains (e.g., familial relationships).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8QQmc0yIpb", "reviewer": "Reviewer_yEz4", "summary": "This paper presents WebDetective, a benchmark for evaluating web agents on hint-free multi-hop deep search tasks within a controlled Wikipedia sandbox. Unlike prior datasets that leak reasoning paths or entity attributes, it enforces autonomous discovery of reasoning chains and enables fine-grained attribution of failure modes. The authors also propose a factorized evaluation framework—separating knowledge sufficiency, generation quality, and refusal behavior—and an agentic baseline (EvidenceLoop) integrating evidence tracking and verification. Experiments on 25 state-of-the-art models show that while systems can retrieve sufficient evidence, they often fail to synthesize it effectively, exposing a major synthesis bottleneck. The benchmark demonstrates that performance gains cannot be achieved by scaling context or computation, emphasizing that true progress in deep reasoning requires better evidence utilization and calibration, rather than brute-force search or larger models.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper proposes a very interesting and promising setting, hint-free multi-hop deep search within a controlled Wikipedia sandbox, which effectively exposes reasoning weaknesses that are usually hidden in traditional benchmarks with embedded hints. The benchmark design is well-motivated and enables fine-grained diagnosis of model behavior.\n2. The experiments are comprehensive, covering 25 state-of-the-art systems across multiple providers, and the analysis is insightful, identifying key failure modes like synthesis bottlenecks and calibration issues. \n3. The additional analyses are also strong, including detailed model profiling and robustness studies on scaling context length and computational budgets, which further validate the benchmark’s diagnostic value.", "weaknesses": "1. The presentation could be improved, including writing clarity, figure design, and overall structure. Some key insights are buried in the text and would benefit from clearer highlighting and visual emphasis.\n2. Certain methodological details are missing or underexplained. For instance, the process for testing Parametric Inaccessibility and Evidence Sufficiency is not clearly described—it’s unclear whether these evaluations are performed via prompting LLMs directly or through a separate automated verification strategy.\n3. While the benchmark is well-motivated, the paper could further clarify its generalization scope, such as whether the hint-free design can extend beyond Wikipedia-style domains or integrate with open web environments to evaluate real-world reasoning robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ReXl8xCEQ2", "reviewer": "Reviewer_rUPx", "summary": "This paper introduces WebDetective, a new benchmark for evaluating web agents on hint-free multi-hop deep search tasks. \nUnlike prior datasets that leak reasoning paths (path-hinting) or entity fingerprints (spec-hinting), the authors construct a controlled Wikipedia sandbox that enforces step-by-step reasoning without shortcuts. \nThey also propose a two-level evaluation framework that disentangles knowledge sufficiency, utilization, and refusal. \nExperiments on 25 state-of-the-art models show systematic weaknesses: models often retrieve enough evidence but fail at synthesis, and almost none refuse appropriately. \nThe authors further design EvidenceLoop, an agentic workflow baseline that uses verification loops and memory tracking to partially address these gaps.", "questions": "- Which LLM does your method use in Table 1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies and argues against the hidden hinting in existing multi-hop deep research datasets.\n- The co-design of hint-free questions with a sandbox masking mechanism is clever and ensures true multi-hop reasoning.\n- 25 strong models are compared, giving a broad empirical picture.", "weaknesses": "- The controlled Wikipedia sandbox may be too artificial compared to real-world open web environments.\n- While an interesting method, improvements seem modest. It feels more like a proof-of-concept than a strong new method for EvidenceLoop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)