Do LLM Agents Know How to Ground,  Recover, and Assess? Evaluating Epistemic Competence in Information-Seeking Agents
Authors: Jiaqi Shao
Yuxiang Lin
Munish Prasad Lohani
Yufeng Miao
Bing Luo
Venue: ICLR 2026 Poster (2026)
Keywords: Epistemic Competence
Evidence-Grounded Reasoning
LLM Search Agents
Primary area: datasets and benchmarks

Abstract: Recent work has explored training Large Language Model (LLM) search agents with reinforcement learning (RL) for open-domain question answering. However, most evaluations focus solely on final answer accuracy, overlooking how these agents reason with and act on external evidence.
We introduce **SeekBench**, the first process-level evaluation framework for LLM search agents that operationalize *epistemic competence* through metrics derived from an annotation schema.
We develop and validate our annotation schema using an expert-annotated dataset of 190 traces (over 1,800 steps). 
To evaluate at scale, we introduce an LLM-as-judge pipeline.
Our framework provides granular analysis of whether agents demonstrate: (1) **groundedness**, by generating reasoning steps supported by observed evidence; (2) **recovery**, by adaptively reformulating searches to recover from low-quality results; and (3) **calibration**, by correctly assessing whether current evidence is sufficient to provide an answer. 
By applying our evaluation framework to state-of-the-art search agents tuned on Qwen2.5-7B, we uncover critical behavioral gaps that answer-only metrics miss, as well as specialized skills such as Search-R1's synthesis abilities. These analyses highlight distinct epistemic competencies, offering actionable insights for the development of more capable and trustworthy agents. Code is available at
https://github.com/SHAO-Jiaqi757/SeekBench.

Published as a conference paper at ICLR 2026
Do LLM AGENTS KNOW How TO GROUND, RE- COVER, AND ASSESS? EVALUATING EPISTEMIC COM- PETENCE IN INFORMATION-SEEKIN AGENTS
Jiaqi Shaot.2, Yuxiang Lin', Munish Prasad Lohani', Yufeng Miao", Bing Luo
Duke Kunshan University
The Hong Kong University of Science and Technology
3 Microsoff AI
ABSTRACT
Recent work has explored training Large Language Model (LLM) search agents with reinforcement learning (RL) for open-domain question answering. How- ever, most evaluations focus solely on final answer accuracy, overlooking how these agents reason with and act on external evidence. We introduce SeekBench, the first process-level evaluation framework for LLM search agents that opera- tionalize epistemic compelence. We develop and validate our annotation schema using an expert-annotated dataset of 190 traces (over 1,800 steps). To evalu- ate at scale, we introduce LLM-as-judge pipeline. Our framework provides granular analysis of whether agents demonstrate: (1) groundedness, by gener- ating reasoning steps supported by observed evidence; (2) recovery, by adap- tively reformulating searches to recover from low-quality results; and (3) calibra- tion, by correctly assessing whether current evidence is sufficient to provide answer. By applying our evaluation framework to state-of-the-art search agents tuned Qwen2.5-7B, we uncover critical behavioral gaps that answer-only met- rics miss, as well as specialized skills such as Search-R1's synthesis abilities. These analyses highlight distinct epistemic competencies, offering insights for the development of more capable and trustworthy agents. Code is available at https://githut: com /ow. 757.Sleeeeeee7eeeeeeeeeeee
INTRODUCTION
Recent advances in Large Language Models (LLMs) have spurred a shift from models that require explicit prompting, such as chain-of- of-thought (Wei et al., [022] Yao al.) 2023). toward autonomous LLM agents (Zhang et al.) (025a). These agents learn to solve complex tasks by optimizing a reasoning policy with reinforcement learning (RL), enabling them to implicitly learn decision-making strategies without requiring step-by-step external guidance (Jaech et al.) 2024 Guo ict al., (025).. Among these, search agents are developed to tackle information seekiinnn--eiiggg problems by enabling LLMs to interact with external knowledge sources and handle queries beyond training data (Zheng et al.) 2025) Gao et al.] 2025). Modern agents increasingly use richer toolboxes (e.g., page-level browsing, selective scraping. Python execution) yet they still manifest the same epistemic loop: identify an information gap - obtain external evidence >> reason over it + decide the next action final answer. Our framework targets this process, independent of the surface tool API, to maintain precise empirical scope. They process multi-turn reasoning, using external search tools, and integrating evidence (Xi al. 2025a). The agent's responses produce multi-turn traces, which can be represented as:
T= (71, T2. .2.....,77) Tr),
where each turn consists of a tuple that can contain four possible steps: reasoning Tt, search evidence C₁, and answer at Non fonal turns < T) are of the form 7. = (re, St, et), while the final turn (1 = 7' concludes with an answer, i.e,, PT (rT, aT).
"Corresponding author

---

Published as a conference paper at ICLR 2026
Poor Process, Correct Answer
Question: What the current population City 2?
SEERBENCH
"City Z's 2025 population"
455,000 (a00s):
(as.000
(Evidence is Unclear)
REASONING NOT GROUNDED
"eeee City 2 455,000 population"
iggorr source
Answer: 455,000
Answer only
Answared early!
Good Process, Correct Answer
Question: purrent 2?ty
"City Z population"
Aemortte Soumar 4-000000 (Dová ectimote)
Source -00 (anao estimute)
REASONING GROUNDED
"There discrepancy and both sources are
Resognized ambigutt refinsmmnt
outdated. I should search for more recent number"
"City Z population latest 2015 official statistics"
Refine SEARCH
Returne: Source 435 ooo (sosos)
(Evidence Sufficient)
"Source C looks like the only dataset with a 2025 update. will the 2025 number?
Answer: "455,000"
Answer--nly
Figure I: Contrasting accuracy with epistemic competence. Both scenarios yield the correct answer "455,000" but are evaluated differently by SeekBench. Top (Poor Process): Agent ignores conflicting sources, fails to recognize ambiguity, and answers prematurely without grounding reasoning or recovering from unclear evidence. Bottom (Good Process): Agent recognizes ambiguity, refines search to obtain recent official data, and answers only after sufficient evidence.
Information-seekinng tasks pose a distinct epistemic challenge: unlike code generation or mathemat- ical reasoning, where correctness can be externally verified by execution or proof (Nguyen et al.] 2025 Lightman et al., 2023), search agents must reason about real-world textual information that lacks objective verifiers. However, current evaluation protocols for search agents largely ignore the process-level information captured by the structure of T. They rely predominantly on answer-only (aT) metries such exact match or Fl score (Jin al.) [025 Zheng et al.) 2025 Li al.) 20256).
This lack of process-level evaluation is problematic. As illustrated in Figure I agents may achieve high benchmark scores while exhibiting poor epistemic behaviors, such as hallucinating unsupported claims or failing to recognize knowledge gaps. This epistemic challenge necessitates process-level and evidence-driven evaluation to assess: groundedness (reasoning supported by evidence), recov- ery (adaptive search strategy to improve evidence), and calibration (determining when evidence is sufficient to answer).
To address the above challenges, we introduce SeekBench for evaluating agents' epistemic com- petence-the the ability to reliably acquire, evaluate, and act upon knowledge in a justified man- ner (Greene et al., 2016: Zhang al. 2023). Our framework proceeds in three stages. First, developed a robust, extensible annotation schema for search agent traces that captures both the functional role (reasoning, search, answer) and epistemic quality (e.g., evidence groundness and suf- ficiency) of each step. Following established Content Analysis methodology (Krippendorfi) 2018)) we analyzed and annotated seven open-source search agents across 190 traces (over 1,800 steps, Figure [] see Appendix [.I details). This expert-annotated dataset allowed us to validate and refine the schema, and the final schema achieves high inter-annotator agreement (Cohen's 's Kappa > 0.8). Second, we formalize three epistemic competencies: (1) Groundedness (evidence- grounded reasoning). (2) Recovery (adaptive evidence recovery), and (3) Calibration (evidence- aligned calibration). Third, we design precise, interpretable metrics to quantify these competencies across diverse traces, as detailed in Table [ To evaluate at scale, we introduce an LLM-as-judge pipeline that enables scalable evaluation. This pipeline uses our validated annotation schema to automatically annotate agent traces and evaluate the competencies on a large scale.
Contributions. Our contributions are summarized as follows:
1. SeekBench: A Process-Level Evaluation Framework. We present SeekBench as a practical evaluation frameowork for empirical analysis of LLM information--eeking agents. SeekBench comprises process-level annotation schema, operational epistemic metrics, and automated
2

---

Published as a conference paper at ICLR 2026
Table 1: Epistemic competencies and associated metrics. Each competency is quantified by specific metric calculated from annotated features within the agent's trace (shown in the rightmost column). enabling sys- tematic evaluation of reasoning quality, recovery behavior, and evidence-aligned decision decision-making. making.
Competency (Type)
Definition Metrie
Aunofated feature(s)
Generate ceasoning steps direcily supported by retrieved information.
Informat Cermmatiinin sonsynt hesla/
Groundedness
Metrie: Reasoning Quality Index (RQI. Section
Planf Seanfoommeteo ion/ Stt./Staattee Statehasessment grounding
(Reasoning)
Adaptively reformalate queries when initial search results insuf- Iiiiet.
Initial Repeat #ollowup/ Refined
Recovery
(Search)
Recovery
Section
Accurately whether the curcetlyy retrieved Intormation is
Calibration
sullicient to answer the question.
correct (final amswer's correctness)
(Answer)
Metrie:
Calihration Ermr (CE, Section
Quality
MI
(49.2%)
Sutteient (1609)
244%
Goourded (7..5%)
Insufficum (540%%)
19
19/77
Reasoning, 53%
Search, 47%
PinFFommtion (16.2%
Evidenee
Kopesr
Fallow
Batieed
InferratioSyshosi
Clear (40.5))
Unclor (99.2%
SEEKBENCH
Human Annotated Traces: N=190
Total steps:
Trajectory
Reasoning
Search
Evidlence
Search
Fvidence
Answer
Figure 2: Overview of the SeekBench dataset and annotation schema. Each trace comprises multi-turn steps annotated for process-level evaluation. We categorize agent behaviors into three main types: (1) Search steps retrieve information, (2) Reasoning steps process evidence and guide the investigation, and (3) Evi- dence steps that capture the quality and clarity retrieved information. This structured annotation approach enables systematic measurement of how well agents handle information throughout their reasoning process.
LLM- judge pipeline. Our schema achieves high expert agreement (Cohen's 's Kappa K 0.8), and our LLM judge demonstrates strong alignment with expert annotations > 0.7). 2. Operational framework and metrics. We formalize an evidence state and three core epis- temic competencies evidenceieeeeeeeeeeee groondddd reasoning. evidence recovery. and calibrated answer- ing measurable properties over agent traces.
3. Experimental case study. We apply our framework to RL agents tuned on Qwen2.5-7B across seven QA benchmarks (28,493 traces), finding that they excel at gathering evidence but struggle with reasoning. Standard accuracy metrics fail to reveal strengths between agents (c.g.. Search- RI's synthesis vs. Base model's 's reasoning), which can be combined to enhance performance.
RELATED WORK
Process-Level Reasoning Quality and Epistemic Competence. A fundamental limitation of answer-only evaluation is the disconnect between final-answer accuracy and reasoning process qual- ity. Recent work on faithfulnees, defined logical consistency of reasoning with respect questions and retrieved evidence (Lee & Hockenmaier 2025), reveals that models can achieve high accuracy despite unfaithful reasoning processes (Shen et al. 2025). Other approaches evaluate reasoning quality through causal analysis of question-reasoning rew-oningoonnngoe answer triples (Paul et al. 2024), alignment with golden reasoning chains (Li al. (024), and graph-based dependency modeling (Xiong al., 2025 Nguyen et al. 2024| Mukherjee et al. 2025). For search agents that interact with external information sources, epistemic competence becomes essential for avoiding overconfidence, halluci- nation, and poor decision-making under uncertainty. However, none of the existing frameworks can capture these capabilities. Our framework addresses this gap by formalizing three core epistemic competencies (groundedness, recovery, and calibration) with precise mathematical definitions and large-scale evaluation protocols.
Search Agent Evaluation. Existing evaluations primarily prioritize final-answer metrics (exact match, F1, LLM-as-Judhe Judge) (Zhang al. 2025b Song al. 2025 Zheng al. 2025 Jin et al.
2025), neglecting the underlying epistemic processes. While some approaches examine intermediate
steps-such such as ground-truth tracking (Shi al. 2025 retrieval separation et. 2025b) they
3

---

Published as a conference paper at ICLR 2026
fail to assess critical epistemic competencies. Final-answer metrics also fail to attribute performance sources, as (Shao et al., 2025) finds that RL-training tends to elicit existing reasoning behaviors rather than introducing ones. This methodological limitation creates significant blind spots regarding agent reliability.
METHODOLOGY
To understand the information-ssekingg process of search agents, framework connects observ- able behaviors with underlying competencies, which are then evaluated using quantitative metrics. First, construct and validate an annotation schema that reliably labels observable behaviors in agent response traces, producing the expert-annotatee dataset that we also use to calibrate LLM judges (Section 3.1] Second, we analyze these annotations identify three fundamental eeisammeeemleiitmiiiii competencies (Section 3.2 Finally, we translate the competency definitions into concrete quan- titative metrics and apply them to annotated features, enabling systematic measurement these underlying epistemic competencies at scale (Section 3.3) Our approach draws on established qual- itative research principles from Content Analysis (Krippendorfi, 2018), a systematic methodology for categorizing and interpreting patterns in data through rigorous coding procedures.
3.1 PIASE 1: OBSERVABLE FEATURES AND SCHEMA CONSTRUCTION
The foundation of our methodology is robust annotation schema for systematically labeling ob- servable features in agent response traces. Our development of SeekBench's 's schema follows an iterative, data-driven approach, grounded in established qualitative research principles from Content Analysis (Krippendorfi, 2018).
During the initial exploratory phase, we closely examined variety of agent traces and documented the key behaviors we observed. We noted that even within the same type of steps, search agents can serve distinct functions, that is, specific cognitive or operational role that step plays within the agent's cetrrmaatio--eeekin process. For example, among the reasoning steps, some identified in- formation gaps, while others summarized retrieved findings or formulated plans for future searches. Similarly, search steps might function as initial exploration, targeted verification, or follow-up in- vestigation. Alongside these functions, we identified critical failure patterns in agents' traces, such as reasoning without supporting evidence or executing repetitive search queries that failed to adapt.
From these observations, developed an annotation schema (detailed in Appendix A Figure 9 for agent response steps with two key aspects:
Functional type categorizing the step's cognitive purpose, e.g., for reasoning steps, this in- cludes Inrormat t. tetrmaaiiinyal ionsynthesis (evidence integration), PlanForrm ormt in (search strat- egy development), and StateAsse ssment (knowledge gap identification).
Quality attribute evaluating epistemic soundness, such as whether reasoning is grounded in evidence. This structure captures both what the agent was doing and how well it was doing it.
We apply this schema to construct an expert- annott-a validation dataset of 190 traces (Figure 2). which enables formalize three epistemic competencies and design quantitative metrics that systematically measure how annotated features map to underlying competencies [Table
Following established Content Analysis methodology (Krippendorfi, 2018), we rigorously enhanced schema robustmess through iterative refinement. Using setup comparable to previous works scaling from human LLM--baed annotation (Cemri et al. 2025), three expert annotators independently coded 190 agent traces across three rounds of annotation, with inter-annotator reliability measured using Cohen's ' Kappa (k) (Cohen) 1960p. For features exhibiting low agreement (N < 0.5), we either pruned features (when infrequent ambiguous) merged them (when conceptually overlapping). After this process, we reduced our initial 12 candidate annotation fields to well-defined features with high interpretability and consistency across annotators. We confirm this schema generalizes to complex agents in Appendix M
We further validated schema robustness using GPT-5 (OpenAl 2025 to generate adversarial edge cases that expose boundary conditions (c.g., reasoning steps containing both factual claims and planning elements). This ensures the mutual exclusivity of our annotation definitions.
4

---

Published as a conference paper at ICLR 2026
Table 2: Examples of evidence states. E
C Q. where C' {0, 1) indicates clarity and Q {0,
indicates sufficiency.
Question: Who is the singer for the band Black Sabbath?
E
C Search Result
Explanation
0
0
Doc I:
The band's vocal slot has many.
0: text is vague and evasive.
(Poor)
2:
Black Sabbath's personnel list vocals long.
0: no names are given.
I:
Black Sabbath famous heavy metal from England.
on-topic easy to understand.
(Partial)
Doc 2:
Tony Jommi is the guilar player for Black Sabbath.
0: docs not name the singer.
1 (Partial)
0
Doc I:
The first singer for Black Sabbath was Ozzy Osbourne.
C = 0: two singer names creates ambiguity.
Doc 2:
Ronnie James Dio became the new singer for Black Sabboth.
1: the names present.
2 (Good)
1
Doc I: Ozzy Oshourne the original lead singer of Black Sabbath.
C 1: the name is consistent.
Doc 2: The most farnous singer for Black Subbuth Ozzy Osbourne.
Q
1: the singer is ideatified.
Finally, we evaluated the schema on reasoning traces using both human experts and state-of-the-art LLM judges with standardized prompts that provide clear annotation guidelines and consistent evaluation criteria (see Appendix Dj. The results demonstrate substantial agreement with human annotations (overall x = 0.811), confirming the schema's interpretability and consistency. LLM judges achieved strong alignment with human experts: GPT-4.1 (K = 0.731) and GPT-5 (K = 0.754). We further conducted a cost-fffectsveness analysis, evaluating per-trace costs (token and time) across six LLM models. GPT-4.1-mint emerges the most cost-effective solution, achieving strong human alignment (N = 0.731) at minimal per-trace cost (S0.0087 and 2.48s), making optimal for large-scale deployment (see Appendix [ for cost analysis details). This substantial agreement across multiple LLM models and human annotators confirms the schema's clarity and establishes reliable foundation for large-scale evaluation through LLM judges.
3.2 PHASE 2: LATENT CONSTRUCTS AND COMPETENCY DEFINITION
Our annotation analysis in Phase I revealed three distinct behavioral patterns in search agents: (1) variation in reasoning quality, with successful agents producing evidence-supporte grounded reasoning while unsuccessful agents generated unsupported claims; (2) divergent strategies when facing poor search results, where effective agents adapted their search approach while ineffec- tive agents persisted with repetitive queries; and (3) differences in decision timing, where some agents responded prematurely with insufficient evidence while others appropriately withheld an- swers until sufficient evidence was gathered. To interpret these systematic behavioral differences, we apply latent construct inference (Cronbach & Meehl 1955) and arrive at three theory-backer competencies that explain the observed patterns (Table D (I) Groundedness captures whether each reasoning step is faithful to retrieved evidence, extending faithfulness checks from final an- swers to intermediate traces (Lee & Hockenmaier, 2025); (2) Recovery measures how effectively an agent reformulates and adapts its search to move from insufficient to sufficient evidence, mir- roring Cformationnforaginn dynamics between exploration and query refinement (Pirolli & Card 2005); ; (3) Calibration evaluates whether answer timing aligns with evidence sufficiency. draw- ing on metamemory research about calibrated confidence and appropriate abstention (Nelson 1990] Ming et al. 2024). These three core competencies constitute epistemic competence the essen- tial capability that enables search agents to reliably interact with external information sources. By systematically evaluating how agents seek, reason with, and make decisions based on retrieved evi- dence, our framework provides a comprehensive assessment of search agents' epistemic capabilities beyond traditional accuracy- based metrics.
3.3 PHASE 3: COMPETENCY METRICS AND OPERATTIONALIZATION
Following the concept of construct validity (Cronbach & Meehl 1955) originally proposed in psy- chology, unobservable attributes (competencies) must be assessed through observable indicators (metrics) with demonstrated reliability and validity. In this section, we translated the three epistemic competencies in Phase 2 (defined in Table TT into quantitative metries. The validity of our metrics is twofold: (1) high inter-annotator agreement on the coded features (Phase 1), and (2) the correlation between evidence state (defined below) and answer accuracy (Section 4.4
We begin by formally defining evidence state in Definition 3.1 which encodes the sufficiency and clarity of evidence retrieved at turn (examples in Table This provides the foundation for eval-
5

---

Published as a conference paper at ICLR 2026
uating all three epistemic competencies: (i) groundedness is assessed by determining whether rea- soning is supported by evidence (Section 3.3.1; (ii) recovery is measured by tracking improvements in evidence quality through search (Section B.3.2p; and (iii) calibration is evaluated on whether the agent answers and only if the evidence state is good (Section 3.3.3).
Definition 3.1 (Evidence State) Let Ci,,, Qs E {0. 13 denote the annotated clarity and quality (Appendix D.3] of the retrieved evidence al turn t. of trace i, where: Ca, = if the evidence is clear (unambiguous and interpretable), and Qú = if the evidence is sufficient (con- tains enough information address the query). Note that quality here specifically refers to sufficiency (whether the evidence contains enough information to answer the query), not a general quality assessment. The evidence state E ( (, is defined as:
E = Ci,t + Qi,t
(2)
E 0 denotes poor evidence (unclear and insufficient). E.tt = denotes partial evidence (either clear or sufficients, and E, = denotes good evidence (both clear and sufficient).
3.3. GROUNDEDNESS
To evaluate whether an agent's reasoning verifiably supported by retrieved evidence, we assess the groundedness of each reasoning step via the grounding label. For each reasoning step at turn é trace i, the binary grounding label G.,, E {0, 1 indicates whether its factual content is supported by retrieved evidence.
To investigate the impact of the functional types of the reasoning steps, each reasoning step is also assigned type Ca,, SA),, corresponding to InformariionSytthesii P PanFormatio on, or StateAssessment (Appendix(D.])
We formalize two metries: the model-level reasoning quality index (RQI) (Definition 3.2 and the type-level RQI (Definition 3.3). both of which quantify groundedness by aggregating Gi, values and can be decomposed by the evidence state E,,--
Definition 3.2 (Model-level Reasoning Quality Index (RQI) Consider a fixed model evaluated on N traces with index set I = {1,... . N.. For each trace let St fI,..., T.} be the index set of reasoning steps. Then, the model-level ROI is the average of trace-level groundedness scores:
RQQmedel
E [RQU].
where RQI, = Eiss [Gr,),
To better understand how reasoning quality depends on the strength of retrieved evidence, we de- compose the RQI with evidence state Eig:
RQL:
PrES, (Bit = k)
ExEs, [Gi, Ei,- k)
(4)
K=O
proportion of turns with evidence state le
expected groundedness given E..
Similarly, we can define the type-level RQI with reasoning type e [IS SA}:
Definition 3.3 (Type-Level Reasoning Quality Index) For each trace i and reasoning type E IS, PF, SA, let s($) = (t E S; Ci, c} denote the index set of steps of type The type-level RQI is the average of groundedness on type
RQU(e) type := E (ggs(s) [ROU!
where RQI = E sfe [G.,]. This quantity admits an evidence-state decomposition analogous to the trace-level decomposition:
RQI = Ei=
P
PLES ( (Ei, k)
X
E E [G.
Ei,
kj
(6)
prop. of reasoning type with evidence level
expected type groundedness given E,,
3.3.2 RECOVERY
A fundamental challenge for LLM-based agents is recovering from information gaps knowledge limitations, where initial queries yield insufficient information. Thus, an agent achieves high recov- ery when it utilizes adaptive search strategies to quickly escape such states of poor evidence.

---

Published as a conference paper at ICLR 2026
To capture this behavior, the evidence state E (0,1,2) track the sufficiency and clarity of retrieved information at each turn in trace i. We define recovery event (Equation (7) the first turn where the agent either (i) enters high-evidence state (Ei, 2), (ii) produces correct answer. Formally:
T recover, min {l [T) Bi, = or correct; = 1}:
where cor correct, indicates whether the agent's 's final answer in trace i correct.
To measure recovery behavior, introduce the Evidence Recovery Function (ERF), which quanti- fies the cumulative proportion of traces that have successfully recovered by each turn:
Definition 3.4 (Evidence Recovery Function (ERF)) Let N denote the total number of traces. The Evidence Recovery Function at turn is defined as
(RF == Σ ( (Treooer,,) S1),
where I(-) is the indicator function. ERF( measures the proportion of traces that have recovered by turn
3.3.3 CALIBRATION
We evaluate calibration as the agent's ability to decide when to answer based on the quality of retrieved evidence. well-calibrated agent should answer only when has acquired evidence that is both clear (unambiguous and directly relevant) and sufficient (contains enough information to support reliable answer).
Let answer;; E {0. 1) indicate whether the agent provides an answer at turn t of trace i. We assess calibration behavior by examining the answer rate conditioned on the evidence state:
P(answeri, = E.,- k).
High values at A: = O indicate epistemic overconfidence, where the agent answers prematurely with poor partial evidence. Conversely, low values at k = 2 suggest epistemic overcautiousness where the agent refrains from answering when evidence good.
To quantify calibration performance, we introduce Calibration Error (CE) that measures how much an agent's answering behavior deviates from the ideal policy. The ideal policy one that answers if and only if the evidence is good (E,, 2), which maximizes expected accuracy while minimizing wasted effort. This metric captures both epistemic failures: overconfidence (answering with insufficient evidence) and overcautiousness (not answering despite having good evidence).
Definition 3.5 (Calibration Error (CE)) Let I := (I....,N) be the index set of traces. Let o (k) = I[k 2] represent the ideal policy that answers and only if evidence is sufficient. The CE for a model is defined as:
CE : = Eezz
(10)
where for each trace CE;:- DIo:-E- [P(answeri, = Ei,, = k) -k)=x*(00 (k)].
For a perfectly calibrated agent following the ideal policy =l:), achieves CE = 0.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Models & Datasets. We evaluate Qwen-2.5577BIInsttuc ("Base" for training, Qwen et al. (2024) )
its few-shot prompted version ("Few-shot"), and state-of-the-art "RL-trained" agents based
Qwen-2.5-7B Instruct, including: SEARCH-RI (Jin et al.) 2025). RESEARCH (Chen et al. 2025).
ASEARCHER (Gao et al. 2025) and DEEPRESEARCHES (Zheng al. 2025). We also evaluate
Chain-of- Thought (Co'T) and ReAct prompting strategies, which show similar performance to Base
(see Appendix for details).
We evaluate the agents on a diverse set of seven QA benchmarks: NQ (Kwiatkowski et al. 2019) TriviaQA (Jshi et al. 2017), and PopQA (Mallen al. 2022) [single-hop]: HotporQA Yang al.

---

Published as a conference paper at ICLR 2026
Model
0.27
Bacc
0.23
0.9
Few-sket
RQI RQ!aodk
0,16
0.6
0:11
0.5
Searchl11
ReSearch
RQ
0.4
Aliendher
6?
8.16
4.5
0.2
6
0.0
Bler
Forusarr
Ardeand
ASunnerr
InfomatiinSynnhesis
PlanForatioon
StateAssessmmnt
Figure 3: RQI Analysis Summary. Left: RQI by model level, showing the overall reasoning quality across different agent types. Right: RQI by reasoning type, revealing that models struggle most with plan formation and state assessment compared to information synthesis.
2018), 2Wiki (Ho et al. 2020p, MusiQue (rivedi et al. 2022). and Bamboogle (Press et al. 2022) mmulti-hop-- Additionally, we evaluate search agents (32B ASearcher and WebSailor) on GAIA to assess epistemic competencies with web browsing capabilities (see Appendix K for details).
Each model runs and evaluates on the sanitized test datasets, where we remove ambiguous ques- tions and data contamination cases to ensure evaluation quality; we also removed questions that can be answered by the internal knowledge of the evaluated models alone since evidence state (Sec- tion 3.3) should depend only retrieved evidence (details for data sanitization in Appendix B Our evaluation comprises 28,493 traces and 283,950 steps across all models and datasets. com- prehensive statistical analysis of the dataset composition and annotation distributions is provided in Appendix C
For this large-scale annotation, we employ GPT-4. 1-mini, which demonstrates sub- stantial alignment with human judgments under our validated schema.
Answer-level performance. Aggregate F1 ranking is A ASEARCHER > Search-R1 > RESEARCH >> Few-shot DEEPRESEARCHER Base (Appendix Ep. Our primary analysis moves beyond out- comes to assess process-level epistemic competencies: evidence grounding (Section 4.2) recovery
dynamics (Section 4.3). and calibration (Section 4.4). We identify agent-specific competencies that drive performance gains and expose the overestimation of RL training (Section 4.5)) 4.5).
4.2 EVALUATING JUSTIFIED REASONING VIA EVIDENCE GROUNDING
A fundamental criterion of agent competence is not merely producing the correct answer, but doing so through reasoning process explicitly grounded in retrieved evidence. To measure this, we utilize the Reasoning Quality Index (RQI, defined Section B.3.I). which quantifies the proportion of an agent's & reasoning steps that are supported by retrieved evidence.
RL Training Fails to Develop Evidence-Grounded Reasoning, Figure (Left) presents the av- erage RQI scores across models. Few-shot prompting achieves the highest reasoning quality (RQI 0.27), outperforming all Rl-ttriined agents. This reveals a disconnect between answer-level suc- cess and reasoning groundedness: RI training may optimize for correct final answers, but it fails to develop the epistemic reasoning skills to justify those answers with evidence-gre eridence-grounded reasoning.
Plan Formation and State Assessment Are Core Reasoning Failures. To understand where rea- soning breaks down, we analyze performance by reasoning type (Figure Right). Specifically,
Information Synthesis emerges as a relative strength across models (e.g., ASEARCHER: 0.56),
demonstrating agents' proficiency in summarizing and restating retrieved information.
Plan Formation constitutes the most significant weakness for all agents (consistently scor- ing below 0.2), highlighting fundamental difficulties in breaking down complex queries and formulating coherent search strategies.
State Assessment shows notable improvement in few-shot models (0.28), suggesting enhanced metacognitive capabilities compared to their RL-trained counterparts.
For detailed analysis of evidence-conditionee reasoning quality across different evidence states and reasoning types, see Appendix
4.3 RECOVERY ANALYSIS
This section evaluates whether models can effectively recover from low-quality evidence through adaptive search strategies.
8

---

Published as a conference paper at ICLR 2026
Eviiencc Recovery Function (ERF)
Base (Dwen)) Fow shot (Qwar)
AGearcher
ReSearch Searchi1
Refine
DecsReszarcher
Follow-up
Search
types
P(romaiiing B-22)
Grounded reasoning
0.4
0.3
0.0
0
10
Figure 4: Recovery Analysis. Left: ERFs by model showing recovery from low sufficient evidence (E=2) turn increases. Right: Recovery efficiency by action type. Steeper curves indicate faster escape from low evidence states. REFINE and FOLLOW- UP enable fastest recovery, while REPEAT shows minimal improvement.
Recovery Competence with ERE. We first assess overall recovery competence using the Evidence Recovery Function (ERF, Equation (8p), which measures the cumulative probability reaching suf- ficient evidence (E 2) over time. As shown in Figure g (Leett, ASEARCHER, which has the highest Fl score on answer correctness, shows superior recovery performance compared to other agents. In contrast, DEEPRESEARCUEE shows the poorest recovery performance. This demonstrates that effective algorithm design should prioritize developing adaptive evidence-seeking strategies that agents recover from insuffi- cient evidence and improve final performance.
which has the lowest FI among all RL-trained agents,
Refine and Follow-up Search Strategies Drive Effective Recovery. To identify the most effec- tive search strategies for recovery, we analyze how different action types affect recovery rates over time. We categorize all search and reasoning steps by their types. For each step of specific type in trace, we measure the proportion of turns remaining low evidence states (E < 2) at subsequent turns + At). Given the variable-length traces and resulting right-censored data (traces ending before recovery occurs-when observation periods end before the outcome), we employ Kaplan-Meier sur- vival analysis (Kaplan Meier I958). which provides robust estimation of recovery probabilities despite incomplete observations. As shown in F (Right), survival curves reveal that REFINE and FOLLOW-CP strategies enable the fastest recovery from low-quality evidence, while REPEAT provides minimal benefit. Additionally, GROUNDED REASONING also effectively improves evi- dence utilization in responses.
4.4 SUODENEE-ALIGNER CALIBRATION
This section evaluates whether models calibrate their answering behavior to the ideal policy, where they answer when and only when it has good idence, avoiding both overconfidence (answering with poor evidence) and overcautiousness (failing answer despite good evidence).
Puwwer to Nomeeeooves to Trode:
Picoroot snswer, E)
Foodoo
0.35
Pranswer
1.10
0.20
Evidence Quality Drives Answer Accuracy. We
0.06
0.1
first validate that evidence quality correlates with swer accuracy. As shown in Figure 5 RL-trained models achieve 31.6% accuracy when answering with good evidence (P(correctaanswar E 2)). compared to only 8.4% accuracy when answering without supporting evidence. This significant differ- ence demonstrates that evidence quality is positively associated with answer correctness.
B=00
Figure 5: Evidence State Drives Answer Probability and Accuracy. Lines: Answer probability increases with evidence state. Bars: Answer accuracy improves with evidence state. RL-trained models show lower answering rate but higher accuracy with good evidence.
Interestingly, RL-trained models exhibit lower answering rates (P(answer) E), formally defined in Equation across all evidence states compared to base models. This suggests that RL training encourages models to be more selective about when to provide final answers, potentially reducing instances of overconfident responses.

---

Published as a conference paper at ICLR 2026
Table 3: Calibration Error Analysis. traces categorized calibration error, overconfident, overcautious.
Lower values indicate better calibration. RL-trained agents show the lowest overconfident answer rate and lowest CE. Bold indicates best performance.
Model (1) Overconfident & (2) Overcautions + (3) Calibration Error +
Base
0.631
0,030
0.329
Few-shot
0.511
0.024
0.317
RL-trained
0.353
0.085
0.309
To further understand the calibration behavior, we measure calibration quality using calibration error (CE, Equation (TOb) and analyze specific failure modes to identify where models fail:
(1) Overconfident answering: providing a final answer when the trace never reached good evi- dence state (E 2 for all (), indicating overconfidence;
(2) Overcautious abstention: failing to provide a final answer despite having reached good evi- dence state (,,, 2), indicating underconfidence.
RL Training Improves Calibration. As summarized in Table RL-trained models show substan- tial improvements calibration behavior. They reduce overconfident answering from 63. % to 35.3% and achieve the lowest overall calibration error (0.309). This demonstrates that RL train- ing successfully teaches models to align their answering decisions with evidence quality, moving toward the ideal policy of answering only when evidence is sufficient. This finding contrasts with the earlier result that RL training degrades reasoning groundedness (Section 4.2). highlighting the competency-spectte nature of RL training effects. For detailed analysis of individual RL-trained agents and evidence-conditione answer timing patterns, see Appendix
4.5 EXPLOITING EPISTEMIC COMPETENCIES FOR PERFORMANCE GAINS
Our evaluation reveals distinct agent specializations: ASEARCHER excels evidence acquisition and recovery mechanisms (highest overall Fl score), while SEARCI-RI demonstrates superior information synthesis (RQI=0.63 63 for information synthesis) with minimal overconfident answer- ing (see Section 4.400m444ann and Appendix HH). These differences primarily driven by training objectives: SEARCH-R R1 optimizes only for final answer correctness, achieving high accuracy (Jin et al. 2025) but sacrificing reasoning groundedness (Section 4.22, while ASEARCHER emphasizes data synthesis with failure and recovery strategies to teach adaptive re-planning (Gao et al. 2025), leading to superior evidence acquisition and recovery capabilities (Section 4.3). This motivates us explore agent synthesis- one agent's is evidence collection input for another's answer generation.
We provided agents with reasoning traces and evidence from others, then measured Fl score im- provements. SEARCH-RI emerges the most effective synthesizer (+2.61 Fl on average), sig- nificantly outperforming other agents (see details in Appendix []. Surprisingly, Base achieved the highest Fl gains (+2.42 average) when paired with other models for answer generation. This veals that accuracy-only evaluation may underestimate Base's 's reasoning abilities while overstating the gains from RL training.
Our method reveals distinct agent profiles by systematically benchmarking their epistemic compe- tencies, for example, SEARCH- R1 synthesis strength and conservative answering. These insights provide a reliable foundation for designing effective systems that capitalize on complementary agent strengths. Furthermore, actionable inference-tine timeenece-imeeeeeee feedback signals, achieving 8.4% Fl improvement on ASEARCHER without training (see Appendix D). Overall, beyond outcome-based metrics, our approach delivers procedural evaluation that enables more interpretable assessments of agent competence.
demonstrate that our epistemic competency framework can provide
CONCLUSION
SeekBench evaluates epistemic competence in LLM search agents through expert-annotated traces, revealing gaps in current evaluation approaches. Our evidence state framework and metrics (RQI. ERF, CE) uncovers agent-specific strengths masked by accuracy-only evaluation: Search-RI excels evidence synthesis, while Base models demonstrate stronger reasoning capabilities than accuracy metrics suggest. This work establishes epistemic competence as essential for developing reliable information-seeking agents. Future work should explore modular architectures combining comple- mentary strengths and training approaches that improve reasoning alongside answer calibration.
10

---

Published as a conference paper at ICLR 2026
Acknowledgments. This work is supported by Suzhou Frontier Science and Technology Program (Project SYG202310).
Ethics Statement. This research evaluates LLM search agents using publicly available datasets and involves only expert annotation of agent traces. Our framework aims to improve Al system reliability and transparency for developing more trustworthy information-seeking agents. The SeekBench dataset will released with appropriate documentation while respecting licensing terms.
Reproducibility Statement. To ensure reproducibility of our findings, we provide comprehensive documentation of our methodology and evaluation framework. Our annotation schema and inter- annotator agreement analysis (Appendix A establish the reliability of our epistemic competency measurements, with Cohen's x = 0.811 for human annotators and strong L.LM-judge alignment (k 2 0.693). The complete annotation guidelines and evaluation prompts are detailed in Ap- pendix enabling replication of our step-level reasoning quality assessments. Our three core met- rics Reasonisn Quality Index (RQI), Evidence Recovery Function (ERF), and Calibration Error (CE) are formally defined with mathematical specifications in Section 3.3 The evaluation spans seven established QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MusiQue, Bam- boogle) with 28,493 traces and 283,950 annotated steps across six agent variants (Qwen-2.5--B- Instruct and few-shot, Search-R1 1, ReSearch, ASearcher, DeepResearcher), Data sanitization proce- dures to remove ambiguous questions and contamination cases are documented in Appendix B All experimental results, including detailed agent-specific breakdowns and wiidence-conditioned anal- yses, are provided in the main text and appendix. The SeekBench dataset of 190 expert-ance aapert-annttaeed traces with over 1,800 response steps will be made available as supplementary material, along with our annotation schema and evaluation code to enable community replication and extension of our epistemic competence framework.
REFERENCES
Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, al. Why do multi-agent Ilm systems fail? arXiv preprint arXiv:2503.1300 2025.
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff 7. Pan, Wen Zhang. Huajun Chen, Fan Yang, et al. Learning to reason with search for ilms via reinforce- ment learning. arXiv preprint arXiv:2503. 19470, 2025.
Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological mea- surement, 20(1):37 -6, 1960.
Lee J Cronbach and Paul E Meehl. Construct validity in psychological tests. Psychological bulletin, 52(4):281, 1955.
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint arXiv:2508.077 07977, 2025.
Jeffrey A Greene, William A Sandoval, and Ivar Bråten. An introduction to epistemic cognition. In Handbook of epistemic cognition, pp. 1-16. Routledge, 2016.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprini arXiv:2501.12948. 2025.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint 20Xi::0011001060 2020.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167200 2024.
11

---

Published as a conference paper at ICLR 2026
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprini arXiv: 1705.0355,, 2017.
Edward L Kaplan and Paul Meier. Nonparametric estimation from incomplete observations, Journal of the American statistical association, 53(282):457-481, 1958.
Klaus Krippendorff. Content analysis: An introduction to its methodology. Sage publications, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Jinu Lee and Julia Hockenmaier. Evaluating step-by-step reasoning traces: A survey, arXiv preprint arXiv:2502.12288, 2025.
Kuan Li. Zhongwang Zhang. Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baix- uan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592 2025a.
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baix- uan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02533 025592 2025b.
Ruosen Li, Zimu Wang, Son Tran, Lei Xia, and Xinya Du. Meqa: A benchmark for multi-hop event-centric question answering with explanations. Advances in Neural Information Processing Systems, 37:126835-126662, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Daniel Khashabi, and Hannaneh Hajishirzi.
When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:22 /,. 105 11, 2022.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference Learning Representations, 2023.
Yifei Ming, Senthil Purushwalkam. Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. Faitheval: Can your language model stay faithful to context, even if" the moon is made of marshmallows". arXiv preprint arXiv: 14.00377, 2024.
Sagnik Mukherjee, Abhinav Chinta, Takyoung Kim, Tarun Anoop Sharma, and Dilek Hakkani Tur. Premise-augmented reasoning chains improve error identification in math reasoning with LLMs. In Forty-second International Conference Machine Learning, 2025. URL https:
Thomas O Nelson. Metamemory: A theoretical framework and new findings. In Psychology of learning and motivation, volume 26, pp. 125-173. Elsevier, 1990.
Minh-Vuong Nguyen, Linhao Luo, Fatemeh Shiri, Dinh Phung, Yuan-Fang Li, Thuy-Trang Vu, and Gholamreza Haffari. Direct evaluation of thain-of-thought in multi-hop reasoning with knowl- edge graphs. arXiv preprint axXiv:2402.11199 2024.
Xuan-Phi Nguyen, Shrey Pandit, Revanth Gangi Reddy, Austin Xu, Silvio Savarese, Caiming Xiong, and Shafiq Joty. Sfr-deepresearch: Towards effective reinforcement learning for autonomously reasoning single agents. arXiv preprint arXivv2509.06288, 2025.
12

---

Published as a conference paper at ICLR 2026
OpenAl. 2025.
Introducing httoss//oeaaaaaco///a/aa///ttttt
Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. Making reasoning matter: Measur- ing and improving faithfulness of chain-of-though reasoning. arXiv preprint arXiv:2402.13950 2024.
Pirolli and SK Card.
The sensemaking process and leverage points for analyst technology as iden- tified through cognitive task analysis, proceedings of the international conference on intelligence analysis. 2005.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Mcasuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
A Yang Qwen, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, F Huang, H Wei, et al. Qwen2. technical report. arXiv preprint, 2024.
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, et al. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947 2025.
Xu Shen, Song Wang, Zhen Tan, Laura Yao, Xinyu Zhao, Kaidi Xu, Xin Wang, and Tianlong Chen. Faithcot-bench: Benchmarking instance-level faithfulness of ohhin-h reasoning. arXiv preprint wrXiv:2510.0404:4 2025.
Yaorui Shi, Sihang Li, Chang Wu, Zhiyuan Liu, Junfeng Fang, Hengxing Cai, An Zhang, and Xiang Wang. Search and refine during think: Autonomous retrieval-augmented reasoning of Ilms. arXiv preprint arXiv:2505. 1277, 2025.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in ilms via reinforcement learning. arXiv preprint arXiv:2503.05550 2025.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2488 21443,,2000 2022.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. /2516, 2025.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang,
Yulan He, Deyu Zhou, Pengjun Xie, et al. Webwalker: Benchmarking Ilms in web traversal. arXiv preprint arXiv:2501.075555 2025.
Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. A survey of llm-based deep search agents: Paradigm, optimization, evaluation, and challenges. arXiv preprint arXiv:2508.00 056668. 2025a.
Yunjia Xi, Jianghao Lin, Menghui Zhu, Yongzhao Xiao, Zhuoying Ou, Jiaqi Liu, Tong Wan, Bo Chen, Weiwen Liu, Yasheng Wang, et al. Infodeepseek: Benchmarking agentic information seeking for aurreval-augmenneed generation. arXiv preprint arXiv:2505. 15872, 2025b.
Zhen Xiong, Yujun Cai, Zhecheng Li, and Yiwei Wang. Mapping the minds of 11ms: A graph-based analysis of reasoning 11m. arXiv preprint wrxiv:2505.18800 2025.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: / 1809.0800000 2018.
13

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, et al. The landscape of agentic reinforcement learning for llms: A survey. arXiv preprint arXiv:2509.02547 2025a.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, al. Siren's s song in the ai ocean: survey on hallucination in large language models. arXiv preprini arXiv:2309.012 2023.
Yuxiang Zhang, Yuqi Yang, Jiangming Shu, Xinyan Wen, and Jitao Sang. Agent models: Internaliz- ing chain-of-action generation into reasoning models. arXiv preprint arXiv:2503.06580 2025b.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments. arXiv preprint arXiv:2504.00 03/60, 2025.
14

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Related Work
3
Methodology
Phase I:
Ohservable Features and Schema Construction
3.2
Phase 2: Latent Constructs and Competency Definition
3.3
Phase 3: Competency Metries and Operationalization
5
3.3,
Groundedness
3.3.2
Recovery
3.3.3
Calibration
4
Experiments
Experimental Setup
Evaluating Justilied Reasoning via Evidence Grounding
Recovery Analysis
SridecceAAligne Calibration
9
4.5
Exploiting Epistemic Competencies for Performance Gains
10
Conclusion
10
A Inter-Annotator Agreement Analysis
17
Data Sanitization
18
Dataset Statisties
19
Dataset Construction
19
SrgeeScale Evaluation Dataset
19
LLM-as-Judge for SeekBench
21
D.I Reasoning Type Annotation and Grounding Evaluation
21
D.2 Search Behavior Annotation
23
D.3 Search Result Quality Assessment
23
Accuracy-level performance
24
F
Eviuence-Grounded Reasoning Analysis
24
Evidence-Aligned Reasoning: Do Agents Ground Their Inference What They Know?
24
Sppe-Specific Evidence Aligument: Which Reasoning Skills Are Evidence-Grounded]
25
Recovery Analysis
26
Evidence Calibration Analysis
27
15

---

Published as a conference paper at ICLR 2026
Agent Synthesis: Leveraging Epistemic Competencies for Answer Generation
28
Inference-Time Feedback Using Epistemic Competencies
30
Evaluation GATA
30
Discussion: Understanding Metric Trade-ofis)
31
M Extended Validation: Generalizahility Across Diverse Tools and Models
31
Use of Large Language Models
33
16

---

Published as a conference paper at ICLR 2026
Forthdd Arnobation Agreement (Cahen's
Ovarall Agreament Surrarry
0,01 0,60
0734-033
0.75460055
6.3
0.91330042
Ae255 6.009
0.4350 500
0P343
095
Figure 6: Inter-annotator agreement for SeekBench. (Leff) fer-field annotation agreement across different com- petency dimensions. (Righr) Average Cohen's comparing human annotators with GPT-4.1, GPT-4. I-mini, and GPT-5, demonstrating strong alignment between expert human judgments and advanced LLM assessments.
A INTER-ANNOTAOOR AGREEMENT ANALYSIS
Annotation Schema Overview, Our annotation schema (Figure 9 provides structured framework for labeling agent traces, capturing both the functional role and epistemic
quality of each step. The schema consists of two key dimensions: (1) Func- tional Type, which categorizes the cognitive purpose of reasoning and search steps (e.g., StateAssessment, PlanFormation Query, Refi Refi nedquryy, Fol owwppuery, RepeatQuery for search), and (2) Quality Attribute, which evaluates epistemic soundness (e.g., Groundedness for reasoning, Quality and Clarity for search results, Correctness for final answers). This dual- dimensional structure enables comprehensive process-level evaluation by capturing both what agents are doing and how well they are doing it. The schema's reliability, as measured through inter- annotator agreement, establishes the foundation for our epistemic competence evaluation frame- work.
innormationSSyteess s for reasoning;
Initial Initialquery,
Per-Field Agreement Analysis. The left panel of Figure demonstrates robust inter-annotator agreement across all four annotation fields. The Functional Type field achieves the highest agree- ment (k > 0.8), indicating that annotators can reliably distinguish between different reasoning purposes (e.g., Information Synthesis vs. Plan Formation). The Quality Attribute field shows simi- larly strong agreement (x > 0.75), confirming that evaluative judgments of epistemic soundness are consistently interpretable across annotators. These results establish that our schema captures mean- ingful, distinguishable patterns in agent reasoning behavior rather than subjective interpretations.
Human- -LMM Alignment Assessment. The right panel reveals substantial alignment between hu- man expert judgments and L.LM assessments across all three evaluated models. Human annotators achieve the highest overall agreement (x 0.811), establishing the reference standard for annota- tion quality. Among LLM judges, GPT-5 demonstrates the strongest aligmment with human experts (x 0.754), followed by GPT-4.1-mini (x 0.731) and GPT-4.1 sive improvement across model versions suggests that more advanced language models can better approximate human reasoning patterns in epistemic evaluation tasks.
0.693). This progres-
DosttEffectiveness Analysis for LLM Judges. To evaluate the practical deployment of LLM-as- judge systems, we analyze the trade-off between annotation quality (measured by inter-annotator agreement, IAA) and per-trace cost (combining token cost and time cost) across six language mod- els. Table provides detailed per-trace cost and IAA metrics for all evaluated models. Figure 7 presents Pareto frontier analysis, identifying models that achieve optimal balance between cost
17

---

Published as a conference paper at ICLR 2026
Model Pareto Frontier Analysis
Pareto-optimal models balance cost and utility
GPT-5
0.74
1.--inlmminl.....
Cost Details:
Average Kappa
0.72
GPT-4.1-mini: $1.65 472s
GPT-S-nano: $1.87 || 867s
0.70
SPT mini
Gemini-2.5-flash $4.52 5.
GPT-4.1: $8.38 || 279s
GPT-S-nan
GPT-S-mini: $9 999s
0.68
GPT-5: $19.30 30 I| 5410s
0.66
Dominated
2.5-flash
Pareto Frontier
0.64
Normalized Total Cost (Token Cost + Time Cost)
Figure 7: Model Pareto Frontier Analysis. Trade-off between normalized per-trace cost (x-axis) and average Cohen's (IAA, y-axis) across language models. The x-axis is computed by min-max normalizing token and time costs per trace separately [0, 1], then summing with equal weight. GPP-4.1-min and GPT-5 lie the Pareto frontier, with GPT-4.1-mi I-mini offering optimal cost-effectiveness for large-scale deployment.
and utility. Our evaluation reveals that GPT-4.1-mini emerges the most cost-effective solution, achieving strong human alignment (IAA = 0.731 + 0.057) at minimal per-trace cost ($0.0087 and 2.48s), positioning it the Pareto frontier alongside GPT-5. While GPT-5 achieves the highest agreement (x = 0.754 0.032), its substantially higher per-trace cost ($0.1016 and 28.47s) makes it less suitable for large-scale deployment. These results establish GPT-4. 1-mini the optimal choice for scalable evaluation frameworks that require both high-quality annotations and economic feasibility.
Agreement (IAA) for LLM Judges 190 sampled traces. Token cost (in USDS per trace), time cost (in seconds per trace), and IAA measured Cohen's with standard deviation. Models marked with i are Pareto-optimal.
Table 4: Per-trace cost and
Model
Token Cost ($/trace)
Time Cost (s/trace)
TAA (x+ std)
GPT-4.1-mini!
0.0087
2.48
0.731 0.057
GPT-S-nano
0.0098
4.56
0.683 0.038
Gemini-2.5-flash
0.0238
4.77
0.643 0.102
GPT-4.1
0.0441
1.47
0.693 +: 0.043
GPT-5-mini
0.0485
5.26
0.791 + 0.028
GPT-S*
0.1016
28.47
0.754 1 0.032
Implications for Large-Scale Evaluation. These agreement results establish the feasibility of de- ploying our annotation framework for comprehensive agent evaluation. The strong human-LLM alignment enables cost-effective scaling of our evaluation methodology, while the robust per-field agreement ensures that competency assessments reflect genuine behavioral differences rather than annotation artifacts. This validation is particularly crucial for our three core competencies (Ground- edness, Recovery, and Calibration), as confirms that these constructs can reliably measured across diverse reasoning traces and evaluators.
B DATA SANITIZATION
To ensure the quality of our evaluation, sanitize the test sets of our seven benchmark datasets using the following two criteria:
(1) Ambiguous Unanswerable Questions. We discard questions where benchmark agents ceive full credit for speculative answers, while a stronger reference model (GPT-4.1-min) abstains with justified explanation. For example, in response to the question "Who developed the CPU?". a benchmark agent might confidently output "John von Neumann" achieving EM=1. In contrast,
GPT-4.1-min responds:
"Anawer:
don't
know; Reasons
information ions gigures ki
18

---

Published as a conference paper at ICLR 2026
John von Neumenn J. Presper ackert,
not identify single developes." These ques- tions are excluded to avoid rewarding superficial matching over careful reasoning.
(2) Data Contamination. We discard questions where agents succeed (Pass 3) without issuing any search queries, as this indicates the question likely part of the model's pre-training data. These removed to focus evaluation retrieval-dependent reasoning.
C DATASET STATISTICS
C.I DATASET CONSTRUCTION
To validate our annotation schema and establish LLM LLM-as-judge feasibility, we constructed an expert- annotated validation dataset of 190 traces. We sampled traces from the benchmarks described in our experimental setup (Section[4.) [.1).. For each agent-dataset combination, we selected 2 correct and 2 incorrect answer traces, targeting an ideal sample size of 7 7 4 = 196 traces. The final dataset contains 190 traces (6 fewer than idcal) because some traces, particularly from single-hop QA datasets, lack multiple search queries. We prioritize multi-query traces because our framework focuses on process-level analysis to evaluate how agents reason, adapt search strategies, and make decisions across multiple steps. This sampling strategy ensures our validation dataset captures the epistemic behaviors our framework is designed to assess.
agents and seven QA
C.2 LARGE-SCALE EVALUATION DATASET
This section provides a comprehensive statistical analysis of the evaluation dataset, which comprises 28,493 traces across seven cresstoonansweringg benchmarks. Figure 8 presents six complementary views of the dataset composition and annotation distributions, revealing key characteristics that in- form our epistemic competency evaluation.
Dataset Composition. The evaluation dataset spans seven established question-answeringg bench- marks, providing diverse coverage both single-hop and multi-hop reasoning tasks. As shown in Figure 8 (top-left), the dataset distribution is relatively balanced across sources: PopQA (28.9%) and MusiQue (26.3%) constitute the largest portions, followed by HotpotQA (12.5% and 2Wiki- MultihopQA (12.5%). TriviaQA (7.6%) and NQ-Search (7.6%) contribute smaller but substantial portions, while Bamboogle (4.6%) provides the smallest contribution. This distribution ensures that our evaluation framework tested across diverse question types, from factual single-hop queries to complex multi-hop reasoning tasks requiring information synthesis across multiple sources.
Information Quality and Clarity. The quality and clarity of retrieved evidence fundamental to evaluating epistemic competence, as they directly determine the evidence state E.e used in our metries. Figure 8 (top-middle and top-right) reveals critical challenges in the information landscape that agents must navigate:
Information Clarity: A substantial majority of search results (62.5%, 48,134 134 instances) are categorized as unclear, meaning they contain ambiguous, vague, or confusing information that could match multiple entities or interpretations. Only 37.5% (28,905 instances) are classified as clear, indicating straightforward, unambiguous information. This distribution highlights the prevalence of ambiguous search results information-seedt seeking scenarios, emphasizing the importance of agents' ability to handle uncertainty and adapt their search strategies.
Information Quality: The distribution of information sufficiency is nearly balanced, with 52.7% (40,559 instances) classified insufficient and 47.3% (36,402 instances) as sufficient. This near-even split reflects the inherent difficulty of information-seckee queries often fail to retrieve complete answers, requiring agents to demonstrate recovery capa- bilities through adaptive search refinement.
tasks, where initial
19

---

Published as a conference paper at ICLR 2026
Comprehensive Statistics
Dataset Dittmbtttin
intormatonn Clartty Distriuuion
Information Distiiuttion
48114
4030
1399
bemtooge
25305
1000
1300
530
ONI
Deananing Amnetation Types
Query Ammotation Types
Grouningg Datribut
0000
30000
1500
2.20
10300
1000
1000
1000
2000
2X04
Lo1a
Palowe
Counding
Quary Type
Reeoniin Tale
Figure 8: Comprehensive Dataset Statistics. (Top-left:) Distribution of traces seven QA benchmarks, showing the relative contribution of each dataset. (Top-middle:) Information clarity distribution, categorizing search results as clear or unclear. (Top-right:) Information quality distribution, indicating whether retrieved evidence is sufficient or insufficient. (Bottom-left:) Distribution of reasoning annotation types across all rea- soning steps. miidle)-idddl:) Distribution of query annotation types across all search steps. (Bottom-right:) Grounding distribution, showing proportion of reasoning steps that are grounded versus not grounded retrieved evidence.
Reasoning Type Distribution. Our annotation schema categorizes reasoning steps into three fune- tional types that capture distinct cognitive processes in information-sseeking Figure & (bottom-left) shows that Plan Formation (41,531 instances, 43.7%) and State Assessment (41.550 instances, 43.7%) dominate the reasoning landscape, with nearly identical frequencies. These two types to- gether account for 87.4% of all reasoning steps, indicating that agents spend substantial effort on metacognitive processes: identifying knowledge gaps (State Assessment) and formulating search strategies (Plan Formation). In contrast, Information Synthesis (12,022 instances, 12.6%) repre- sents smaller but critical component, where agents integrate retrieved evidence to form conclu- sions. This distribution reveals that effective information seeking requires sophisticated planning and self-monitoring capabilities, not merely the ability to synthesize information once it retrieved.
Query Type Distribution. Search behavior analysis reveals how agents adapt their information- seeking strategies. Figure 8 (bottom-middle) shows that RefinedQuery (32.323 instances, 60.3%) the most common query type, demonstrating that agents frequently modify their search approach based on previous results. This adaptive behavior is essential for recovery from poor initial evidence. RepeatQuery (19,729 instances, 36.8%) represents a substantial portion of search behavior, indicat- ing that agents sometimes persist with similar queries, which may reflect either strategic refinement or ineffective adaptation. FollowUpQuert (1,573 instances, 2.9%) is relatively rare, suggesting that agents infrequently employ exploratory follow-up questions that diverge from their primary search trajectory. This distribution underscores the importance of query refinement as a core mechanism for evidence recovery, while also highlighting the potential for improving agents' exploratory search capabilities.
20

---

Published as a conference paper at ICLR 2026
Grounding Distribution. The grounding analysis evaluates whether reasoning steps supported by retrieved evidence, which is central to the Groundedness competency. Figure g (bottom- right) reveals critical finding: only 17.3% (16,663 instances) of reasoning steps classified as grounded, while 82.7% (79,367 instances) are not grounded. This substantial imbalance indicates that the majority of agent reasoning steps lack direct support from retrieved evidence, representing a fundamental epistemic challenge. This finding aligns with our RQI analysis (Section 4.2). The prevalence of ungrounded reasoning highlights the critical need for evaluation frameworks that sess epistemic competence beyond answer-level accuracy, as agents may produce correct answers through epistemically unsound reasoning processes.
Implications for Epistemic Competence Evaluation. These statistical patterns reveal several key insights that inform our evaluation framework. First, the high proportion of unclear and insufficient evidence (62.5% unclear, 52.7% insufficient) establishes a challenging information landscape where agents must demonstrate robust recovery capabilities. Second, the dominance of PlanFo omat on and Statee ses sment reasoning types (87.4% 1% combined) suggests that metacognitive capabilii- ties are central to effective informationnseeking yet our RQI analysis reveals these are precisely the reasoning types where agents struggle most. Third, the overwhelming prevalence of ungrounded rea- soning (82.7%) confirms that seddence-grounder reasoning is a critical competency gap that current agents fail to address, validating the necessity of process-level evaluation frameworks like Seek- Bench.
D LLM-AS-JUDGE FOR SEEKBENCH
This section presents the comprehensive LLM-as-judge used in SeekBench evaluate agent rea- soning, search behavior, and answer quality. The schema is organized into: reasoning types with grounding evaluation, search behavior, search result quality.
D.1 REASONING TYPE ANNOTATION AND GROUNDING EVALUATION
The reasoning annotation schema categorizes agent reasoning steps into four functional types and evaluates their grounding. This comprehensive evaluation helps identify both the cognitive function of reasoning steps and whether they are properly supported by evidence.
Reasoning Type Classification and Grouding Evaluation
You expert cognitive scientist and evidence-based critical thinking expert. Your task to classify the reasoning type of an agent's 'ss step and evaluate its grounding based *only* on the evidence it had at the time. The Agent's Goal (Original Question):
Context:
(question)
Evidence: The Search Results the Agent Had Access To:
Agent's Reasoning Text Analyze:
treasoning_text!
Task:
1. Classify the reasoning type:
StateAssessment: Assess the current knowledge state, usually identifying
knowledge gap.
PlanFormation: The agent is forming plan of action.
InformatiinSSyttesi:: Synthesize new information (from search results) to form a conclusion.
2. Evaluate grounding:
Extract the atomic factual premises from the step (skip meta/plan-on wording that contains no factual claim).
21

---

Published as a conference paper at ICLR 2026
Functional Type
Reasoning
StateAssessment
Def: Identify the knowledge gap Ex: "I don't know who won the 2020 election yet"
PlanFormation
Def: Form plan
Ex: "I should search for 2020 US election results"
Information isnnnthhssn
search conclusion Ex: "Based on CNN and BBC reports, Biden with 306 electoral votes"
Def: Synthesize
Search
InitielQuery
Def: First query reasoning chain
Ex: "2020 US presidential election winner" got: "Biden won"
RefinedQuezy
Def: Refine previous query based on new info
Ex: "2020 US election electoral college vote count" (added "electoral college" for precision)
FollowdgQuezy
Def: Follow-up question, direct refinement
Ex: "What was the voter turnuut percentage in 2020?" (sclated but different topic: turnout vs. winner)
RepeatQuery
Def: Same as previous query
Ex: "2020 US presidential election winner" (repeated same query)
Quality Attribute
Reasoning
Groundedness
Def: Claim supported by evidence
G: "Based on CNN reporting 306 electoral votes, Biden won election"
NG: "I conclude Biden won because he reccived more votes" (factual claim without evidence citation)
Search Result
Quality
Def: Contains sufficient info answer query Suff: "Biden won with 306 electoral votes Nov 7, 2020"
Insuff: "The 2020 election occurred"
Clarity
Def: Clear, not vague confusing presidential election" Unclear: "Biden or Trump won" (ambiguous)
Clear: "Joe Biden won the
Final Answer
Correctness
Def: Final answer is correct not
Figure
Annotation schema overview with definitions and examples. Abbreviations: Def=Definition. Ex=Example, G=Grounded, NG=Not Grounded, Suff=Sufficient, Insuff-Insufficient.
For each premise, find direct supporting span in the provided evidence. If no
exact or nerr-verbatim support exists, mark that premise as unmatched.
Decide the label with STRICT rules:
Grounded: ALL atomic premises are supported by explicit evidence spans. Not Grounded: ANY atomic premise lacks supporting span; OR the step contains only meta/plan text without factual premises.
Additional rules for grounding:
QUESTION anchor alone is NOT sufficient for Grounded: do not label as grounded solely for restating the task/intent.
anpperaaaiee/eempprralquaaaiiiii claims (e.g., last/first/olly, years, counts) require ex- plicit evidence spans.
Your Final Output:
greasoninggtypor  "StateAssessseent",
"grounding": "Grounded", "anchor_type"; "EVIDENCE",
22

---

Published as a conference paper at ICLR 2026
"justification", "brief explanation"
D.2 SEARCH BEILAVIOR ANNOTATION
The search annotation schema categorizes agent search queries into four behavioral types:
Search Behavior Classification
You an expert information retrieval specialist. Your task classify the type of search query issued by the agent.
Current Search Query:
(current_query]
Previous Search Query (if any):
(previous_query)
Task: Classify the search query type:
InitialQuery: The agent is issuing its first query in a reasoning chain. RefinedQuery: The agent is refining previous query based new information.
FollowUpQuery: The agent is asking follow-up question that is not a direct refine- ment.
RepeatQuery: A query that is the same the previous query.
Your Final Output:
"search_type"" "Initialquery", "justification": "brief explanation"
D.3 SEARCH RESULT QUALITY ASSESSMENT
This prompt evaluates the quality and clarity of search results retrieved by the agent. It helps identify when agents work with insufficient or ambiguous information.
Search Result Analysis Prompt
You expert data analyst. Your task is evaluate the quality of search result based on the query that produced it. Search Query: (query)
Search Result Documents:
(documents_jsee
Your Task:
Analyze the search result's sufficiency and clarity.
1. Information Quality: Does the result contain enough information to likely answer the user's implicit question in the query? Choose one:
Sufficient:
The answer seems to be present.
Insufficient: The answer is likely not here.
2.
Information Clarity: Is the information clear or does it create confusion? Choose one:
Clear: The information straightforward and addresses one subject.
Unclear: The results mention multiple distinct entities that could match the query (c.g.,
two movies with the same title) or the information vague.
23

---

Published as a conference paper at ICLR 2026
Your Final Output:
Your response must be single, valid JSON object with the following
attributes:
ietpmmatiionquuaiitte.. Either "Sufficient" or "Insufficient"
inforattion.laritty Either "Clear" or "Unclear" clarity.justificattionn Brief explanation for your clarity rating
E ACCURACY-LEVEL, PERFORMANCE
Agent
Overall F1 (%)
ASearcher
39.77
Search-RI
39.29
ReSearch
38.30
Few-shot
36.04
DeepResearcher
36.00
Base
33.50
CoT
32.72
ReAct
31.25
Table 5: Overall Fl performance across agent variants. Trained agents consistently outperform the base model, with ASearcher achieving the highest score.
Trepots the aggregate Fl scores across all evaluated agents. We observe that all trained agents outperform the base Qwen model, with ASearcher achieving the best performance (39.8%). Search- R1 and ReSearch follow closely, while Few-shot prompting and DeepResearcher attain comparable scores.
F GVDDENCE-GROONDED REASONING ANALYSIS
This section provides detailed analysis of evidence- evidence-grounded reasoning capabilities for Section 4.2 examining how agents ground their reasoning in retrieved evidence and identifying critical gaps in epistemic alignment. We present two complementary analyses: (1) cridence-conditioned reason- ing quality across different evidence states, and (2) type-specific reasoning capabilities that reveal heterogeneous grounding patterns across reasoning skills.
F.1 EVIDENCE-ALIGNEE REASONING: Do AGENTS GROUND THEIR INFERENCE IN WHAT THEYY KNOW?
To evaluate whether agents ground their reasoning in retrieved evidence, analyze the expected groundedness of reasoning steps conditioned on the agent's evidence state E E {0, 1,2). The quantity E,G,, E,, k] measures how reliably an agent produces well-supported reasoning at each evidence level k. An epistemically sound agent should avoid unsupported reasoning when
E O. provide partial grounding at 1, and fully leverage complete evidence when 2. This conditional analysis enables assessment of epistemic alignment: whether agents reason more confidently only when they possess sufficient evidence.
Base models show better epistemic alignment than specialized agents. Empirical results in Fig- ure TO reveal substantial variation across models. Most agents demonstrate appropriate behavior at E O, with E[G E 0) ~ 0.07-0.09, indicating minimal hallucinated reasoning. How- ever, SEARCI-RI exhibits significant epistemic misalignment, with elevated groundedness even under insufficient evidence ( 0.10 at E and 0.14 at E 2), suggesting grounded reasoning. In contrast, BASE and FEW-SHOT variants demonstrate the clearest evidence-condiitoned reason- ing, with groundedness rising from 0.49 (E 1) to 0.64 (E 2), indicating effective epistemic modulation. ASEARCHER also shows notable improvement (0.50 - 0.55), while RESEARCH and DEEPRESFARCHER stagnate around 0.47-0.51, failing capitalize stronger evidence. These
24

---

Published as a conference paper at ICLR 2026
Model
0.04
967
DeepResearche
84
ReSearc?
055
SearchAl
ASearchar
Ger
Plarundeed
6.99
8.7
5.0 6.7
6.16
0.0
Figure 10: Fvidence-comddiiooed reasoning quality for evidence state k E 0, 1, across search agents. Bars denote 95% confidence intervals. The quantity reflects the expected groundedness of reasoning steps given the epistemic evidence state E. Higher values E 2 indicate effective evidence utilization.
Figure 1I: Lyve-Level Conditce-Coddttooned Groundedness. Expected groundedness for each reasoning type (Information Synthesis, Plan Formation, State Assessment) under evidence levels E 0, 1,2. Bars show 95% confidence intervals. Models exhibit heterogeneous capabilities in grounding specific reasoning skills available evidence.
results demonstrate the necessity for evaluation metrics like RQI that isolate whether reasoning reflects the agent's actual knowledge state.
Case Study: Correct Answer with Ungrounded Reasoning, We examine case where ReSearch agent correctly answers "Who won the first celebrity big brother on channel 5?" despite completely ungrounded reasoning. After retrieved the first evidence, the agent retrieves conclusive evidence stating "Celebrity Big Brother I... concluded on 16 March 2001 when comedian Jack Dee was crowned the winner." Despite having the answer, the agent ignores this evidence and conducts unnecessary searches, stating: "I need to clarify which Big Brother series I am referring to... Now, have find out the winner of that show." The agent eventually answers "Jack Dee" correctly, through an epistemically unsound process. This demonstrates why accuracy metrics alone fail to capture critical reasoning deficiencies.
F.2 TYPE-SPECIFIC EVIDENCE ALIGNMENT: WHICH REASONING SKILLS ARE EVIDENCE- EVIDENCE-GROUNDEE""
We further decompose agent reasoning groundedness by reasoning type 7 {IS, PF, SA), leverag- ing the Type-Level Reasoning Quality Index from Definition 3.3 and the evidence-state decomposi- tion in Equation (6).
Figure I1 illustrates the groundedness of each reasoning type across three evidence states. Our
analysis reveals several significant patterns:
First, Information Synthesis (IS) demonstrates the strongest wvidence-responsivenn across all models. With complete evidence (F=2), IS steps achieve superior groundedness, indicating robust
25

---

Published as a conference paper at ICLR 2026
ASearcher
CoT (Qwen)
Few-shot (Qwen)
ReSearch
Base (Qwen)
DeepResearcher
ReAct (Qwen)
SearchR1
0.8
reached
0.6
Fem-shal (Qwan)
Deepleeweenceer
0.4
Base (Owen)
0.2
0.0
10
Turn
Comprehensive Recovery Analysis. Evidence Recovery Function (ERF) showing the cumula- tive fraction of traces reaching sufficient evidence state (E' = 2) across all evaluated agents, including RL- trained agents (ASearcher, Search-R1, ReSearch, DeepResearcher), base models (Base), and prompting strate- gies (Few-shot, CoT, ReAct), Steeper curves indicate faster recovery from low-quality evidence.
Figure 12:
capabilities in aggregating retrieved information. Even with partial evidence (E=1), agents maintain moderate IS groundedness, suggesting effective utilization of incomplete knowledge.
In contrast, Plan Formation (PF) and State Assessment (SA) exhibit substantially lower ground- ednesy even with complete evidence. For PF, only ASEARCHER and RESEARCH exceed 0.5 at E=2, while others (e.g., SEARCH-RL. DEEPRESEARCHER remain below 0.4, revealing fragile macksig processes despite available knowledge. Similarly, SA demonstrates critical limita- tions: although scores improve under E=2, most models underperform relative to IS, with several agents (e.g., SEARCH-RI,
showing minimal evidence seddenne-responsivenesss between E=1 and 2 Notably, SEARCH-RI performs comparatively well for IS across all evidence levels but demonstrates exceptionally poor grounding for PF and SA (0.15 and 0. at E-2, respectively), suggesting specialized evidence synthesis narrowly constrained reasoning capabilities.
These findings demonstrate that only specific reasoning capabilities (particularly synthesis) are con- sistently grounded in retrieved information. This underscores the necessity for developing evidence- grounded reasoning policies, especially for higher-order cognitive functions like plan formation and state assessment that currently show significant epistemic disconnection from retrieved knowledge.
G RECOVERY ANALYSIS
This section provides detailed analysis of evidence recovery capabilities, extending the main text analysis (Section 4.3) with additional agent comparisons including Chain-of-Thought (CoT) and ReAct prompting strategies.
Extended Agent Comparison. Figure 12 presents the Evidence Recovery Function (ERF) for all evaluated agents, including the additional CoT and ReAct prompting strategies. The results reveal a clear performance hierarchy: SEARCH-RI demonstrates the best recovery performance, followed by CoT, RESEARCH, ReAct, FEW-SHOT, DEePRESFARCHER, and BASE (in descending order). Notably, CoT outperforms several RL-trained agents (RESEARCH, DEEPRESEARCHE and base models, achieving the accond-highest recovery rate. ReAct also shows competitive recovery perfor- mance, outperforming FEW-SIIOT, DEEPRESEARCIIG RL-trained agents like ASEARCHER and SEARCH-R- achieve superior recovery through explicit search adaptation mechanisms, prompting strategies like CoT can also demonstrate effective idence recovery capabilities, potentially through their structured reasoning approach that enables better query refinement.
and BASE. This finding reveals that while
26

---

Published as a conference paper at ICLR 2026
H EVIDENCE CALIBRATION ANALYSIS
To further understand the epistemic calibration capacity of RL-trained agents, we analyze four RL- based agents: ASEARCHER, SEARCH-RI. RESEARCH, and DEEPRESFARCHER. Each model ex- hibits distinct behavior patterns in groundde answering.
RL Agents: Plansweri E) model
Agents: Plcorrect/answer.BB by model
FeSearth
fadeurch
Searchtt
Seare13
Despllxecccer
Perowelf
0.3
0.3
0.2
Figure 13: Evidence-Calibratte Answering by RL Agents. Left: Answering propensity for each agent under different evidence levels (E=0/1/2). Right: Answering accuracy for each RL agent under different evi- dence levels (F=0/1/2).
As shown in Figure 13 (Left), all agents show increased answering with stronger evidence, sug- gesting basic epistemic alignment, but they vary in evidence gradient-the increase from E=0 to E=2. ASEARCHER and DEEPRESEARCHER exhibit higher gradients, indicating stronger sensitiv- ity epistemic evidence. However, all agents maintain relatively low absolute response rates even with sufficient evidence. ASeARCHER and DEEPRESEARCHER reach 19-21%, %, while SEARCH-RI remains 8.5%, suggesting more conservative behavior.
Calibration Accuracy As noted in Section E ASEARCHER achieves the highest overall F1 score, followed by SEARCH-Ri, RESEARCH, and DEEPRESFARCHER.
To evaluates whether agents can defer or respond based on the epistemic adequacy of observed evidence. As shown in Figure across trained agents, we observe distinct calibration profiles:
ASEARCHER demonstrates superior evidence sensitivity coupled with high accuracy. re- sponds predominantly when sufficient evidence is available (F=2) at substantial rate (19.3%) while minimizing overconfident responses. As shown in Figure 13 (Right), it achieves the sec- ond highest conditional accuracy P/correct/answer, E = 2). This strategic alignment between evidence-based answer timing and correctness yields optimal performance.
SEARCH-RI achieves the highest accuracy when answering with complete evidence (Fig- ure 13 Right), but exhibits extreme conservatism, answering in merely 8.5% of high-evidence states (Figure 13 Left). While this demonstrates exceptional calibration awareness, the exces- sive caution significantly constrains overall performance, representing clear trade-off between coverage and precision.
conclude that well-calibrated agent behavior requires satisfying two critical conditions: (1) de- ferring responses until evidence strength reaches sufficient levels (e.g.. E=2), and (2) producing correct answers when responding, demonstrating effective utilization of the available evidence. This dual requirement highlights the challenge of balancing epistemic caution with informational utility.
RL Training Reduces Calibration Errors. Table 6 provides a detailed breakdown of calibration performance across individual RI-ttriined agents. While all RI. agents show substantial improve- ments over base models in reducing overconfident answering, ASEARCHER achieves the lowest overall calibration error (0.302), closely followed by DELPRESEARCIER and RESEARCH (both 0.305). Notably, SEARCI-RI I exhibits the most conservative behavior with the lowest overconfident answeringrate (0.226) but highest overcautious rate (0.187), suggesting a trade-off between differ- ent types of calibration failures. These results demonstrate that RL training consistently improves evidence-band based decision making, though specific training approaches yield different calibration pro- files.
27

---

Published as a conference paper at ICLR 2026
Table Calibration Error Breakdown by Agent Type. Trajectories categorized overconfident answering (answering before sufficient evidence), overcautious abstention (failing to answer despite strong evidence), and overall calibration error. Lower values indicate better calibration. ASearcher show the lowest calibration errors. Bold indicates best performance in cach category.
Model
(1) Overconfident
(2) Overcautious +
(3) Calibration Error
Base
0.631
0.030
0.329
Few-shot
0.511
0.024
0.317
CoT
0.731
0.006
0,351
ReAct
0.660
0.012
0.336
AScarcher
0.343
0.044
0.302
DeepResearcher
0,461
0,048
0,309
ReSearch
0.406
0,047
0.305
Search-R1
0.226
0.187
0,319
Few-shot
RL Agents
1.0
yei 11
Alneady High
Fraction arcwered by 2300
0.8
0.6
0.4
0.2
0.0
Figure 14: Evidence-Condittoned Answer Timing Analysis. For each model group, plot the fraction of answered trajectories time (x-axis: number), split by whether high evidence (F=2) has been observed. agents defer until sufficient evidence, orange curve (already high) should rise earlier blue curve (not yet high). However, all models show little separation between the two, confirming widespread overconfident answering behavior.
Do agents defer answering until strong evidence arrives? To evaluate whether agents appropri- ately delay answering until they have observed sufficient evidence, we analyze the timing of answers across different evidence states. Figure 14 presents temporal analysis comparing the cumulative fraction of answers over time for two distinct trajectory groups: those where agents have already encountered strong evidence (E=2) versus those where they have not.
In an ideally calibrated system, we would expect agents to predominantly answer after observing strong evidence, resulting in clear separation between snjeectories-spccffieee curve (evidence already observed) and a lower blue curve (evidence not yet observed). However, our analysis reveals that BASE and FEW-SHOT models demonstrate minimal separation between these curves-indicating that answers are generated with similar timing regardless of evidence availability. RL-trained agents, while showing marginal improvement, still exihibits overconfident answering in 76.5% of trajectories before reaching sufficient evidence (E=2).
higher orange
This finding highlights a fundamental calibration deficiency: current models consistently make over- confident decisions without aligning their answer timing with epistemic sufficiency.
AGENT SYNTHESIS: LEVERAGING EPISTEMIC COMPETENCIES FOR ANSWER GENERATION
Our comprehensive evaluation reveals several critical insights into agent capabilities: ASEARCHER demonstrates superior performance in evidence acquisition and recovery mechanisms (achieving the highest overall Fl score), while SEARCH-RI exhibits exceptional proficiency in information synthesis (with RQI score of 0.63) coupled with minimal overconfident answering behavior (as discussed in Section 4.4). This observed specialization of epistemic competencies motivated
28

---

Published as a conference paper at ICLR 2026
Policy (P)
Overall
Synth. (S) Search-R1 ASearcher DeepRes.
ReSearch
Base
Fewshot
CoT
ReAct
Avg. AF1
Search-RI
-0.08
+0.19
+1.66
+3.50
+1.10
+6.0
+5,9
+2.61
ASearcher +0.38
-0.73
-0.39
+0.93
+1.46
+5.9
+5.4
+1.85
DeepRes. -0.63
+0.20
+0.00
+2.24
+1.45
+6.3
+5,9
+2.21
ReSearch -0.42
+0.13
+0.36
+2.99
+1.89
+5.5
+4,9
+2.19
Table 7: Agent Synthesis Performance. Each cell shows F1 score improvement (AF1) when using agent (S) as synthesizer generate answers based on evidence collected by column agent (P), Positive values indicate the synthesizer improved upon the original policy's performance. Search-R1 demonstrates the highest overall improvement (+2.61 F1) across all evidence sources.
us to explore the potential of agent synthesis where we leverage one agent's evidence collection capabilities as input for another agent's answer generation process.
Synthesizer Evaluation Methodology. To test this hypothesis, we evaluate each agent synthe- sizer by providing it with the complete reasoning traces, search results, and evidence from other agents' trajectories. The synthesizer's task is to generate a final answer based solely on this infor- mation, without performing additional searches. This setup isolates the agent's 's ability to synthesize information from existing evidence, separate from search and retrieval capabilities.
Search-R1 Emerges the Superior Synthesizer. As shown in Table Search-R1 delivers the largest average Fl gain (+2.61), significantly outperforming other agents such as ASEARCHER (+1.85), DEEPRESEARCIES (+2.21), and RESEARCH (+2.19). This result aligns with our earlier findings that Search-RI I exhibits strong information synthesis capabilities (RQI = 0.63 for Informa- tion Synthesis steps) and conservative answering behavior (lowest overconfident answering rate), and it persists even when synthesizing CoT or ReAct evidence traces.
The superior synthesis performance of Search-RI can be attributed to its specialized reasoning capabilities. Despite its low overall RQI (0.08), Search-R1 demonstrates particular strength in in- formation synthesis when provided with clear evidence. Its conservative answering behavior, while limiting coverage in standalone scenarios, becomes an advantage in synthesis tasks where can carefully evaluate and integrate information from multiple sources before providing a final answer.
Hidden Behaviors: How Accuracy-Leeel Evaluation Obscures Profound Reasoning Capabil- ities. Surprisingly, several agents demonstrate stronger capabilities as evidence sources than their standalone Fl scores suggest. BASE evidence collection achieved substantial Fl gains when paired with other models for answer generation (up to +3.50 F1 improvement with Search-R1), despite having the lowest standalone F1 score (33.5%). Similarly, CoT and ReAct, which show lower standalone performance (32.72% and 31.25% Fl, respectively), enable even larger synthesis im- provements (up to +6.3 Fl with CoT and +5.9 Fl with ReAct). This reveals that final accuracy metrics can be misleading when used in isolation, as they obscure critical process-level competen- cies. These agents may be collecting high-quality evidence producing well-structured reasoning traces but struggling with final answer synthesis, a nuance completely missed by traditional eval- uation methods that focus solely final answer accuracy rather than decomposing the reasoning process into its constituent competencies. The agent synthesis framework enables us to identify these hidden strengths and leverage complementary capabilities across different agent architectures.
Implications for Agent Design. These findings demonstrate that our benchmark and evaluation framework enables modularization of agent-specife. epistemic competencies to create more effec- tive information-seeking systems. This represents a significant advance for process-level evaluation of agents compared with traditional answer-Jevel evaluation, enabling the identification and combi- nation of complementary strengths across different agent architectures.
29

---

Published as a conference paper at ICLR 2026
J INFERENCE-TIME FEEDBACK USING EPISTEMIC COMPETENCIES
In this section, leverage the L.LM-as-judge framework provide inference-time feedback signals
to improve agent behavior.
Inference-Time Feedback Signals. We augment ASEARCHER-7B with inference-time feedback signals derived from our epistemic competency framework. Specifically, we incorporate three process-level supervision signals computed at each turm: (1) groundedness feedback indicating whether reasoning steps are supported by retrieved evidence, (2) evidence state signals (E,, . E {0, 1,2)) encoding the sufficiency and clarity retrieved evidence, and (3) calibration feedback on whether the agent's s answering behavior aligns with evidence quality. These signals are computed using our validated LLM-as-jun -uddgge framework and provided as real-time guidance during inference, enabling the agent to adjust its behavior based process-level epistemic feedback without training.
Results and Implications. Our experiments reveal that incorporating epistemic feedback signals at inference time significantly improves agent performance. The augmented ASEARCHER-7B achieves a 8.4% increase final F1 score compared to the baseline without feedback signals. More importantly, observe substantial improvements across all three epistemic competencies: (1) groundedness (RQI) increases by 13.3%, indicating better evidence-supported reasoning, (2) recovery (ERF) improves by 6.5%, demonstrating more effective adaptation insufficient evi- dence, and (3) calibration (CE) decreases by 5.8%, showing better alignment between answering behavior and evidence quality.
K EVALUATION ON GAIA
In this section, we evaluate the epistemic competencies of ASearcher and WebSailor on GAIA, and GPT-5-mini GATA with Web search tools.
We have evaluated WebSailor (Li et al. 2025a) and ASearcher (with web browsing + visit capabili- ties) GALA with Pass@2 scores across different model sizesß Our framework's tool selection follows epistemic scope rather than tool diversity. We include web browsing and visit capabilities (as in WebSailor) because these require epistemic evaluation-the quality of retrieved information depends on reasoning processes and evidence interpretation, not objective correctness.
Answer-level Performance. At 7B scale, ASEARCHER achieves 16.5% while WEBSAILOR achieves 18.2%. At 32B scale, ASEARCHER achieves 29.2% while WEBSAILOR achieves 32.4%. In addition, we evaluate GPT-5-mini on GAIA, achieving 15.7% Pass@2.
Epistemic Metries Analysis We apply our epistemic competency framework to analyze ASEARCHER and WEBSAILOR behavior across the three metrics. Table g presents the epistemic metrics. For 32B scale, ASEARCHER demonstrates superior groundedness (RQI = 0.28) and re- covery (ERF 58% by Tum 8) compared to WEBSAILOR. However, WEBSAILOR shows better calibration (CE 0.31) than ASEARCHER (CE = 0.35), indicating more conservative answering behavior. These metrics reveal distinct epistemic competency profiles that explain the performance ASEARCHER'S strength evidence acquisition and recovery compensates.
differences:
We then apply epistemic competency framework analyze GPT-5-mini's behavior across the three metrics: groundedness (RQI), recovery (ERF), and calibration (CE). Based on this analysis, provide inference-time feedback signals (as described in Appendix [], which improves the final score to 22.5%, a 43% relative improvement.
hhtpss/hugggngffac.cco www.aaatsscc//ggaaaaeeenncccaace hewchaarkhchhaAAAA
Note that WebSailor does not have an open-sourced 14B variant, evaluate the available model sizes
30

---

Published as a conference paper at ICLR 2026
Table 8: Epistemic Metries for ASearcher and WebSailor GAIA (7B and 32B) and GPT-5-mini GAIA with Web search tools. RQI (groundedness) ERF recovery rate by Turn and CE (calibration error, lower better).
Model
7B
32B
GPT-5-mini
RQI
ERF
CE
RQI
ERF
CE
RQI
ERF
CE
ASEARCHER
0.22
42%
0.38
0.28
58 %
0.35
WEBSAILOR
0.15
35%
0.33
0.19
45%
0.31
GPT-5-mini
0.45
42%
0.34
L DISCUSSION: UNDERSTANDING METRIC TRADE-OFFS
Our evaluation framework reveals distinct trade-offs between epistemic competencies that traditional accuracy-only evaluation fails capture. These insights are critical for both interpreting agent performance and designing effective systems.
The Cecuracy-Reaooingg Trade-off. We observe particularly concerning inverse relationship be- tween answer accuracy (F1) and reasoning quality (RQI) among RL-trained agents. While RL training improves final answer correctness and calibration. it simultancously degrades evidence- grounded reasoning quality. This reveals a fundamental tension: agents can be optimized to produce correct answers without developing sound reasoning capabilities critical consideration for Al safety and interpretability.
Calibration vs. Reasoning Quality. RI-ttrained models demonstrate better calibration (lower CF) despite worse reasoning groundedness (lower RQI), indicating that well-calibrated agents may still produce poorly justified reasoning. This highlights the necessity of evaluating both when to answer (calibration) and how to reason (groundedness) as separate competencies.
Implications for Agent Selection and Design. These tradc-off's directly impact deployment deci- sions: applications requiring high accuracy may favor ASEARCHER despite reasoning limitations; those requiring interpretable reasoning may prefer Few-shot models despite lower accuracy; and ap- plications demanding both may benefit from agent synthesis approaches (Section 4.5) Future agent development should explicitly address these trade-off's through multi-objective optimization and po- tentially modular architectures that separate evidence acquisition, reasoning, and decision-making components rather than optimizing solely for accuracy. Furthermore, our epistemic competency metries (RQI. ERF, CE) can drive automatic orchestrators that monitor agent behavior in real- real-time and dynamically switch to the best-suited agent for each stage of the codomattionssckiing process. For example, an orchestrator could deploy agents with strong recovery capabilities (high ERF) dur- ing initial evidence gathering, then switch to calibrated synthesizers (low CE) once evidence quality stabilizes, leveraging the complementary strengths identified through our framework. This metric- driven orchestration represents a practical application of our evaluation framework for building more effective multi-agent systems.
M EXTENDED VALIDATION: GENERALIZABILITT ACROSS DIVERSE TOOLS AND MODELS
To assess the generalizability of our framework, we conducted the following validation using traces from of-te-of-the-ar art multi-tool agents challenging tasks.
1. Data and Trace Sampling, We obtained traces from a diverse set of state-of-the-ee the-trrrrt agents: Deepresearch-300 ASearcher-300 Each agent, equipped with web browsing, Python interpreter, and website visitation tools, was eval- uated on four benchmarks: WebWalker (Wu et al. 2025). GAIA (Mialon et al. 2023), BrowseC-
GPT-4o, GPT-5, and Claude Sonnet 4.5.
31

---

Published as a conference paper at ICLR 2026
Table 9: LLL Agreement Across Diverse Tool Types. Scores for Type and Groundedness pertain tool input actions (such search queries), while Clarity and Sufficiency scores assess the tool output (i.e., produced evidence).
Tool
(Type)
(Groundedness))
(Clarity)
(Sufficiency)
Web Browse / Visit
0.80
0.71
0.68
0.70
Code Execution
0.77
0.76
0.73
0.74
omp (Wei et 2025). ) and KBench-DeepSearfff For every combination of agent and benchmark, we randomly sampled 10 trajectories for analysis.
Judge Annotation Protocol and Adaptation. We applied our LLM-as-judge to the 200 traces with gpt-4. 1-iii.. The core annotation schema and prompts for reasoning types and quality at- tributes remained unchanged. The only adaptation was in the evidence-parsing logic: the judge was instructed to treat the direct output of any tool as the 'retrieved evidence' for the subsequent step. For example, the output of a Python Interpreter call becomes the evidentiary basis for evaluating the groundedness of the agent's next reasoning step.
Human Annotation and Results. From the L.LM-as-judge annotations, we drew stratified ran- dom sample of 65 traces, stratified by agent (model) tools combination to ensure proportional repre- sentation across different tool usage patterns and models. The sample size ensures 95% confi- dence level with a +10% margin of error, remain- ing cost-effective annotate and evaluate our pro- posed schema. The sampled traces exhibited a di- verse tool distribution, with search and visit opera- tions accounting for approximately 99% of tool us- age (69% and 35.2% respectively, fig. [5]. Expert human annotators independently annotated the sam- pled traces, with human-LLM agreement Cohen's 0.65 (Table
search
64.0%
0.8%
python
35.2%
visit
Tool Distribution. Tool usage dis- tribution across different agents and benchmarks. Search and visit operations accounting for approx- imately 99% of tool usage.
Figure
4. Stress-Testing the Judge Ambiguous Rea-
soning. We also assessed the judge's ability on ambiguous reasoning, we select an additional 20 steps where the final answer is correct but reasoning is ungrounded. For example, the agent in- jected "Indian" into the query despite no geographical constraint in the question. This represents an ungrounded inference from internal parametrie knowledge, which is critical epistemic flaw that cannot be verified.
Example: Ungrounded Inference from Internal Knowledge
Question: "An animated series for children first premiered America between 2001 and 2010. The creator of this show is a physicist who obtained their PhD from university established between 1940 and 1960... Name the reviewer."
Agent's Search Query: "Indian physicist created animated series America 2000s"
Key Issue: agent injected "Indian" into the query despite no geographical constraint in ques- tion. This represents an ungrounded inference from internal parametric knowledge, which a critical epistemic flaw that cannot be verified.
Label: LLM judge and human annotator both label this "Not Grounded".
Re-evaluation showed that human-LLM agreement remained acceptable (n R 0.70 for grounded- ness, k. 0.66 for clarity, x 0.65 for sufficiency), confirming our framework's robustness in exposing even nuanced reasoning failures.
32

---

Published as a conference paper at ICLR 2026
N USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) played significant role in this research, warranting disclosure of their contributions. LLMs were extensively used as annotation helpers to develop our epistemic competency schema, analytical assistants for reasoning trace analysis, and as automated judges (detailed in Section D). They contributed significantly to the writing process by generating initial drafts of technical sections and assisting with revision, and influenced research ideation by suggest- ing evaluation metrics and identifying methodology gaps. All LLM-ggeneratd content underwent rigorous human review and validation, with human authors verifying analyses, validating schemas through expert review, and thoroughly editing all contributions. While LLMs served as powerful assistive tools, all final decisions regarding research direction, experimental design, result interpre- lation, and manuscript content were made by human authors.
33


Reviews:
[{"review_id": "0m17TEcX4l", "reviewer": "Reviewer_a7jj", "summary": "The paper introduces SeekBench, a benchmark for evaluating the epistemic competence of LLM-based search agents. Rather than judging agents solely by final-answer accuracy, SeekBench analyzes step-level response traces—reasoning, search, evidence, and answer—to assess how agents acquire, evaluate, and act on knowledge. The benchmark enables fine-grained diagnostics of three core competencies: (1) evidence-grounded reasoning, (2) recovery, and (3) calibrated answering. Using this schema, the authors design interpretable metrics and show that widely used accuracy-only evaluations can mask important behavioral differences. SeekBench thus provides a principled, scalable framework, supported by high human agreement and aligned LLM judges.", "questions": "1. Can you provide the examples of poor/partial/good evidences? I checked the actual prompts in Appendix, but I think it can be very subjective (especially for partial evidences) even for humans. I assume there can be some gray areas to divide these categories, which should be discussed in th paper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novelty of the research question. I do think the step-by-step debugging of the retrieval-intensive search agents is an important yet unexplored direction in current answer-only evaluation frameworks.\n\n2. Clarity of the proposed framework. The framework seems a bit complicated (since it contains many evaluation aspects), but well documented, making it easy to understand the overall process and the details of each evaluation component.\n\n3. Effective visual communication. Figures and plots are of high quality and well chosen. These improve interpretability and aid readers’ comprehension of the empirical findings.", "weaknesses": "1.  Rationale behind the metric\n\nI get the rationale behind the metric and do think they are well-designed to capture groundness, recovery, and calibration. However, a few questions remain. \n\nFirst, why do we have to measure the groundness of an insufficient evidence state (E=0)? In other words, in Eqn 4, the RQI is decomposed w.r.t. evidence states E while accounting for E=0. Isn't it more reasonable to only consider E=1 and E=2, since groundness on insufficient or incorrect evidence is not meaningful? \n\nSecond, in Eqn 8 to define ERF, why not account for the length of the traces T_i? In my point of view, adding 1/T_i term seems more natural if the goal is to measure \"how fast the agent can escape from poor evidence\", since it then becomes independent of the trace length. For example, if there are two gold traces, A with 3 steps and B with 10 steps, having both ERF(3), the meaning can be very different since it can mean faster escape in trace B than in trace A.\n\n\n2. Unclear points on identifying failure patterns & Lack of human annotators\n\nIn Section 3.1, the authors seem to identify failure patterns in agents' traces to develop the benchmark evaluation scheme. However, which models & agents have been observed is unclear. Are the models in different sizes & different families (Llama, GPT, Qwen ... ) accounted for, or is only Qwen used (the ones used in the experiments)? Furthermore, three human annoataors seems insufficient to support human aggreement with LLM judges.\n\n\n3. Lack of experimental baselines\n\n\nThe paper only evaluates two Qwen models, which limits the effectiveness of experimental results to support the claims. Also, while the paper compares \"base\", \"few-shot\", and \"RL-trained\" methods, prompting methods such as CoT and ReAct should be included.\n\n\n4. Insufficient cost analysis\n\nWhile using LLM-as-Judge provides a scalable approach (and high human agreements according to the paper's claim), it may incur a significant cost. The paper should include a cost analysis on time and API costs (in case of using proprietary models like GPT-4 or GPT-5 as used in the paper) and discuss potential limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3x82CfFCQ", "reviewer": "Reviewer_CVjV", "summary": "This paper introduces SeekBench, the first benchmark for evaluating the epistemic competence of Large Language Model (LLM) search agents through step-level analysis of their reasoning traces. SeekBench comprises 190 expert-annotated agent traces generated by LLM agents on open-domain QA tasks, and formalize three metrics (Groundedness, Recovery, Calibration).", "questions": "What is the specific token consumption for LLM-as-judge?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper obtains some findings based on SeekBench such as 1) RL-trained agents achieve higher answer accuracy but exhibit lower reasoning groundedness than few-shot or base models. 2) All agents struggle with Plan Formation and State Assessment, though they perform better at Information Synthesis.", "weaknesses": "- The current epistemic metrics (Groundedness, Recovery, Calibration) rely on binary judgments (e.g., grounded vs. not grounded), which may overlook nuanced or intermediate reasoning states—such as partially supported claims or ambiguous evidence.\n\n- As illustrated in the paper, RL typically optimizes only for the correctness of the final answer, while ignoring the goals at the process level. If adding supervision signals during the RL training process can solve this problem, how can the cognitive abilities proposed in the paper be integrated into the training process?\n\n- The current trace schema focuses on textual search and reasoning steps but does not account for richer agent behaviors—such as invoking code interpreters, calling external APIs, browsing web pages, or analyzing structured data (e.g., tables, JSON). it is important to enhance its applicability of SeekBench to real-world  search agentic systems.\n\n- While the expert-annotated subset (190 traces) ensures high-quality grounding, its limited size may hinder coverage across diverse question types and failure modes. Moreover, scaling annotations via LLM-as-judge—despite strong inter-annotator agreement—risks propagating systematic biases or blind spots inherent in the judge model itself, especially on adversarial examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yxRqK1EslT", "reviewer": "Reviewer_HSxE", "summary": "This paper proposes the SeekBench benchmark: the first process-level benchmark for LLM search agents, decomposing agent traces into four steps (reasoning, searching, evidence, and answering). It establishes an operational framework with metrics, formalizing three core cognitive abilities—\"evidence-based reasoning,\" \"adaptive evidence recovery,\" and \"evidence-aligned calibration\"—into measurable indices and designs precise, interpretable metrics to quantify these capabilities. Based on these metrics, the evaluation is conducted across seven QA benchmarks (28,493 traces), revealing that reinforcement learning (RL) agents excel in evidence gathering but exhibit weak reasoning. Additionally, the study highlights that standard accuracy metrics fail to distinguish specific agent strengths (e.g., Search-R1's synthesis capability vs. base models' reasoning ability), while combining agents with complementary strengths improves overall performance.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Addressing the limitation of \"answer-only\" agent evaluation, this work focuses on cognitive behaviors (e.g., evidence usage, uncertainty decision-making) during retrieval and reasoning, offering a more comprehensive assessment aligned with real-world agent capability requirements.\n2. It introduces theoretical concepts like \"evidence states\" alongside concrete, quantifiable metrics. Large-scale evaluations yield actionable findings (e.g., combining agent strengths enhances performance), providing practical guidance for future research and applications.", "weaknesses": "1. Lack of dataset details: The experiments mention 28,493 traces from seven QA benchmarks, and the appendix briefly outlines the dataset selection process. However, the main text (and appendix) lacks detailed sample descriptions or a dedicated section for statistical analysis of the data.\n2. Insufficient validation of core metrics: The three proposed metrics—Reasoning Quality Index (RQI), Evidence Recovery Function (ERF), and Calibration Error (CE)—are inadequately justified, undermining their validity and innovation. Key gaps include:\n\n    1）Weak motivation: Only definitions are provided without explaining the limitations of existing metrics (e.g., accuracy, F1) in assessing cognitive behaviors or comparing them to related work. Theoretical foundations (e.g., cognitive science frameworks) are absent, weakening the design rationale.\n\n    2）Missing case studies: A case analysis in the introduction could clarify why the three-stage division better evaluates agent capabilities.\n\n    3）No superiority demonstration: While proposing a process-level framework, the paper fails to prove its advantages over task-bound evaluation methods—lacking dimensional comparisons (e.g., comprehensiveness, interpretability) or experiments revealing overlooked flaws. The claim that \"three-stage evaluation is superior\" lacks validation.\n3. Underdeveloped agent combination insight: The finding that combining agent strengths improves performance lacks deeper exploration—no discussion of viable strategies, applicable scenarios, or optimization potential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I58HkHqc83", "reviewer": "Reviewer_9ECN", "summary": "This paper proposes SeekBench, an evaluation framework for analyzing the intermediate reasoning and search process of search-based LLM agents. The authors construct a framework centered on 3 competencies (groundedness, recovery, and calibration), proposing various cases of each competencies and associated metrics.", "questions": "- LLM-as-judge agreement with human annotations is presented at an aggregate level, without per-metric breakdowns. What is the per-metric breakdown? Are judges more reliable for certain metrics? If yes, can we still reliably trust judges for **all** metrics?\n- What is approximate cost of annotating one search agent in terms of LLM-as-judge API calls? It seems that each intermediate step in an agent requires multiple API calls.\n- The authors have a unique opportunity to more thoroughly benchmark LLM-as-a-judge models on the human annotations to determine the best model based on cost-performance tradeoffs. Why were GPT-4.1, 4.1-mini, and GPT-5 chosen? Why not evaluate GPT-5-mini,nano, or models like Claude Sonnet, Gemini 2.5 Pro, or more capable local models that are “free” (in terms of API cost)? Further, what reasoning effort was GPT-5 evaluated with? To inform practitioners, how does this performance vary with reasoning effort setting?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Evaluation beyond outcome is an important topic: Validating that the processes search agents take in finding the final answer is necessary for reliability, transparency, and can be a valuable feedback signal for improving future agents.\n- For search-based agents, the creation of the competencies, annotated features, and metrics seems reasonable, involving three rounds of review with humans and LLM-as-judges. This results in an evaluation schema that is both principled and scalable, with automatic evaluators exhibiting high levels of aggregate-agreement with humans.\n- The analysis identified clear shortcomings of RL-trained search agents: outcome-based RL training leads to better calibrated agents (i.e., agents tend to produce answers when the evidence state is both clear and sufficient to answer the question). However, the intermediate reasoning steps produced by agents are not necessarily grounded, reflecting the outcome-driven nature of RL training.", "weaknesses": "- In my opinion, many parts of the abstract and introductory sections are insufficiently clear, and largely have to do with the repeated statement that SeekBench is a process-level benchmark consisting of 190 traces with 1800+ annotated steps (i.e., in abstract, L88, L97). This framing lead me to believe that SeekBench was a benchmark for evaluating process-level evaluators, like those present in math domains, e.g., ProcessBench. **In reality, an annotation schema is derived from said traces, and used to validate automatic evaluation approaches, like LLM-as-judges**. This should be made clearer to the reader, especially as the evaluation process in Section 4 does not really directly use these annotations.\n- The paper seems to only analyze search agents that can use a single web-search tool. This may not be representative of state-of-the-art search agents, e.g., [1,2], which are capable of multiple tools, such as using a diverse array of tools like code interpreters or more complex forms of search like web browsing. It’s unclear how SeekBench analysis holds when agents are given toolkits more representative of real-world use-cases. I don’t think states with other tools can easily be folded into existing competencies, nor do I think the existing framework can be applied directly to other tools.\n- Evaluated baselines: \n  - It seems that only agents of 7B model size are evaluated, even though many baselines have larger, more capable model variants. For example, ASearcher has 14B and 32B variants, trained from Qwen2.5-14B and QwQ-32B, respectively. \n  - Furthermore, newer, more capable tool-calling baselines, such as WebSailor [2] (and v2 variant) are missing. These missing baselines may be a result of the above weakness, as they are typically trained to use multiple tools.\n  - Lastly, API models with search tool-calling ability (e.g., OpenAI models) are not analyzed.\n  - This weakness is critical in my opinion. Because this paper mainly performs analysis on the behavior of search-based agents, it should aim to cover as many search agents are reasonably possible. But a critical dimension, agent model size, is missing! How many of the unearthed agent shortcomings can be mitigated or solved simply by scaling up the agent size?\n\n\n[1] https://arxiv.org/abs/2509.06283\n\n[2] https://arxiv.org/abs/2507.02592", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKVevLHYg4", "reviewer": "Reviewer_ESgZ", "summary": "This paper proposes SeekBench, a benchmark to evaluate LLM search agents on epistemic competence of three dimensions/metrics: reasoning groundedness, search recovery, and answer calibration.", "questions": "1. What are the raw data source(s) and domain(s) for constructing SeekBench?\n2. What does each annotated feature in Table 1 mean?\n3. To clarify, the human/LLM annotators will label the correctness of each trace i, as well as the clarity and quality of the retrieved evidence at each turn t in each trace, right? What exactly is \"quality\" defined here?\n4. Section 3.3.3 Definition 3.5 (Calibration Error (CE)): Why should the ideal policy answer if and only if the evidence is sufficient? Whether the explicit evidence is sufficient or not also depends on the internal knowledge of the policy, so it would also be acceptable if the policy answers correctly without much retrieval, especially for easy questions.\n5. Section 4 \"We evaluate the agents on a diverse set of seven question-answering benchmarks\": Does this mean the original QA datasets are transformed as SeekBench-style benchmarks using the methodology (annotation process) described in Section 3? If so, as many QA datasets do not provide reasoning traces, where do the original traces and steps come from? Why not evaluate the agents on the proposed SeekBench (\"SeekBench comprises 190 expert-annotated traces with over 1,800 response steps\")?\n6. Section 4 analyzes the differences between LLM agents regarding the proposed metrics/capabilities, but what causes such differences? Is it because of the training schema or other aspects of different LLM agents? What do the comparisons reveal? What are the insights into improving the current LLM agents based on the findings of SeekBench?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed benchmark is well-designed, and the methodology is well-elaborated.\n2. The evaluation is extensive on seven QA datasets, using multiple LLM search agents.\n3. Further analysis provides insights into the differences between LLM agents regarding the proposed metrics/capabilities.", "weaknesses": "- Some concerns regarding methodology, experiments, and analysis. Please refer to \"Questions\" below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)