Repurposing Synthetic Data for Fine-grained Search Agent Supervision
Authors: Yida Zhao
Kuan Li
Xixi Wu
Liwen Zhang
Ding-Chu Zhang
Baixuan Li
Maojia Song
Zhuo Chen
Chenxi Wang
Xinyu Wang
Kewei Tu
Pengjun Xie
Fei Huang
Jingren Zhou
Yong Jiang
Venue: ICLR 2026 Poster (2026)
Keywords: Search Agent
Web Agent
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: LLM-based search agents are increasingly trained on entity-centric synthetic data to solve complex, knowledge-intensive tasks. However, prevailing training methods like Group Relative Policy Optimization (GRPO) discard this rich entity information, relying instead on sparse, outcome-based rewards. This critical limitation renders them unable to distinguish informative "near-miss" samples—those with substantially correct reasoning but a flawed final answer—from complete failures, thus discarding valuable learning signals. We address this by leveraging the very entities discarded during training. Our empirical analysis reveals a strong positive correlation between the number of ground-truth entities identified during an agent's reasoning process and final answer accuracy. Building on this insight, we introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity-aware reward function. E-GRPO assigns partial rewards to incorrect samples proportional to their entity match rate, enabling the model to effectively learn from these ''near-misses''. Experiments on diverse question-answering (QA) and deep research benchmarks show that E-GRPO consistently and significantly outperforms the GRPO baseline. Furthermore, our analysis reveals that E-GRPO not only achieves superior accuracy but also induces more efficient reasoning policies that require fewer tool calls, demonstrating a more effective and sample-efficient approach to aligning search agents.

Published as a conference paper at ICLR 2026
REPURPOSING SYNTHETIC DATA FOR FINE-GRAINED SEARCH AGENT SUPERVISION
Yida Zhao 1.,,3 Kuan Li Xixi Wu', Liwen Zhang Dingehu Zhang?. Baixuan Maojia Song". Zhuo Chen 1,2,3 Chenxi Wang'. Xinyu Wang".
Kewei 1:2 Pengjun Xie Fei Huang Jingren Zhou², Yong Jiang
School of Information Science and Technology, ShanghaiTecc University
2 Shanghai Engineering Research Center of Intelligent Vision and Imaging
3 Tongyi Lab, Alibaba Group
ABSTRACT
search agents are increasingly trained on entity-centric synthetic data to solve complex, knowledge-intensive tasks. However, prevailing training meth- ods like Group Relative Policy Optimization (GRPO) discard this rich entity infor- mation, relying instead on sparse, bassede-based rewards. This critical limitation renders them unable to distinguish informative mnear-miss" samples- ttose with substantially correct reasoning but flawed final answer-ffroe thus discarding valuable learning signals. We address this by leveraging the very entities discarded during training. Our empirical analysis reveals strong posi- tive correlation between the number of ground-truth entities identified during an agent's reasoning process and final answer accuracy. Building on this insight, we introduce Entity-aware Group Relative Policy Optimization (E-GRPO), a novel framework that formulates a dense entity- aware reward function. E-GRPO as- signs partial rewards to incorrect samples proportional to their entity match rate, enabling the model to effectively learn from these "near-misses". Experiments diverse cresstoon-nswerring (QA) and deep rescarch benchmarks show that E- GRPO consistemtly and significantly outperforms the GRPO baseline. Further- more, analysis reveals that E-GRPO not only achieves superior accuracy but also induces more efficient reasoning policies that require fewer tool calls, demon- strating more effective and sample-efficient approach to aligning search agents.
complete failures,
INTRODUCTION
The advent of Large Language Models (LLMs) has catalyzed the development of sophisticated autonomous agents, with search agents emerging
prominent class for solving complex, knowledge-intensive tasks (Yao et al.] 2023 Wang et al.) 2024, Xi et al. 2025). Training these agents to navigate the vast, noisy web effectively requires abundant and challenging data (Google) Team 2025b) OpenAl 2025 xAI Team 2025 Moonshot All 2025). To meet this demand, a dom- inant paradigm of synthetic data generation has emerged (Wu et al. 2025b: Li et al. 2025b Gao et al. 2025). In this paradigm, as shown in Figure [ (left), complex questions are often created by systematically transforming simple "seed" questions through operations like fact injection or delib- erate obfuscation. This process creates an intricate problem structure, paved with key entities that form the factual backbone of the correct answer.
This synthetic data is then used to train agents within the now-dominant reinforcement learning (Wen ct al. 2024, Singh et al. 2025). especially with Group Relative Policy Optimization (GRPO) (Shao et al. 2024) and its numerous variants (Yu et al.] 2025; Dong et al. 2025) Hu 2025 Xu et al.]
2024 Zhao et al.) 2025). These methods typically rely on outcome-based rewards, utilizing only the final answer while discarding the intermediate entity information meticulously embedded during
data synthesis. This mechanism leads to the reward sparsity problem (Qian et al.] 2025 Deng
et al.] 2025), which manifests critically for search agents (Song et al.) 2025 Wu et al.) 2025a
Jin et al., 2025 Li et al., 2025c Zheng et al. 2025 Zhang et al. 2025al Li et al.) 2025b Gao
"Corresponding authors. tukw fuk wllw@sshanghaitcch.dd...m (7lw439616, yongjiang.jy @alihaha-inc.com

---

Published as a conference paper at ICLR 2026
Initial Question: was the director Titanic?
1939
Questions are coregorzed ino three groupe besed oo comportonn or the averge entr match rite. e.g..
1750
"Coerect Rute Wrong Rate' meam that for spostion, its correct milusis gereally verntily more
2500
1350
miliis (higher onlyy mucch nie) than faiked unce.
Fact Injection ("Titanic" discarded)
1000
Rumbr
356
Who was the director the 1997 film starring Leonardo
Correct Bate
Correct Rate
Equal Rates
3.5
Fact Injection ("Leonardo" discarded)
3.0
Wronng
2.5
2.0
Final Question:
Who the director of the 1997 film starring the actor who won the Academy Award for Best Actor the film "The Revenant" ?
0.9
Normalized Entity Match Rate
Figure Left: Example of entity-centric synthetic data generation. Right: Analysis of the correlation between entity matching and agent performance.
ct al. 2:25): by treating all negative samples uniformly, GRPO fails to distinguish a "near-miss" response with correct intermediate reasoning steps but flawed answer-from complete failure. For instance, in answering Who was the director of the 1997 film starring the actor who won the Academy Award for Best Actor for the film 'The Revenant's ?, "near-miss" that correctly identifies the actor (Leonardo) and the film (Titanic) but fails on the final answer far more informative than one that misunderstands the query entirely. By penalizing both equally, standard GRPO discards crucial learning signals embedded in partially correct reasoning, forcing the model to re-learn steps had already mastered.
One natural approach to address this sparse reward problem is to incorporate fine-grained, process- level supervision. In domains such mathematics and code, this is achieved either by evaluating each intermediate step with a Process Reward Model (PRM) (Fan et al.) 2025 Anonymous 2025 Zhang et al.] (0255) or by employing complex sampling mechanisms (e.g.. tree-based search) to derive seep- advantages (Yang et al. 2025 Hou et al. (025). However, these methods are ill- suited for the open-ended nature of web search. The sheer scale and dynamic nature of web content render the annotation required for PRM prohibitively expensive. Similarly, the extensive length of search agent trajectories, often involving dozens of tool calls and reasoning steps, makes intricate sampling strategies computationally intractable.
This leaves a critical gap: how can we obtain a fine-grained, informative, yet computationally effi- cient reward signal for search agents? The answer, find, lies hidden in plain sight: within the very entity-centric information from synthetic data generation that GRPO-like methods discard. These entities, forming the factual backbone for the answer, intuitively represent untapped source of fine-grained supervision. To validate the potential of these ground-truth entities, we analyze the re- lationship between agent performance and the number of entities matched during reasoning (entity match rate). As illustrated in Figure [ (right), the strong positive correlation we observed (further discussed in Section 3.1 validates our core hypothesis: the entity match serves as a powerful proxy for factual correctness and can be repurposed as fine ane-grained reward signal that standard GRPO lacks.
Based on this core insight, we propose Entity-aware Group Relative Policy Optimization (E- GRPO), novel RL. framework that enhances policy optimization by formulating a dense, entity- aware reward function from the entities within the synthetic training data. Specifically. instead of applying uniform penalty, our method assigns a bonus to negative samples proportional to their entity match rate. By doing a "near-miss" sample, which contains many correct entities and is highly informative for learning, receives better reward than a complete failure. This fine- one-grained reward, obtained with negligible computational cost, compels the model move beyond simply avoiding errors and towards mastering the process identifying and synthesizing key information, thereby addressing the limitation of standard GRPO in complex search tasks.
2

---

Published as a conference paper at ICLR 2026
Our comprehensive evaluation on benchmarks, spanning diverse models and environments, demonstrates that E-GRPO significantly and consistently surpasses the GRPO baseline. Critically, beyond superior accuracy, E-GRPO also enables more efficient reasoning policies that consistently require fewer tool calls. Further analyses validate our core hypothesis, confirming the importance of the entity-aware reward.
In summary, the key contributions this work are as follows:
We identify the "near-miss" problem in GRPO-based training and propose the core insight that entities from synthetic data can be repurposed fine-grained reward signal, supported by em- pirical analysis revealing a strong correlation between entity match rate and task accuracy.
We introduce E-GRPO, novel framework that enhances policy optimization by formulating an entity-aware reward function to differentiate the quality of negative samples and provide more granular supervision.
We conduct experiments on multiple QA and deep research benchmarks, demonstrating that E- GRPO not only outperforms the GRPO baseline in accuracy but also learns more efficient policies.
PRELIMINARY
In this section, we provide brief overview of key concepts in search agents and review of entity- centric data synthesis methods. More discussion of related work is available Section
2.1 SEARCH AGENTS
Multi-turn Rollout. We adopt the ReAct (Yao et al. 2023 paradigm for search agents. The LLM agent iteratively performs thought and action, and receives observation from the environment. Specifically, in each iteration, the LLM agent generates a free-form thought (T) and executes a valid action (e.g., tool call a). Then it waits for the environment's feedback the observation
(o). In the web search setting, the action space typically consists of searching queries, webpage browsing, and generating the final answer. The iteration terminates when the LLM generates final answer. A complete rollout with T iterations can be defined as:
H (Ti, 41, 01, Ge Or, Tr, QT),
where Tr, at on represent thought, action and observation at step f, with 7ı, at sampled from a policy Ty based on all previous context (Te, a,) To(- g. Tı, 01, ......... T--1 0,-1, 0.-)). The specific format of multi-turn rollout detailed in Appendix
Tool Design. Following existing search agent studies (Li et al. 20256 Gao al. 2025). we define
the agent's web exploration action space with two essential tools:
Search: A search engine that accepts multiple queries and retrieves the top-10 relevant results per query, including titles, snippets, and the corresponding URLs.
Visit: A browser agent that accesses several web pages simultaneously, given the corresponding URLs and browsing goals. It first retrieves the full webpage and then uses Qwen3-30B-AB Instruct-2507 (Team 2025al extract relevant information based on the browsing goal.
2.2 ENTITY-CENTRIC DATA SYNTHESIS
A significant line of research has focused the autonomous generation of complex and grounded question-answer (QA) pairs (Wu et al. 2025a b) et 20256) Gao et al. (025) sharing a common thread in their entity-centric approach. We briefly summarize two state-of-the-art (SOTA) methods that exemplify this paradigm below.
Starting with a seed question, ASearcher's synthesis agent iter- atively increases difficulty via two entity-focused operations: Injection, which replaces named entities with descriptive facts, and Fuzzing, which substitutes specific entities with more ambigu- ous, general descriptions.
ASearcher Gao et al., 2025):

---

Published as a conference paper at ICLR 2026
SailorFog-QA (Li et al. 2025b): SailorFog -aeeroog-QAAaa first constructs complex knowledge graph via a random walk from a seed entity, creating intricate entity couplings. It then generates questions by sampling subgraphs and applying information obfuscation, which involves replacing specific entity attributes with vague descriptions.
3 METHODOLOGY
In this section, we first give a detailed analysis the correlation between agent performance and synthetic-data entity matching. Then, propose the E-GRPO algorithm, designed to improve GRPO with a fine-grained entity-aware reward function.
3.1 ANALYZING ENTITY MATCHING IN AGENTIC REASONING
Inspired by the entity approach for data generation, where entities are intuitively the factual backbone of the synthetic data, we conduct an empirical analysis to investigate how these entities correlate with the performance of a search agent.
Metrics. To quantify this correlation, we first define the entity match rate. Given syn- thetic QA pair (g.gt), we retain all the m ground-truth entities during QA generation F, {e(1), (e())) ,e!,),...., e }. and sample a group of G rollouts ( ,((2), H()) For each roll- out 7((0) in the group, let T { 1
(8) be the collection all thoughts in rollout We identify the set of entities matched within the thoughts as:
......T..
(+) matched = {e
f Bg E (I....,,i},, e is mentioned in T
(I)
An entity is considered "mentioned" if its full phrase appears thought's text (more discussion available in Appendix B). The entity match rate for rollout i, de- noted as %i is then calculated the ratio of matched entities to the total:
exact string match in the
Lx
(i) matched
%
Egl
Furthermore, to enable robust comparison across different questions which may have varying dif- ficulty, we introduce the normalized entity match rate, fg. This is calculated by normalizing the raw rate "/s against the maximum rate, "maxx: observed within its question group:
%
if 7max >> 0
t/ = Xmax
where
Ymax
7/- jE (1.....G)
max
otherwise
This normalization allows aggregate the match rate of all rollouts common 0-to-1 scale.
Analysis. To investigate the correlation between entity match rate and accuracy, we first conduct per-question analysis on sampled subset of SailorFog-QA (Li al., 20255) using the WebSailor- 7B agent (Li et al., 2025b). For each question, we perform 8 rollouts and calculate the average entity match rates correctly solved and failed rollouts, respectively (further explanation in Appendix [ As shown in Figure [ (upper right), the average entity match rate of correct rollouts was higher than that of failed ones in the vast majority of the questions, outnumbering the reverse scenario by a clear 4-to-1 margin (1939 vs. 487 questions). This establishes strong correlation between the entity match rate and the correctness the final answer.
Moving beyond this aggregate, per-question view, we analyze the distribution of the normalized entity match rate across all individual rollouts. As shown in Figure (bottom right), the distri- butions for correct and incorrect rollouts diverge significantly. The distribution of correct samples (green) peaks sharply at a normalized rate 1.0. In contrast, incorrect samples (red) show a bi- modal distribution: large peak 0.0, and notable spread across the mid-to-high range. This
4

---

Published as a conference paper at ICLR 2026
BBB
OBPO
Fdsto
e Uoe daary sheter the duriing
deseardo ctEE
<toal
Camaron -3 100e1>
Aademy doandirr "Bhe bucur"
BBB
BBB
Foet ropeeses..
Terrnd vilias
<oL pupcoso
Comeras
BBB
leal
BBB
X
OEFO
Adar Leoaaro
Nin Hosel
Hotd pall flee oEl
inpuarie
shadl salo -olod oi
clitierr
Go
0:15
Figure 2: Comparison of GRPO and E-GRPO. GRPO applies outcome- reward, while E- GRPO additionally assigns a bonus to negatives proportional to their normalized entity match rate. The three rollouts illustrate success, a complete failure, and a "near-miss", respectively.
latter group represents the informative "near-misses", where most entities were found but the final reasoning failed.
This analysis shows that the entity match rate is more than just a pass/fail indicator. Instead, provides granular scale distinguish valuable "near-misses" from complete failures, offering richer signal of an agent's 's reasoning quality.
3.2 ENTITY-AWARE GROUP RELATIVE POLICY OPTIMIZATION
The preceding analysis shows that entity match rate offers a fine-grained signal of an agent's reason- ing quality. Conventional policy optimization methods, however, largely ignore this signal by relying on sparse, outcome-based reward tied only to answer correctness, thereby treating all failures as equally undesirable. Therefore. we introduce Entity-aware Group Relative Policy Optimization (E-GRPO), framework that directly incorporates the entity match rate into its reward function to guide policy learning better.
Limitations of Reward Formulation in GRPO. Existing GRPO-like frameworks (Shao et al. 2024) for search agents typically employ outcome-based reward. Specifically, the reward R2 for rollout (((() is defined simply as 1 it leads to a correct answer, and O otherwise. This reward is then used to compute a group-relative advantage. This advantage value, denoted A.g, is calculated once for the entire rollout i and then applied to every token j within it, serving as the core learning
signal:
Á.j R4 - mean( { (Ru (R
std({ Rg 1
The limitation of this formulation evident: as shown in Figure standard GRPO assigns an iden- tical reward of 0 to both complete failure (middle rollout, O entities matched) and an informative "near-miss" (bottom rollout, entity matched), thus rendering their different reasoning qualities indistinguishabl..
Entity-aware aware Reward Formulation. E-GRPO addresses the limitation of outcome-based rewards by redefining the reward function with an entity-aware bonus. We utilize the normalized entity match rate ? rather than the raw rate, as its consistent 0-to-1 scale is essential for a stable advantage calculation across different groups. Our entity awtrre reward is thus defined as:
if H(() is correct if H() is wrong
R;
a.
(5)
0
if er becurs in H((1)
where E [0, 1] is a hyperparameter balancing the value of accuracy and entity matching. This formulation yields two significant advantages. (1) It creates a dense reward spectrum to distinguish the quality of incorrect rollouts. As shown in Figure 2 a "near-miss" that identifies correct entity (Leonardo) is rewarded (a 0.5), unlike a complete failure which receives zero. (2) It provides a
meaningful training signal even in all-wrong groups where standard GRPO offers no gradient.
Errrr (format and overlength prohlems) are detailed in the subsequent paragraph Implementation De-
tails
5

---

Published as a conference paper at ICLR 2026
Overall Training Objective. With our entity-aware reward defined, now formalize the complete E-GRPO objective. First, the refined reward from Eq. 5 is used to compute a more infor-
mative advantage At, via Eq. The policy is then optimized by maximizing the GRPO objective [(),, defined as:
(() E (g.9t) D, (7
1
G
|
Σ min (c. - -Eoow 1 EEigh) 2..)
(6)
Σ i (7((6)
Tu( 1g. H1
1)
where (0,
the importance sampling ratio.
me (H) 19.95
Implementation Details. Based this objective, additionally apply the following practical modifications to the training of all models (both our method and the baselines):
KL-Free Objective and Policy Exploration. Following DAPO (Yu et al., 2025). we remove the KL-divergence regularization term in GRPO and apply the "clip-higher" method, which increases the upper clipping bound Ehigh to better encourage policy exploration.
Handling Format Errors. Rollouts with format errors (defined in Appendix A are assigned a reward of O. This striet penalty is justified because RL training is preceded by cold-start SFT phase that ensures the model is already familiar with the required output format.
Handling Overlength Rollouts. Overlength rollouts (i.e., those exceeding token or tool-call lim- its) are also assigned reward of O. We observed preliminary experiments that directly optimiz- ing on these rollouts can lead to policy collapse. Therefore, we adopt specific handling strategy: while these rollouts contribute to the advantage normalization (i.e., computing the group's s mean and standard deviation), they are excluded from the final loss computation to prevent instability.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Benchmarks. Our evaluation spans diverse set of 11 benchmarks to comprehensively assess E-GRPO's effectiveness. For question -anoonn-nnwninngg tasks, we use three single-hop datasets: Natural
Questions (NQ) (Kwiatkowski al. 2019), TriviaQA (TQ) (Joshi et al., 2017). and PopQA (Mallen et al. 2022); and four multi-hop datasets: 2WikiMultiHopOA (2Wiki.) (oo et al. 2020). HotporQA (HQA) ( Yang al. 2018), Bamboogle (Bamb.) (Press et al.) 2022), and MuSiQue (Musi.) (Trivedi et al. 2022). We further test our agent on four challenging deep research benchmarks: GAIA (Mi- alon al. 2023). BrowseComp (Wei et al.) 22025). BrowseComp-ZZi (Zhou et al.) (025). and (xbench-DS) (Xhench- Tehan (025). Following Asearcher (Gao et al. 2025), we use 1000 sampled instances from the validation sets of HQA, 2Wiki., and Musi. For GAIA, we use the 103-sample text-only validation subset (Li al. 2025c). For all other benchmarks, use their full test sets.
Obench-DeepSeaal
Baselines and Reference Agents. Our primary baseline is the direct counterpart trained with GRPO (Shao et al., 2024), allowing for a controlled comparison of the algorithmic enhancement. We also compare against a suite of ReAct-based agents. For QA benchmarks, this includes RI Searcher-7B (Song al. 2025), DeepResearcher:7B (Zheng et al. 2025). Search-R1-32BB (Jin et al. 2025), Simple-DD DaseeDS--QQQ (Sun et al., 2025), and ASearcher- 14B (Gao et al. 2025 For deep research benchmarks, we include advanced models like OpenAllo3, Claude-4-Sonnet (Anthropic 2025 Kimi-K2 (Team et al. 2025), and DeepSeek-V3.1 (Liu et al. 2024), alongside open-source mod- els with more than 32B parameters such as R1-Searcher-7B. WehThinker-RL. (Li al. 2025d) WebDancer-QWQ (Wu et al. 2025a), and WebSailor-7B/32B ([i et al. 2025b).
Environment Settings. We conduct training in two distinct environments to validate E-GRPO's robustness: closed-world local knowledge base (Local) and an open-world web exploration

---

Published as a conference paper at ICLR 2026
Table 1: Overall Pass @ performance on standard QA benchmarks. Results with are sourced
from Gao et al. 2025). The top scores of each evaluation environment are bolded.
Multi-Hop QA
Single-Hop QA
Environment
Model
Avg
2Wiki.
IQA
Bamb.
Musi.
NQ
TQ
PopQA
Comparison among Our Models
Local-7B-SFT
74.0
66.7
72.8
30.2
49.8
78.4
49.6
60.2
Local
Local-7B Seaal7BBGRRPO
75.1
65.1
74.4
31.2
51.5
82.0
50.4
61.4
Local-7B-E-GRPO
79.6
69.0
78.4
32.8
55.8
83.9
50.2
64.2
Comparison with Other Reference Agents
RI -
69.4
61.6
72.0
25.3
48.7
79.5
45.2
57.4
DeepReseancher-7B5
64.1
61.0
76.8
24.5
52.9
82.8
45.7
58.3
Search-R1-3251
69.3
64.2
81.6
30.8
51.1
86.6
53.6
62.5
Simple-DS-QwQ"
80.4
67.5
83.2
32.9
55.3
90.2
65.3
Web
AScarcher-14B-
79.8
70.5
80.8
33.8
55.4
88.5
50.5
65.6
Local-7B-SFT
76.8
70.7
80.2
32.2
55.4
88.7
48.9
64.7
Local-7B-GRPO 77.2
73.8
82.4
34.9
55.9
89.3
50.1
66.2
Local-7B-E-GRe
80.4
73.7
85.6
34.9
59.1
90.4
50.2
67.8
(Web) environment. In the Local setting, search and visit tools are simulated via information re- trieval over Wikipedia 2024 corpus (Karpukhin al.) [2020: Gao et al. (2025). In the Web setting, the agent interacts with the live web using Google Search and Jina (Jina.ai 2025) for page fetching.
Training Details. Our experiments are based on Qwen2.5--B-Inntruct Yang et al. 2024) and 23B--Instruct-2007 (Team] 2025a), covering differentt model sizes and architectures (dense and MoE). It is important to note that our study aims to validate the effectiveness of E- GRPO at the algorithmic level, not merely to pursue state-of-the-art performance. Therefore, use limited data to ensure training efficiency while still enabling performance comparison.
Cold-start SFT: We first fine-tune the base models on 11K samples from SailorFog QA (Li et al., 20256). This step, following Dong et al. (2025), mitigates reward collapse and ensures the model understands the agentic format before RL.
RL: We generate two distinct sampe datasets for RL training. For the Local environment, we synthesize data using the Asearcher (Gao et al.) (025) method over the 2024 Wikipedia corpus. For the Web environment, we use the SailorFog-QQ. data generation pipeline. Note that both methods are anchored in entities from Wikipedia despite the distinct environments they use. Cru- cially, for both datasets, we retain all ground-trui truth entities generated during the synthesis process to enable E-GRPO. We train the 7B model in both environments, while the 30B model trained only in the more complex Web environment. For each setup, we apply both GRPO and E-GRPO.
We denote our models by their training environment, model sizes, and the training algorithm, e.g., Local-7B-GRPO. Detailed hyperparameters are presented in Appendix C
Evaluation Metrics. Model answers. extracted from the model output enclosed in <answer> and </answer> tags (detailed in Appendix A are evaluated for correctness using Qwen2.5-725) -22---nrtttttt under the LLM-as-Judge setting. We report the average Pass @ over all test samples, well the Pass@: across three rollouts.
4.2 MAIN RESULTS
We present the experiment results across three evaluation settings: (1) 7B models trained and eval- uated with the Local environment standard QA benchmarks, (2) the same 7B models evaluated with the Web environment on the same benchmarks, and (3) all models trained and evaluated with the Web environment deep research benchmarks.

---

Published as a conference paper at ICLR 2026
Table 2: Overall performance on four challenging deep research benchmarks. Results with sourced from Wu et al. (202c)). The top two Pass @ scores of agents 32B are bolded and underlined. The top Pass @ scores of our agents are bolded.
GATA
BrowseComp
BrowseComp-2It
xbench-DS
Model
Pass@1
@
@
Pass@3
Pass@1
Pass@3
Pass@]
Pass@3
Advanced Models
OpenAl-o3'
70.5
50.9
58.1
66.7
Claude-4-Sonmet
68.3
12.2
29.1
64.6
Kimi-K2
57.7
14.1
28.8
50.0
DeepSeek-V3.1r
63.1
30.0
49.2
71.0
Open-sone source Agents S 32B
R1-Searcher-7B
20.4
WebThinker-RL.
48.5
2.8
7.3
24.0
WebDancer-Qwe
51.5
3.8
18.0
39.0
WebSailor-7B
37.9
6.7
14.2
34.3
WebSailor-32B
53.2
10.5
25.5
53.3
Our Agents
Web-7B-SFT
31.7
44.7
5.7
10.5
13.2
25.6
37.3
55.0
Web-7B-GRPO
33.0
44.7
11.7
31.5
40.7
Web-7B-F-GRPO
36.9
51.5
9.3
16.1
18.1
32.1
42.0
59.0
Web-30B-SFT
45,0
60.2
10.8
18.5
23.8
38.1
43.7
63.0
Web-30B-GRPO
47.6
62.1
12.3
18.9
25.7
38.8
45.3
65.0
Web-30B-E-GRPO
48.5
65.1
12.9
21.0
26.4
41.2
46.7
66.0
Performance in the Local Environment on QA benchmarks. The top block Table presents the results for models trained and evaluated within the controlled Local environment. Our Local- 7B-E-GRPO model achieves the highest average score of 64.2, marking a substantial improvement of 2.8 points over the GRPO baseline and 4.0 points is consistent across most individual benchmarks, demonstrating that the entity-aware reward allows the model learn more effective reasoning policy than the aatcome-based reward.
the initial SFT model. This superiority
Performance in the Web Environment on QA benchmarks. As shown in the second block of Table even when evaluated with the unfamiliar web environment, our Local-7B-E-GRPO model again achieves the highest average score among its peers 67.8, outperforming the GRPO coun- terpart and other open-source baselines with larger sizes. This result strongly validates the gen- cralizability and robustness of our method, allowing a locally trained model achieve superior performance in completely different, real-world setting.
Performance on Deep Research Benchmarks. As presented in Table 2 results on deep research benchmarks consistently underscore the superiority of E-GRPO. Across both 7B and 30B scales, our E-GRPO models significantly outperform their GRPO counterparts. Notably, Web-30B-E-GRe achieves the best performance among open-source agents BrowseComp (12.9) and BrowseComp- ZH (26.4), even surpassing advanced models like Claude-4-Sonnet on BrowseComp, and narrows the gap with others.
The algorithmic advantage of E-GRPO is most evident in the Pass@3 results. While GRPO of- fers minimal gains over the SFT baseline (e.g., 44.7 on GAIA), E-GRPO delivers substantial im- provements (e.g., 6.8-point jump to 51.5). This stems from a key algorithmic difference: GRPO's outcome- based reward tends to refine existing successful strategies, whereas E-GRPO's s entity entity-aware aware reward explicitly encourages exploring promising but incomplete paths. This allows the agent to build more diverse set of solutions, which directly increases its chances succeeding within a few attempts and explains the significant @ gains.
8

---

Published as a conference paper at ICLR 2026
compariton.o of Traning Accurncy
Comparioon of Soat Stops
Comparion of Gaicty Mardiing and Acouaay
CEPP Accurncy
GSPO Taii Steps
bicaro sirtity Natch sate
13.0
Geeo Enity Netch
oeFo Accuricy
6.6
32.3
WW
Ansroge Enity Metch
2.43
12.0
M
1.40
100
Figure 3: Training dynamies of 30B models with the Web environment, including the comparison of E-GRPO and GRPO over training accuracy, tool call steps, and the analysis between entity matching and training accuracy.
4.3 ANALYSIS
Training Dynamics. We begin by analyzing the training dynamics of E-GRPO against the GRPO baseline. As shown in Figure 3 E-GRPO demonstrates superior learning efficiency and effective- ness. The left panel shows that E-GRPO (purple) consistently achieves higher training accuracy, showing a steadier and more pronounced upward trend than the GRPO baseline (blue). This suggests that the dense, entity-awar reward provides a more effective and stable learning signal. Simultane- ously, the middle panel reveals that E-GRPO learns more efficient reasoning policy, consistently using fewer tool calls per rollout. This efficiency can be attributed to rewarding the discovery of key entities, which guides the agent towards more direct and informative solution steps. Extended training dynamics are provided in Appendix G for reference.
To further validate E-GRPO's mechanism, we analyze the relationship between the entity match rate and the accuracy during training, as illustrated in the right panel of Figure 3 A strong positive correlation is evident: for both GRPO and E-GRPO, the curves of the entity match rate and accuracy rise in tandem. This confirms our core hypothesis that the entity match rate serves proxy for final answer accuracy. Crucially, the plot reveals the direct impact of our entity-awan reward: by explicitly incentivizing higher entity match rate, E-GRPO (orange) consistently out- performs the GRPO baseline (green) on this metric. This advantage, in turn, directly translates into superior final answer accuracy (purple vs. blue), validating that mastering the sub-goal of entity matching leads to better overall performance.
effective
A detailed case study in Appendix D provides qualitative illustration of these dynamics, con- cretely demonstrating how E-GRPO's s focus on entity matching leads to a more efficient and accu- rate reasoning path. Through the case study, we also analyze several failure cases of E-GRPO in Appendix
Ablations of Entity Matching Weights. We conduct an ab- lation study the hyperparameter o, which balances the outcome-based reward and the entity-matching bonus. As shown in Figure 4 setting a 0.0 reduces method to the GRPO baseline. For all four benchmarks, performance consis- tently improves as a increases from 0.0, peaking 0.3. This clearly demonstrates the benefit of incorporating the entity- aware reward. However, further increase to a 0.5 leads to a performance drop on most benchmarks, suggesting that an excessively strong entity- mattt--atchhnn bonus can distract the model from the primary goal of generating a correct final an- swer. This highlights the importance of balancing the two re- ward components, with a moderate value yielding the optimal a value during training is available in Appendix
BreaseCorp
48
25
222
125
179
Entity Matching Weight
Figure 4: Comparison of different entity matching weights. policy. More analysis of decaying
5 RELATED WORK
Search Agents. The capabilities of Large Language Models (LLMs) have fueled a surge in re- search autonomous agents that interact with external environments to solve complex tasks.

---

Published as a conference paper at ICLR 2026
A foundational paradigm in this area is the ReAct framework Yao al. 2023).
which interleaves
reasoning (thought) and action steps. Building on this, a prominent line of research has focused on search agents designed to navigate the web (Song et al., 2025 Zheng al.
et al. 2025c Zhang et al. 2025a, Sun et al. 2025). Advanced models like Gemini Deep Research Goog e Team 20256), OpenAT Deep Research (OpenAl 2025). Grok DeepSearch (xAl Team 2025). along with smaller open-source models such as Asearcher (Gao et al. 2025), WebThinker (Li et al. 2025d), WebWalker (Wu et al. 2025b), WebDancer (Wu et al. (025a), and WebSailor (Li et al.) 2025b a)
2025
have demonstrated increasing proficiency in retrieving and synthesizing information from noisy, real-world web sources. Our work directly contributes to improving the training methodology for this class of agents, addressing the challenge of learning robust policies in complex web environ- ments.
Synthetic Data Generation for Search Agents. The now-dominant paradigm for training search agents relies heavily on high-quality synthetic data (Teaml (025b). A common thread in these
generation methods is an entity-centric approach to complexity generation (Gao et al. 2025 Wu al. 2025b Li et al., 2025b at Tao al. 2025| Wu et al. 2025c). During data synthesis, rich set of ground- truth entities that form the factual backbone of the correct answer are systematically discarded. Prior work has exclusively used the final question question-answer answer pairs from this process for post-training (Dong et al., 2025 Wu et al., 2025a; Li et al.) 2025b). In contrast, our work is the first, to our knowledge, to recognize these discarded entities not as a byproduet, but as an untapped source of fine-graine fine-grained, factual supervision. We pioneer the idea of repurposing this "waste" material to formulate an entity-aware reward function, thereby bridging the gap between the data generation process and the RL alignment phase in novel and efficient manner.
Reinforcement Learning for Search Agents. Group Relative Policy Optimization (GRPO) and
its variants (Shao et al. 2024 Yu et al., 2025; Xu et al., 2024 Zhao et al.) 2025 Hu 2025 Xue
al. 2025 Su et al., 2025) have become dominant paradigm for aligning search agents. Notable advancements within this paradigm, such as ARPO (Dong et al. 2025), , have adapted the framework with an entropy-based rollout mechanism for complex multi-turn web search settings. Despite these refinements, the entire family of GRPO-like methods is fundamentally constrained by its reliance on a sparse, outcome- saacome-based reward signal. While conventional solutions to such sparsity, like Process Reward Models (PRMs) (Fan et al.) 2025; Anonymous 2025 Zhang et al. (025b) or tree-based sampling (Yang et al. 2025 Hou et al. 2025), ) exist in related domains, they are ill-suited for open- ended web search due to prohibitive annotation costs and computational intractability. Our work, E-GRPO, diverges from these approaches by proposing a reward signal that is both fine-grained and computationally efficient, requiring no additional annotation, model training, or complex sampling.
6 CONCLUSION
In conclusion, we propose Entity- Group Relative Policy Optimization (E-GRPO), a novel framework designed to enhance policy optimization for search agents. Our analysis reveals that the ground-truth entities discarded during synthetic data generation serve as powerful proxy for factual correctness, offering fine-gra fine-grained reward signal that standard methods ignore. E-GRPO leverages this insight by formulating an entity-aware reward function, assigning partial credit to negative sam- ples based on their entity match rate to encourage meaningful exploration. Across a wide array of QA and deep research benchmarks, E-GRPO consistently and significantly outperforms the GRPO baseline. Remarkably, it not only achieves superior accuracy but also learns more efficient policies with fewer tool calls, offering more effeecive and sample-efficient solution for aligning search agents in complex, knowledge-intensive tasks.
7
ACKNOWLEDGMENT
This work was supported by Alibaba Group through Alibaba Innovative Research Program, the Core Facility Platform of Computer Science and Communication, SIST, and the HPC platform of ShanghaiTTcch University.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Anonymous. LSRL: Process-supervised GRPO on latent recurrent states improves mathematical reasoning. Submitted ACL Rolling Review May 2025, 2025. URL https::/oppenrevien. net/forum?id- Jfok under review.
Anthropic. Introducing claude 4, 2025. /ttp/tttss///w.a.anrppp....o///s
Yong Deng, Guoqing Wang, Zhenzhe Ying, Xiaofeng Wu, Jinzhen Lin, Wenwen Xiong, Yugin Dai, Shuo Yang, Zhanwei Zhang, Qiwen Wang. Yang Qin, Yuan Wang. Quanxing Zha, Sunhao Dai, and Changhua Meng. Atom-searcher: Enhancing agentic deep research via fine-grained atomic thought reward, 2025. URLhttss://arxix http:/aaii..ooggg/b/a// aataaabb////..0 12800
Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang. Fuzheng Zhang, al. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.1900 198.999...9 2025.
Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu. Posterior-grpo: Rewarding reasoning processes in code generation, 2025. URL
wwtp:///...rraa///......o/ 05170
Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl, 2025. URL https:://axxi..orrrrr
07976
Google Team.
Gemini 2.5 pro. https: hotp:////////oooe////////////r// 2025a.
Google Team. Introducing gemini deep research.
reep-rese 2025b.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hog qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint 17XXx:2011.0006/ 2020.
Zhenyu Hou, Ziniu Hu, Yujiang Li, Rui Lu, Jie Tang, and Yuxiao Dong. Treerl: Lim reinforcement learning with on-policy tree search, 2025. URL htttp::/aarxw.org//bbbtt
11902
Jian Hu. Reinforce++: simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262 2025.
Bowen Jin, Hansi Zeng. Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training lims to reason and leverage search engines with reinforcement learning. arXiv preprint 16X1::2503.095 2025.
Jina.ai. Jina, 2025. URL https::////:aaaaaa
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv: 777500055,,, 2017.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6666666 6681,,711 Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 1226665//1/00000mmmmmmmm.m URL htttps://aalaatthlll/
wrw. 0rg/ wnn.p-mmin..55..
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453 466, 2019.
Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye, Yida Zhao, Liwen Zhang, Litu Ou, Dingchu Zhang, Xixi Wu, Jialong Wu, Xinyu Wang, Zile Qiao, Zhen Zhang, Yong Jiang, Pengjun Xie. Fei Huang, and Jingren Zhou. Websailor-v2: Bridging the chasm to proprictary agents via synthetic data and scalable reinforcement learning, 2025a.
.rg wwtp:/rr/////aa///// abs//509.//22.../ 13305

---

Published as a conference paper at ICLR 2026
Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baix- uan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang. Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. Websailor: Navigating super- human reasoning for web agent. arXiv preprint arXiv:2507.02592 2025b.
Xiaoxi Li, Guanting Dong. Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025c.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprini arXiv:2504.21770 2025d.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint 1.Xiv:2212.105111 2022.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
Moonshot AL. Kimi-researccher: End-to-end rl training for emerging agentic capabilities, http:::////// moonshotai. gi thub. Kimi Kimi-Researcher/
thu... Low.iiiiReeeardeeeeeeeee 2025. URL / /
OpenAl. Openai deep research. 2025.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani- Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs. arXiv preprint arXiv:2504.13958, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0388 2024.
Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. Agentic reasoning and tool integration for IIms via reinforcement learning. arXiv preprint arXiv:2505.01441,20000 2025.
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao Lei Fang, and Ji-Rong Wen. RI-searcher: Incentivizing the search capability in 1lms via reinforcement learning. arXiv preprint arXiv:2503.05592 2025.
Zhenpeng Su, Leiyu Pan, Xue Bai, Dening Liu, Guanting Dong, Jiaming Huang, Wenping Hu, and Guorui Zhou. Klearereasoner Advancing reasoning capability via policy optimization. arXiv preprint arXiv:2508.07629, 2025.
clipping
Shuang Sun, Huatong Song, Yuhao Wang, Ruiyang Ren, Jinhao Jiang, Junjie Zhang, Fei Bai, Jia Deng, Wayne Xin Zhao, Zheng Liu, et al. Simpledeepsearcher: Deep information seeking via web-powered reasoning trajectory synthesis. arXiv preprint arXiv:2505. 16834, 2025.
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang. Yong Jiang, Pengjun Xie. Fei Huang, and Jingren Zhou. Webshaper: Agenti- cally data synthesizing via information seeking formalization. arXiv preprint arXiv:2507.1506/ 2025.
12

---

Published as a conference paper at ICLR 2026
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534 2025.
Qwen Team. Qwen3 technical report, 2025a. URL httLhttsss//aaxii.or///////////2 09388
Tongyi DeepResearch Team. DeepResearch 2025b.
Tongyi-deepresearch. https://giithb..r
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554 2022.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung. Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12511 2025.
Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforcing Ilm agents via policy optimization with action decomposition. Advances in Neural Information Processing Systems, 37: 103774-103805, 2024.
Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. Webdancer: Towards au- tonomous information seeking agency, arXiv preprint arXiv:2505.22 22644.......... 2025a.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, and Fei Huang. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572. 2025b.
Xixi Wu, Kuan Li, Yida Zhao, Liwen Zhang, Litu Ou, Huifeng Yin. Zhongwang Zhang, Yong Jiang. Pengjun Xie, Fei Huang, Minhao Cheng, Shuai Wang, Hong Cheng, and Jingren Zhou. Resum: Unlocking long-horizan search intelligence via context summarization, 2025c. URL
13313
xAI Team. Grok agents: Combining reasoning and tool use.
https://x.aainewss/rre
sattassoningaan--toonoo and- 2025.
-bench-Team. Xbench-deepsearcch, 2025. URL
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et The rise and potential of large language model based agents: A survey. Science China Information Sciences, 68(2):12100,, 2025.
Zheng Xu, Xu Dai, Shaojun Wei, Shouyi Yin, and Yang Hu. Gspo: A graph substitution and paral- lelization joint optimization framework for dnn inference. In Proceedings of the 61 ACM/IEEE Design Automation Conference, pp. 1-6, 2024.
Zhenghai Xue, Longtao Zheng, Qian Liu, Yingru Li, Xiaosen Zheng, Zejun Ma, and Bo An. Sim- pletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning. arXiv preprint 00X20:2509.02479, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint axXiv:2412.55115. 2024.
Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. Treerpo: Tree relative policy optimization, 2025. URL https:///txt.ii. orgg/bss/500 05183
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: / 090000 2018.
13

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. 2025.
Dingchu Zhang, Yida Zhao, Jialong Wu, Baixuan Li, Wenbiao Yin, Liwen Zhang, Yong Jiang, Yufeng Li, Kewei Tu, Pengjun Xie, et al. Evolvesearch: An iterative self-evolving search agent. arXiv preprint arXiv:2505. 22501, 2025a.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503: 12937, 2025b.
Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shao- han Huang, Lei Cui, Qixiang Ye, al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673, 2025.
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments. arXiv preprint arXiv:2504.03: /60 2025.
Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, et al. Browsecomp-zhh Benchmarking web browsing ability of large language models chinese. arXiv preprint arXiv:2504. 19314, 2025.
14

---

Published as a conference paper at ICLR 2026
A
FORMAT
Our ReAct framework follows Li et al. 2025b). ). A complete rollout follows the format below:
Format
<think> thinking process here </think>
<tool call>
("name": "tool name here", "arguments": ("parameter name here": parameter value here,
"another parameter name here": another parameter value ne.))...))
</tool_call>
<tool_response>
tool prol.response here
</tool response>
(more thinking processes, tool calls and tool responses here)
<think> thinking process here </think>
<answer> answer here </answer>
Any response that does not strictly follow the format will be considered with format errors.
B DISCUSSION ABOUT SYNTHETIC-DATA ENTITIES
B. ENTITY CONSTRUCTION
First, we explain the construction of the ground-truth entity sets for two different data synthesis methods used in our experiments.
ASearcher. As illustrated Figure D and Section 2.2 the question is iteratively constructed by selecting an entity and replacing it with descriptive facts or fuzzing it. Consequently, we use all selected and modified entities for question as its ground-truit truth entity set.
SailorFog-QA. As shown in Section 2.2 data generation begins by sampling entity subgraph, followed by prompting an LLM to generate a question centered around these entity nodes. There- fore, the node set of the sampled subgraph is regarded as the ground-truth entity set.
Entity Quality Control. Since the question generation process ensures question quality, e.g., in- jected facts strictly adhere to the selected entity, and generated questions are consistently centered around the sampled subgraph. the resulting entity sets highly precise. Even if there are unex- pectedly noisy entities, our reward mechanism is robust to them. Since any irrelevant entity is likely to be missed by all rollouts within a group, it does not change their relative performance and thus does not affect the normalized reward signal.
B.2 ENTITY MATCHING
Then, we consider two questions related to the entity matching mechanism: (1) Why do we use the exact string match rather than using LLM for matching? (2) Why do only count the entities matched thoughts, excluding those matched in observation?
Rationale for Exact String Matching. Our decision use exact string matching instead of an LLM-based judger is primarily motivated by the nature of our ground-truth entities, which are def- inite, short-formed strings with little ambiguity. This characteristic makes exact matching a natural and sufficient method, which in turn addresses two practical concerns: training efficiency and ro- bustness against reward hacking.
First, employing LLM to semantically parse and match entities within long reasoning traces would introduce significant computational latency, impeding the throughput of the RL training loop. In contrast, exact string matching is computationally trivial and adds negligible overhead.
15

---

Published as a conference paper at ICLR 2026
Correct Wrong
Correct Wrong
Sample Dansity.
2.5
3.0
2.0
15
10
10
0.
Normalized (Thought) Entity Match Rate
Normalized (Trajectory) Entity Match Rate
Figure 5: Comparison of Normalized entity match rate in thoughts and entire trajectories.
Second, while advanced L.Mss can perform semantic matching, they are also more susceptible to exploitation by the policy model. In preliminary experiments, we observed a distinct reward- behavior: the agent learned to extend its thoughts with verbose, superficially relevant phrases that, while not containing the correct entities, would mislead the LLM judge into erroneously assigning partial credit. Exact string matching, being less flexible, provides more reliable reward signal, ensuring the agent is rewarded for factual correctness rather than plausible-soundiigg text.
Rationale for Thought-Only Matching. To justify why we match entities exclusively within the agent's thoughts (<think> </think> blocks), analyze the difference between this approach and matching across the entire trajectory (including observations).
As shown in Figure [] the two methods yield notably different distributions for incorrect trajecto- ries. While thought-based matching (left) shows a clear separation with most failures having a low match rate, trajectory-based matching (right) produces significantly more "false positives": incorrect rollouts that still achieve high entity match rate.
We do several case studies and find the cause of this discrepancy. Often, key entity present the observation returned by a tool (e.g., search snippet), but the agent fails to extract and incorporate this information into its reasoning process. Rewarding the agent based the entire trajectory would grant unearned credit for merely encountering information, not for understanding and acting upon This creates a noisy reward signal that fails to penalize true reasoning failure. Therefore, by confining entity matching to the agent's 's thoughts, we ensure the reward is directly coupled to the model's s ability to identify and internalize key information, providing a cleaner and more targeted learning signal.
C HYPERPARAMETERS
SFT. We apply training batch size of 32, cosine decay learning rate of 5e-6 for about 4.8 epochs, with a linear warm up starting from le- for 0.2 epochs.
RL. We apply context length of 32k, tool call budget of 40, training batch size of 64, a ppo batch size of 64 (exactly on policy optimization), group size of 8, and learning rate of 2e-6 for about 5 epochs. For E-GRPO, we the entity matching weight a as 0.3 by default.
Inference.
We apply a temperature of 0.6, top_p of 0.95, and tool call budget of 40 for inference.
16

---

Published as a conference paper at ICLR 2026
D
CASE STUDY
To illustrate the qualitative differences in reasoning. present comparative study using tra- jectories generated by Web-30B-GRPO and Web-30B-E-GRRPO for the same query. The E-GRPO agent demonstrates fully identify all key entities and culminating in the correct answer. In stark contrast, the GRPO agent embarks on longer, less fruitful trajectory. Despite using more tool calls, it fails pinpoint the critical entity in the thoughts, the First International Polar Year (described as the "international scientific initiative focused on the polar regions"), which ultimately leads to an incorrect answer.
direct and efficient reasoning path, requiring fewer tool calls success-
Case Trajectory on SailorFog-Que Solved by Web-30B-E-GRR
Question: A European naval officer, known for leading a significant polar expedition in the late 19th century aboard vessel named after an Austrian admiral, later proposed major international scientific initiative focused on the polar regions. For his contributions to exploration, received one the highest honors from a major British geographical society. Who is this explorer?
Ground-truth Entities: [TTegtthofff, International Polar Year, Royal Geographical Soci- ety's Founder's Medal]
Answer: Karl Weyprecht
Round
<think>
To identify the explorer in question, I need to connect several specific clues: European naval officer who led notable polar expedition in the late 19th century aboard ship named after an Austrian admiral, later proposed a major international polar scientific initiative, and received a top honor from a major British geographical society. These are highly distinctive details, but the identity is not immediately clear. By structuring my search queries to include all these elements expedition timing, vessel name origin, scientific initiative, and British recognition maximize the chance of surfacing sources that directly link these facts to single individual. This comprehensive approach should yield either the explorer's 's name or authoritative references that will allow searches now to synthesize the relevant information.
deduce the answer. need these targeted
</think>
<tool_call>
("name": "search", "arguments": ("query": ["European naval officer polar expedition late 19th century vessel named after Austrian admiral", "European naval officer proposed major international scientific initiative polar regions", "Furopean naval officer received highest honors major British geographical society"", "late 19th century vessel named after Austrian admiral polar expedition"]]] calol_ccal>>>>
<tool_response>
Google search for 'European naval officer polar expedition late 19th century vessel named after Austrian admiral' found 10 results:
## Web Results 1. (Austro-Hungarian North Pole expedition Wikipedia]
The main ship was the Tegetthoff named for the Austrian Admiral Wilhelm von Tegetthoff, under whom Weyprecht had served. was built for this expedition by Teklenborg Beurmann in Bremerhaven. It was a three-masted barkentine of 220 tons, 38.34 m long, with 100 horsepower (75 kW) steam engine.
hatpss//en.wikipedii..e wttpiipedia..o/wiii//i/AA/.. Austro-Hungariann
2. [Wilhelm von Tegetthoff Wikipedia] (bttps:/een.wiiieedia.oo www.pp/eia..g//wiiiii//e...
</tool response>
Round
17

---

Published as a conference paper at ICLR 2026
<think>
The Wikipedia article on the Austro-Hungante North Pole expedition is likely to provide authoritative details about the expedition's 's leader, the ship's 's name, and its namesake. Since the question hinges on connecting the identity of the naval officer, the vessel named af- ter an Austrian admiral, and subsequent achievements, reviewing this page should clarify who led the expedition and confirm the link between the ship and Admiral Wilhelm von Tegetthoff. Additionally, the article may mention the leader's later scientific initiatives and honors, which are crucial for fully answering the question. need to visit this page to extract these specific details and establish the explorer's identity. </think>
<tool call>
("name": "visit", "arguments": ("url": htttt:::/en.wwkiiediia...
Hungarian_Nee Loana__ortt__ott toleeooeeeeeetionnn expedition"), "goal": "Identify the commander (led by) and the vessel's namesake name, Admiral Wilhelm von Tegetthoff')] </tool.call>
<tool response>
The useful information in https://enwwikipedia..om halkiipedia.og/wwii//Aaan Hungarian_ Barngarinn_Noth__PPP North_oooee_eeteee Pole_expedition for user goal Identify the commander (led by) and the vessel's namesake name, Admiral Wilhelm von Tegetthoff as follows: Evidence in page: The Austro-Hungadde to find the North-East Passage that ran from 1872 to 1874 under the leadership of Julius von Payer and Karl Weyprecht. The expedition discovered and partially explored Franz Josef Land.
North Pole expedition was Arctic expedition
</tool_response>
Round
<think>
To determine whether Karl Weyprecht received one of the highest honors from major British geographical society, need to directly associate his name with such recognition. The previous information confirmed his leadership of the expedition and the vessel's name- sake, but the question also requires evidence of prestigious British honors. By searching combinations of "arl Weyprecht" with terms like "highest honors" and "British geographi- cal society." can efficiently uncover whether he was awarded something like the Founder's Medal similar distinction from the Royal Geographical Society. This targeted search should clarify if Weyprecht fits all aspects of the question. need to run these specific queries to confirm the final piece of information.
</think
<tool.call> call>
("name": "search", "arguments": f"query": ["Karl Weyprecht highest honors British geographical society", "Karl Weyprecht British geographical society", "Karl Weyprecht Founder's Medal", "Karl Weyprecht honors Royal Geographical Society"]]} </tool_call>
responsee
A Google search for 'Karl Weyprecht highest honors British geographical society' found 10 results:
## Web Results 1. [Karl Weyprecht- Wikipedia]
(https:/en..wkiipediaaa. hakkippdda.ogg/iiii//aa......
Missing: highest honors
</tool_response>
Round
<think
The search results strongly suggest that Karl Weyprecht is the central figure in question, with multiple sources mentioning his leadership of the 1872-74 polar expedition aboard
18

---

Published as a conference paper at ICLR 2026
the Tegetthoff and his later scientific initiatives. However, fully confirm his identity and gather all the necessary details-Such as the ship's name, the Austrian admiral it was named after, his role in proposing the International Polar Year, and whether he received the Royal Geographical Society's Founder's Medal-1 need a comprehensive, authorita- tive summary. Visiting the Wikipedia page dedicated Karl Weyprecht should provide consolidated account of his achievements, honors, and scientific contributions. I need to visit this page now to extract these specific details and definitively answer the question. </think>
<tool_call>
("name": "visit", "arguments": ("url": ("https://en.wikipe horkkppdda.og//wiiikkkee.. Weyprecht"),
"goal": "Extract details: vessel name, the Austrian admiral, international scientific initiative, Founder's Medal of Royal Geographical Society")) }}
</tool.call>
@hool.respons>>>
The useful information in https://en. wirkipedia.org/iiikikaarr Weyprecht for user goal Ex- tract details: vessel name, the Austrian admiral, international scientific initiative, Founder's Medal of Royal Geographical Society as follows:
Evidence in page: He co-led, with Julius von Payer, the 1872-1874 MMAustro-Hungarinn North Pole Expedition]] which discovered the archipelago [[Franz Josef Land]] the Arctic Ocean. The expedition's s ship "Admiral Tegetthoff"... </tool sttoo_resspone>>
Round
<think>
To arrive at the answer "Karl Weyprecht," pieced together several key clues from the ques- tion and verified them through targeted searches and authoritative sourees. First, I con- firmed that Weyprecht led a late 19th-century polar expedition aboard the ship "Admiral Tegetthoff." which was named after Austrian Admiral Wilhelm von Tegetthoff -tis di- rectly matched the ship and admiral mentioned in the question. Next, established that Weyprecht was instrumental in proposing the International Polar Year, major collabo- rative scientific initiative for the polar regions. Finally, I verified through multiple sources that he was awarded the 1875 Royal Geographical Society's 's Founder's Medal, one of the highest honors in British geography. The convergence of these facts leadersipp of the Tegetthoff expedition, advocacy for the International Polar Year, and receipt the RGS Founder's </think
that Karl Weyprecht the explorer described.
<answer>
Karl Weyprecht exemplifies the characteristics in your query: as commander of significant polar expedition (aboard the ship "Admiral Tegetthoff") after a promi- nent Austrian admiral, as a proposal target of major international scientific initiative (the International Polar Year), and a recipient of one of the most distinguished honors from major British geographical society (the Royal Geographical Society's 's Founder's Medal). </answer> V
Case Trajectory on SailorFog-QAA. Failed by Web-30B-GRPO
Question: A European naval officer, known for leading a significant polar expedition in the late 19th century aboard vessel named after Austrian admiral, later proposed major international scientific initiative focused on the polar regions. For his contributions exploration, he received of the highest honors from a major British geographical society. Who is this explorer?
Ground-truth Entities: [Tegetthofff, International Polar Year, Royal Geographical Soci- ety's 's Founder's Medal] Answer: Karl Weyprecht
19

---

Published as a conference paper at ICLR 2026
Round
<think>
The question involves European naval officer who led significant polar expedition in the late 19th century, had a vessel named after an Austrian admiral, later proposed ma- jor international scientific initiative for the polar regions, and was honored by prominent British geographical society. To pinpoint the individual, I need to connect the dots between the expedition, the vessel's name, the Austrian admiral reference, and the scientific initia- tive. Using search queries that combine these elements- Austrian admiral, the polar expedition, and the proposal of an international scientific ini- tiative-should surface historical records or articles that directly mention the explorer and clarify his contributions. By searching with these largeted terms, aim identify the officer and gather the necessary context for the rest of the question. need to conduct this search now to find the explorer's identity. </think>
as the vessel's naming, the
<tool_call>
("name": "scarch", "arguments": ("query": ["vessel named after Austrian admiral polar expedition late 19th century". "proposal major international scientific initiative polar regions", "later proposed major international scientific initiative polar regions"]}} s/tool_call>
stool_response:
Google search for 'vessel named after Austrian admiral polar expedition late 19th century' found 10 results:
## Web Results
1. Chustro-Hungarte North Pole expedition Wikipedia]
(https://en, wikipedia..rg/wikkiii Hattikii/uusttt-Huaaaaaaaan Lorraan_ooohh___aa Poreee_Poleeeee Pole_expedition) The main ship was the Tegetthoff, named for the Austrian Admiral Wilhelm von Tegetthoff, under whom Weyprecht had served. was built this expedition by Teklenborg Beurmann in Bremerhaven.
</tool response>
Round
<think>
The Wikipedia page for the Hunstr-Hungarian North Pole expedition likely contain authoritative details about the expedition's leadership, the role of the vessel named after an Austrian admiral, and key figures involved. Since the question requires identifying the naval officer who led this expedition, as well as his later contributions and honors, this page should provide not only his name but also contextual information about his career and achievements. By visiting this URL, can directly extract the name of the expedition leader and if there are references to his subsequent actions, such as proposing major scientific initiatives and receiving recognition from British geographical societies. I need to visit this page now to gather these crucial details. </think>
<tool.call> call>
("name": "visit", "arguments"" ["url": http:::://en.wiiieddiii wttp://wiiiidi...oiiiiiii/////// wrtp/wii///wwwttrrr
Hungarian North Poleeeeecexpetiiionn "goal": "Identify the leader of the expedition and aircraft details including his name, and whether he proposed Polar Year or similar initiative later, and awards")] </tool_call>
<tool.rr croo_rrspoose>>
The useful information in https:/en.wwkkipeia.orgg
Hungarian Hongarian.North. North_ Pol.eooeeeeeeeeee expeediitnnionn for user goal Identify the leader of the expedition and aircraft details including his name, and whether he proposed Polar Year similar initiative later, and awards as follows:
20

---

Published as a conference paper at ICLR 2026
Evidence in page: The Aungro-Hungarian North Pole expedition Arctic expedition to find the North-East Passage that ran from 1872 to 1874 under the leadership of Julius von Payer and Karl Weyprecht.
</tool_response>
Round
<think>
Although I have identified Julius von Payer as leader of the Austro- Huattr--nn North Pole expedition aboard the Tegetthoff and found that the expedition contributed to the idea of international polar scientific collaboration, have not yet pinpointed the exact honor from a British geographical society. This is crucial detail needed fully answer the question and distinguish his achievements from those of his contemporaries. By searching with focused queries such "Julius von Payer British Geographical Society honors', and "Weyprecht British Geographical Society honors', can directly target sources that clarify the connection between the expedition leaders and the specific honors received. I need conduct this search now to fill these key information gaps.
</think>
<tool _clll>
("name": "search", "arguments": ("query": ["Julius von Payer British Geographical Society honors", "Weyprecht British Geographical Society honors"]]} </tlol.call>
<tool response>
A Google search for 'Julius Payer British Geographical Society honors' found 10 sults:
## Web Results [Julius von Payer - Wikipedia]
(https://en.wii) wikipedia bikkiecdi.org/wiii/llliie org/wiki/Julius von Payer) He is chiefly known for the Austro- Hungarian North Pole expedition in 1872-74 and the discovery of Franz Josef Land
Round
<think>
To fully confirm whether Julius Payer received one of the highest honors from major British geographical society, and to directly tie him to any significant international scientific initiatives for the polar regions, need authoritative biographical details. The Wikipedia page dedicated to Julius von Payer likely to provide a concise summary of his major achieve- ments, awards, and any notable proposals or initiatives he was responsible for. By visiting this page, I can efficiently extract explicit references to British geographical society hon- ors ors-such the Royal Geographical Society's Founder's Medal or Patron's Medal and clarify his role in shaping international polar scientific collaboration. need to visit this page now to gather these definitive details.
</think>
<tool_calll call> ("name": "visit", "arguments": {"url": l"ttts::/en.wikipediiaa htt.s./r///ww///iiu...... www.-ooee"P...... "goal": "Extract awards of Julius von Payer, particularly any from British Geographical Society, and clarify his role in shaping international polar scientific collaboration") </tool.call>
<tool responses
The useful information in https://en.wikkk wwwipedia.org//eeee..e wwtp://////wwliisu....../ Payee for goal Extract awards of Julius von Payer, particularly any from British Geographical Society, and clarify his role in shaping international polar scientific collaboration as follows: </tool response>
Round
21

---

Published as a conference paper at ICLR 2026
<think>
To determine which expedition leader-Julius von Payer or Karl Weyprecht-reccivee the Royal Geographical Society's highest honors, need to clarify whether Payer was actually awarded the gold medal or the Patron's Medal, and in which year. The summary on the main Julius von Payer page referenced the Patron's 's Medal in 1971, but individual award listings on the Gold Medal (RGS) Wikipedia page should provide a definitive, year-by-year roster of recipients, ensuring accuracy regarding both the medal and its recipients. By visiting this page, I can confirm the exact year and whether Julius von Payer is indeed listed among the honorees, which is crucial for answering the question about his recognition by the British geographical society. I need to check this official list now confirm the details.
</think>
<tool call>
("name": "visit", "arguments": ("url": "https://en.wikipediia httpppedii.og/wiii//GGoaa. MeeaiiiiiooddMdaallll... Medal_ (RGS)"), "goal": "Check if Julius von Payer is listed recipient of the Royal Geographical Soci- ety's highest honor and }ear"}}
</tool.call>
<tool response>
The useful information in https://en.wikipeedee wikipedia.org// www.//owi//oo...////... Meed_MMeea__SSSSS for user goal Check Julius von Payer is listed as recipient of the Royal Geographical Society's highest honor and year as follows:
Evidence page: 1875 [[Julus Payer]] -Fo explorations and discoveries the Arctic regions.. </tool response>
Round
<think>
To arrive at the answer, I first noticed that the question described a European naval officer who led significant polar expedition the late 19th century aboard vessel named after an Austrian admiral, later proposed major international scientific initiative, and received top honor from a British geographical society. My initial searches confirmed that the Austro- Hungarian North Pole expedition (1872-1874), led by Julius von Payer and Carl Weyprecht, used the ship Tegetthoff, named after Admiral Wilhelm von Tegetthoff. Further investiga- tion revealed that while both men were key figures, it was Julius von Payer who awarded the Royal Geographical Society's Patron's 's Medal (the highest British geographical honor) for his achievements. Therefore, by piecing together the expedition details, the naming of the vessel, and the specific recognition from the RGS, concluded that the explorer in ques- tion is Julius von Payer. </think <answer>
Julius von Payer stands out as the notable European naval officer who fits all aspects of your query, having led landmark polar expedition, played key role in the foundation of international polar science initiatives, and been honored by the foremost British geographi- cal society for his achievements. </answer> X
22

---

Published as a conference paper at ICLR 2026
E
FAILURE ANALYSIS
Our extensive case study reveals that, in addition to the factual errors our paper targets at, agent failures frequently stem from several other distinct error modes, which we detail below.
Broken Reasoning Coherence: The agent loses reasoning coherence by failing to act on its previously stated goal, partieularly after Visit tool call.
Example:
<tool call>
("name": "visit", "arguments": ("url":
Farnese",
"goal": "Extract notable architectural features, innovative aspects, and check perspective scenery introduction")] </tool_call>
<tool.response>
The useful information...a theatre in which the audience views the action through single frame...
</tool response>
<think> To determine whether the Teatro alla Scala is relevant to the question or if it is merely a famous modern opera house... </think>
In this example, the agent exhibits a loss of reasoning coherence. While it correctly visits the page and extracts a key fact ("a single frame"), it subsequently ignores its own goal for that visit ("notable architectural features") and instead begins reasoning about an unrelated subject.
Distracted Querying: The agent loses focus on the core objective and instead pursues sec- ondary information triggered by minor details, deviating from the optimal solution path, finally exceeding the context length.
Example:
Question: ... What year saw the first performance of this multi-screen instrument by the aforementioned performer? <tool_call>>
("name": "search", "arguments": ("query": ["multi-screen instrument designed early 1970s performer Charlotte Moorman"]] //tool.call>
<tool call>
("name": "search", "arguments": ("query": ["MoMA instrument multi screen"]]]
</tool_call>
this example, the agent fails to maintain its core objective. It correctly formulates an early query that includes the temporal aspect ("early 1970s"), but then gets distracted by other entities ("MoMA"). The agent forgets that its primary task was to find the year, as shown by the absence of any time-related keywords in its later search query.
Information Overload from Concurrent Queries: By executing multiple search queries within a single step, the agent generates a large volume of unstructured text. This information overload overwhelms its processing capacity, causing it to fail identify or retain the most critical facts necessary for the subsequent reasoning step.
Example:
23

---

Published as a conference paper at ICLR 2026
<tool_call>
{ ("name": "search", "arguments": ( ("query": l"appointed board central banking sys- 2016", "appointed serve on board central banking system appointed 2016", "ap- pointed board central banking system later became Chair 2016"]}}
</tool call>
In this example, the agent executes three highly similar queries simultaneously. While only the third query is well-targeted, containing the critical term "Chair", the valuable information it returns is buried within the larger volume of text from all three searches. Overwhelmed by the dense context, the agent fails to isolate the key signal from the noise, which misleads down flawed reasoning path.
Our case study indicates E-GRPO mitigates broken reasoning coherence. By incentivizing the agent carry forward critical entities into its thought process, the entity-aware reward implicitly enforces topical continuity and prevents the reasoning from drifting into irrelevance.
The other identified failure modes, however, suggest promising avenues for future work:
For distracted querying, a potential solution to periodically re- re-inject the original ques- tion into the agent's context, orienting the agent's focus towards the primary objective.
We attribute information overload primarily to the inherent limitations of the base model in processing long, unstructured contexts. Addressing this could involve either leverag- ing more advanced models with superior long-context capabilities or designing penalty mechanism for imprecise or redundant queries.
F DECAYING ENTITY MATCHING WEIGHTS
alpha
GAIA
BrowseComp
BrowseComp-ZH
xbench-DS
0.3
48.5
12.9
26.4
46.7
decay
48.2
12.8
26.2
47.3
Table The Pass@]
performance of different entity matching weights.
We also train the 30B model with an alpha that linearly decays from 0.3 to 0.0 over first steps (80 steps in total). The results are presented in Table
Comparing with a fixed 0.3, the results show no clear or consistent advantage for the decaying alpha strategy. This suggests paper's choice of simple, fixed alpha as practical and effective setting. However, we believe a dynamic alpha holds potential. It's possible that applying the decaying schedule over longer training horizons could be more impactful, which is promising direction for future research.
G EXTENDED TRAINING DYNAMICS
We extend the training of 30B models from 80 steps to 120 steps with the Web environment. As shown in Figure O the results align with our analysis in Section 4.3] While both methods' perfor- mance gradually converges, E-GRPO maintains consistent lead over GRPO. Crucially, E-GRPO also continues to use fewer steps on average. This demonstrates E-GR E-GRPO's dual advantages in both effectiveness and efficiency.
H ROBUST MATCHING ANALYSIS
While exact string matching is computationally efficient and effectively mitigates reward hacking (see Appendix(B.2) its rigidity presents practical challenges when faced with natural language vari- ations such as alternative spellings, synonyms, abbreviations. To address this limitation,
24

---

Published as a conference paper at ICLR 2026
Comparison Accuracy
Comparison Steps
0.65
GRPO Accuracy
GRPO Tool Call
E-GRPO Accuracy
E-GRPO Tool Steps
Avera Training Accuracy
13.0
0.60
Awereda Call
12.5
0.55
11.5
11.0
0.45
10.5
100
120
Training Stop
Training Stop
Figure 6: Extended training dynamics of 30B models with the Web environment.
introduce "Robust Matching" strategy. This approach involves first prompting an advanced LLM to generate set of 5-10 plausible variations for each ground-truth entity, detailed in the prompt perform exact string matching against this expanded set of candidates. An entity is matched if any of its generated variations are found in the agent's 's thought process.
below. Subsequently,
Prompt
I need to match an entity in a text, but need to account for various paraphrases, abbrevia- tions, and potential spelling variations. For each of the entities listed below, please provide 5-10 alternative phrasings or variations that I can use for more robust matching system.
When generating the alternatives, please consider:
1. Synonyms and Paraphrasing: Using different words sentence structures to convey the same meaning.
2. Abbreviations: Common initials, acronyms, shortened forms.
3. Spelling Variations: Including regional differences (e.g., US vs. UK English) or common typos.
4. Formality: Both formal and informal ways of referring to the entity.
Note that you do have include words like "the" "a" in your responses in order for robust matching.
I will give you the entity, and you should directly return list alternative phrasings valid JSON list format like ["phrasing 1", "phrasing 2", ...]. do not include any other text in your response. Entity (entity)
With this "Robust Matching" strategy, we conduct the analysis of entity matching and correctness. Figure 7 visualizes the efficacy of this strategy. Compared with exact string matching, the "Robust Matching" strategy demonstrates much stronger correlation with factual correctness. Specifically, for correct samples, the density peak match rate of significantly higher. Conversely, for incorrect samples, the peak at 0.0 is reduced, and the distribution becomes more spread out, better capturing misses.
This analysis confirms that "Robust Matching" serves more accurate proxy for factual cor- rectness. Motivated by this finding, we conducted an additional RL experiment using the "Robust Matching" strategy, keeping all other settings identical to Web-30B-E-GRRI The results pre- sented T
Strategy
GAIA
BrowseComp
BrowseComp-ZIi
xbench-DS
Exact String Matching
48.5
12.9
26.4
46.7
Robust Matching
49.2
12.9
26.8
47.0
Table 4: The Pass @1 performance of different entity matching strategies.
25

---

Published as a conference paper at ICLR 2026
2.
Sample Denaty
2.0
14
35
us Nomalizedd (Thought) Match
Normalized (Thoughti Entity Match Rate (Paraphrased)
Figure 7: The entity match rate of exact string matching and robust matching.
The results are promising. The agent trained with "Robust Matching" achieves slight performance gains on three of the four deep research benchmarks, further validating the benefits of it. We consider the continued refinement of this strategy a promising direction for future work.
I EXPLANATION OF THE ANALYSIS FIGURE
Here we clarify the upper-right subfigure of Figure [ For question we sample 8 responses "1,, r'y for it. Each response matches m. entities. correct responses and that of incorrect ones. For example, if only 22, r' are correct, then correct rate is (m, + m2 + m3)/3 and wrong rate is (mı + m2; + m2 + my + ms)/5. We compare these two rates for each question. The analysis shows that for most of the questions, correct rate is larger than wrong rate, i.e., correct responses of each question often match more entities than wrong ones. This establishes strong correlation between the entity match rate and the correctness of the final answer.
calculate the average matched entities of
J USE OF LARGE LANGUAGE MODELS (LLMs)
In this paper, we leverage Gemini-2..5PPra (Google Team 2025a to polish paper writing grounded on human-written content. The usage is limited to language polishing. We are responsible for any contributions generated by LLMs.
26


Reviews:
[{"review_id": "Lzk5eNH20x", "reviewer": "Reviewer_oC3b", "summary": "The paper proposes an improvement to search agent RL algorithms such as Group Relative Policy Optimization (GRPO), which treat so-called near-miss rollouts as bad as complete miss rollouts. The authors propose entity-awareness for the reward function, which assigns partial credit to rollouts, leading to a denser reward function.\n\nThe approach is evaluated for QA datasets such as TriviaQA or PopQA, and Deep Research datasets such as GAIA or BrowseComp. A local (Wikipedia) and web setting (Google Search, Jina) are evaluated. Baselines are GRPO as well as ReAct-based agent variants.\n\nThe results show that the approach improves the performance of the search agents compared to GRPO, and is also on par or superior when compared to other agents.", "questions": "- Did you experiment with decreasing alpha during training?\n- Can this approach be transfered to other domains with less entities?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- Well-motivated improvement: Model intermediate rewards for partially correct rollouts\n- Improvement over GRPO visible in empirical evaluation\n- Stabilization of training process increases efficiency", "weaknesses": "- Empirical gain rather low compared to GRPO", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hUbLYI9v3U", "reviewer": "Reviewer_fNgM", "summary": "This paper addresses the reward sparsity problem in training LLM-based search agents with reinforcement learning. Current methods totally assign a zero reward even if the trajectory contains partially correct information. The authors use an entity-centric method to synthesize a dataset and save the entities as ground-truth information. Then they add an extra entity-matching reward to improve performance.\n\nContributions: \n* Identifying the \"near-miss\" problem: The paper shows that standard GRPO fails to distinguish informative partially correct samples from complete failures in search-agent training.\n* Entity-aware reward function: A novel reward formulation that assigns partial credit based on entity match rate", "questions": "1. **Figure 1 (upper-right) requires clarification**. I cannot fully understand the upper-right panel of Figure 1, especially the meaning of the x-axis labels. I would like to request more details and straightforward explanations of it.\n2. **Figure 1 (lower-right) distribution analysis**. In the lower-right panel, the distribution shows cases where the entity match rates are not 1 but still lead to correct answers. Are these instances of hallucination where the model just happened to get it right?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **Well-motivated problem**. Although partial correctness has been studied in RL, introducing it in training search agents is a good problem to facilitate this community.\n* **Technical soundness**. The solution using the entity-matching score is reasonable, and the analysis of the relationship between accuracy and the matching score supports the insights of the proposed method.", "weaknesses": "1. **Handling of incorrect reasoning with correct entities**. The entity-matching reward may inadvertently credit erroneous reasoning paths that happen to mention correct entities without proper understanding. For instance, a model might generate factually incorrect statements while coincidentally including the right entity names. The paper does not address how to distinguish between genuine entity identification and spurious mentions.\n2. **Overclaim**. While the entity-aware reward is effective, the core contribution is to augment GRPO with an additional reward term. The branding as \"E-GRPO\" may overstate the methodological advance, as the fundamental GRPO framework remains unchanged. A more modest framing as a reward technique might be more appropriate.\n3. **Insufficient baseline comparisons in controlled settings**. The Local environment experiments (Table 1) would benefit from comparisons with more approaches that are feasible in controlled settings.\n4. **Unexplained performance degradation**. The results show E-GRPO slightly underperforms on PopQA. The paper does not analyze or explain this anomaly.\n5. **Absence of systematic error taxonomy and failure analysis**. Given the lengthy reasoning traces, the paper would benefit from categorizing failure modes (e.g., entity identification errors, reasoning chain breaks, etc.). Besides, the paper does not provide any concrete failure cases of the E-GRPO. While the case study in Appendix E shows a successful E-GRPO trajectory compared to a failed GRPO one, there is no analysis of cases in which E-GRPO fails despite its entity-aware reward.\n6. **Entity matching robustness concerns**. While the authors justify exact string matching, they do not address practical challenges such as spelling variations, abbreviations, etc. These issues could significantly degrade the quality of the reward signal in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x9ZscSLDFR", "reviewer": "Reviewer_4vB2", "summary": "This paper introduces E-GRPO that improves the performance of LLM-based search agents. E-GRPO extends the vanilla GRPO with a denser, entity-aware reward function. It reuses the ground-truth entities from the synthetic training data to assign partial credit to incorrect answers based on how many key entities they successfully identified. Experiments show this approach allows the model to learn more effectively, consistently outperforming the GRPO baseline in accuracy and learning more efficient policies that use fewer tool calls.", "questions": "- It appears that the training datasets for E-GRPO and other baseline models are different because the performance scores are directly cited from those external publications. Can authors clarify how much performance improvement comes from discrepancy in training data?\n- While uniform credit assignment is a common practice in many GRPO-like RL training algorithms, is it technically sound and mathematically stable in E-GRPO when the partial entity matching reward is also uniformly assigned to the tokens, regardless of whether that specific token's \"turn\" was the one that found an entity or the one that made a mistake?\n- What is the accuracy trend of GRPO and E-GRPO after 80 steps in fig. 3? Is it possible that the training happened to stop at the point where the difference is large, given the large fluctuation in both training curves?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The reward formulation originated from a good observation in the data and its design is clear and reasonable.\n- The empirical performance appears to be strong and better than vanilla SFT & GRPO.", "weaknesses": "- The research contribution appears to be incremental. While E-GRPO seems to outperform GRPO, I am not fully convinced E-GRPO is a fundamentally “novel framework” compared to the original GRPO given the fact that it only customizes the reward function. Partial entity matching in RL is also an established method, especially in NL2SQL field (e.g. Reasoning-SQL).\n- It is nuanced whether the performance comparison with other baseline models in the main results is a fair comparison using same training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iWDZosceuB", "reviewer": "Reviewer_LHoD", "summary": "This paper introduces a reinforcement learning framework Entity-aware Group Relative Policy Optimization (E-GRPO), to enhance the training of LLM-based search agents. Existing GRPO methods usually rely on sparse, outcome-based rewards, treating all incorrect outputs equally and overlooking partially correct cases. In contrast, E-GRPO uses entity-centric information from synthetic training data by assigning partial rewards based on the entity match rate, i.e., the proportion of ground-truth entities correctly identified during reasoning. Experiments also show that entity-aware rewards provide a computationally efficient and semantically rich supervision signal for training search agents.", "questions": "1.\tThe notion of “ground-truth entities” in reasoning requires clarification. Are there cases where multiple reasoning paths are valid, and if so, how does the method define or handle ground-truth entities in those scenarios?\n\n2.\tHow well does E-GRPO agent generalize to other **question** domains or verticals that differ from the synthetic training data?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tComprehensive experimental results are reported, including diverse LLM-based search agents and datasets.\n\n2.\tThe experimental settings and evaluation details are clearly provided to facilitate reproducing.\n\n3.\tThe paper is generally clearly-written and easy to follow.", "weaknesses": "1.\tThe empirical finding of “strong positive correlation between the number of ground-truth entities identified during the agent’s reasoning process and final answer accuracy” is relatively trivial and expectable. This simply follows the fact that correct reasoning paths naturally tend to include more correct entities.\n\n2.\tWhile the idea of entity-aware rewards is useful, the contribution seems an incremental improvement on GRPO rather than a fundamental paradigm shift. The core update is scaling reward by entity match rate, which is conceptually straightforward.\n\n3.\tIn the introduction, it argues that PRM methods cannot handle open-ended, dynamic nature due to the need for step-wise annotation. However, the E-GRPO also needs the ground-truth entities for the complete reasoning path to compute the partial reward. This seems to suggest that E-GPRO also cannot handle the tasks that PRM cannot. The utility needs justification.\n\n4.\tThe entity annotations in synthetic datasets are assumed correct and reliable. If entity extraction or labeling is noisy, the reward signal may mislead training. The paper could better address this potential sensitivity to data quality and generalization to non-synthetic data.\n\n5.\tThe paper does not sufficiently address how well E-GRPO generalizes to real-world search questions that differ substantially from the synthetic data distribution.\n\n\nMinor:\n\n1.\tIn the abstract, “We address this by leveraging the very entities discarded during training.”, is the word “very” a typo?\n\n2.\tThe related work section should be a part of the main content to make it self-contained, instead of appearing only in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)