Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs
Authors: Kang An
Ziliang Wang
Xuhui Zheng
FaQiang Qian
WeiKun Zhang
Yuhang Wang
Yichao Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Erasable Reinforcement Learning
Multi-hop Reasoning
LLM-based Agents
Primary area: reinforcement learning

Abstract: While search-augmented large language models (LLMs) exhibit impressive capabilities, their reliability in complex multi-hop reasoning remains limited. This limitation arises from three fundamental challenges: decomposition errors, where tasks are incorrectly broken down; retrieval missing, where key evidence fails to be retrieved; and reasoning errors, where flawed logic propagates through the reasoning chain. A single failure in any of these stages can derail the final answer. We propose Erasable Reinforcement Learning (ERL), a novel framework that transforms fragile reasoning into a robust process. ERL explicitly identifies faulty steps, erases them, and regenerates reasoning in place. This targeted correction mechanism turns brittle reasoning into a more resilient process. Models trained with ERL, termed ESearch, achieve substantial improvements on HotpotQA, MuSiQue, 2Wiki, and Bamboogle, with the 3B model achieving +8.48% EM and +11.56% F1, and the 7B model achieving +5.38% EM and +7.22% F1 over previous state-of-the-art(SOTA) results. These findings suggest that erasable reinforcement learning provides a powerful paradigm shift for robust multi-step reasoning in LLMs.

Published as a conference paper at ICLR 2026
000
ERASE TO IMPROVE: ERASABLE REINFORCEMENT LEARNING FOR SEARCH-AUGMENTE LLMS
001
002
003
004
Kang An 1 Ziliang Wang Xuhui Zheng' 1,3-f
005
Faqiang Qian', Weikun Zhang'. Yuhang Wang Yichao Wu'
006
SenseTime "Shenzhen University "Nanjing University (wangzi Linnziliangg, qianfagiang, zhangweikum wangyuhang, wuyichao ) @senset ime. com
007
008
009
ankang@gm1 wcw.e.mmm.a.........mmmo.oo....a ac.en, zhengxuhui @smaiii@@mmmmii 1 n ju en
010
011
012
013
ABSTRACT
014
015
While search-augmented large language models (LLMs) exhibit impressive ca- pabilities, their reliability in complex multi-hop reasoning remains limited. This limitation arises from three fundamental challenges: decomposition errors, where tasks are incorrectly broken down; retrieval missing, where key evidence fails to be retrieved; and reasoning errors, where flawed logic propagates through the reasoning chain. A single failure in any of these stages can derail the final answer. We propose Erasable Reinforcement Learning (ERL), a novel framework that trans- forms fragile reasoning into a robust process. ERL explicitly identifies faulty steps, erases them, and regenerates reasoning in place. This targeted correction mech- anism turns brittle reasoning into more resilient process. Models trained with ERL, termed ESearch, achieve substantial improvements on HotpotQA, MuSiQue, 2Wiki, and Bamboogle, with the 3B model achieving +8.488 8.48% EM and + 1.56% F1, and the 7B model achieving +5.38% EM and +7.22% F1 over previous state-of- (SOOTA) results. These findings suggest that erasable reinforcement learning provides powerful paradigm shift for robust multi-step reasoning in LLMs.
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
INTRODUCTION
031
032
033
Large language models (LLMs) have produced remarkable advances across broad spectrum of natural language processing tasks, including question answering, reasoning, and code generation (Singh et al. 2025) Yang et al. 2025). Notwithstanding these advances, inherent limitations in their static pretraiming corpora leave them susceptible to hallucination and factual error, especially in knowledge-inten inndgg--ittesiie domains and in tasks that require reasoning over multiple steps (Huang et al. 2025a 2024 Even the most advanced models tailored for rigorous reasoning, such OpenAl ol (Jaech et al. 2024), DeepSeek RI (Guo et al. 2025) and Kimi k2 (Team et al.] (025), still face substantial difficulty in reliably solving complex multi-hop problems that demand precise decomposition, dependable retrieval, and long-term logical consistency (Xi et al. (025). To address these difficulties, cerreval-augmentee generation (RAG) has emerged as a dominant paradigm, enriching large language models with external knowledge sources (Lewis et al.) 2020). Over time, RAG has evolved into sophisticated research agents that integrate search and reasoning within an autonomous loop. Systems such as OpenAl Deep Research (OpenAl 2025). Gemini Deep Research (DeepMind) 2025), and Perplexity Deep Research (AI. 2025) mark significant milestones in this trajectory. Reinforcement learning (RL) (Li 2017) has emerged as central force driving recent
034
035
036
037
038
039
040
041
042
043
044
045
046
breakthroughs in the field of aarrehau agents. An increasing number of studies explore leveraging RL to guide decomposition, retrieval, and reasoning (Jin al., (0256)] [Song al.) [205] Zhao et al.) 2025b). These approaches employ reward signals to improve sub-queri generation, evidence retrieval, and reasoning chains, achieving substantial gains challenging benchmarks such
047
048
049
050
051 1 Work done during internship at SenseTime
'Equal contribution
052
053 I Project leader
8 SCorrespondigg author

---

Published as a conference paper at ICLR 2026
054
2022). and 2WikiMultiHopOA (Ho et al. 2020). Despite these impressive advances, current systems remain highly brittle. They can reliably answer simple factual queries, yet even minor errors in decomposition, retrieval, or reasoning can compromise an entire multi-hop trajectory (Huang et al. 2025b Li et al. rarely fail catastrophically. When recognize flaw reasoning step, we pause, correct the mistake, and continue from the corrected point. This stark contrast highlights a critical limitation of current search-augmented RL systems: they lack the robust self-correction mechanisms that underlie
HotpotQA Yang et al. 2018). MuSiQue (riveddi et al.
055
056
057
2025). In contrast, humans
058
059
060
061
human reasoning.
062
Through extensive empirical analysis of current RL-based search agents, we uncover three critical failure modes that fundamentally limit their capabilities:
063
064
065
Decomposition Errors: Incorrect subqueries derail the retrieval process entirely, preventing
downstream steps from ever accessing the crucial evidence needed to answer the question.
066
067
Retrieval missing: Retrieved documents that are irrelevant even with appropriate subqueries due
068
to noise, ambiguity, or incomplete coverage, causing subsequent reasoning to fail.
069
Reasoning Errors: LLMs may make mistakes when integrating retrieved information, and these errors accumulate across steps, systematically undermining the reliability of the final answer.
070
071
072
Deeper structural flaws significantly exacerbate these issues. Existing reinforcement learning agents typically treat the entire search and reasoning trajectory as single Markov Decision Process (MDP) (Kaelbling et al.] 1996), optimizing only via sparse terminal rewards (Jin et al., 2025b). This monolithic design is fundamentally brittle: single misstep can compromise the entire trajectory.
073
074
075
As reasoning chains extend (Zhang et al. 20255 Jin et al. (025a). this fragility intensifies, causing performance to degrade precipitously beyond ten steps (Gao et al. 2025).
076
077
078
Overcoming these limitations requires a radical paradigm shift: agents must emulate human- self- correction by detecting error,, discarding flawed steps, and resuming reasoning from the most recently corrected state. Analogous to a skilled writer using an eraser to remove a single mistaken word without discarding the entire manuscript. We introduce Erasable Reinforcement Learning (FRL), novel framework that embodies this principle. ERL enables search- sagee--uggmeted LLM agents to identify errors in decomposition, retrieval, or reasoning precisely, selectively erase the faulty segments, and regenerate from the last correct state. This fine-grained corrective mechanism transforms brittle trajectories into resilient ones, allowing agents to recover gracefully from mistakes rather than collapsing entirely. We conduct extensive evaluations on HotpotQA (Yang et al. 22018). MuSiQue (Trivedi al. 2022), 2WikiMultiHopAA (Ho et al., 2020), and Bamboogle (Press al., 2023). The results show that models trained with ERI. not only surpass strong baselines and state-of ofate-of-the-art (SOTA) methods but also consistently improve performance; the 3B model achieves gains of +8.48%
079
080
081
082
083
084
085
086
087
088
EM and +1 1.56% FI, while the 7B model achieves +5.38% EM and +7.22% Fl.
089
090
The main contributions are summarized as follows:
091
092
Systematic identification of three critical failure modes in search-augmented LLMs for complex multi-hop reasoning.
093
094
Introduction of ERL, a framework for fine-grained error detection, erasure, and regeneration that substantially improves reasoning robustness.
095
096
Establishment of new SOTA results on multiple multi-hop QA benchmarks, validating the effec- tiveness and generality of ERL.
097
098
099
PRELIMINARY
100
101
Previous work often models complex multi-hop question answering, combining search and reasoning. as MDP characterized by (Chen et al. 2024):
102
103
104
AS,A,P,R,)).
105
In this framework, the state s, represents the reasoning trajectory up to time t, providing context for subsequent actions [Broekens et al. 2010). We define the state 8 as:
106
107
8 (Q, H.) (Q, (an, eo), (a1, e1), ...,, (a-));
(2)
2

---

Published as a conference paper at ICLR 2026
108
where is the original question and H, is the sequence of interactions up time Each action a, E A represents reasoning or retrieval, and each environment e, corresponds to the evidence information resulting from o; by calling the tools. The agent's action space A = {o,r, g) includes atomic operations governing reasoning. Searching and Answering, while the tool corresponding to the environment will provide searched documents:
109
110
111
112
113
114
Search Query (g:): Produces query qe to retrieve relevant evidence et.
115
Observation (o:): Reasoning an observation o, of the evidence et and previous status 8t-1
116
Sub Answer (ra): Yield an intermediate phased conclusion after observation Or.
117
118
Finish(answer): Produces the final answer Annal when sufficient evidence is gathered.
119
120
The state transition function P(s++1 St: ae) (Huang. 2022 driven by two mechanisms: the stochastic generation by the LLM and the search results from external search engines. For the convenience of representation and to fit the complex answering mechanism of multi-hop QA, each action is defined as fixed and ordered sequence of unit actions of {(oz, rz), 9t) (no observation or response to previous evidence the first round.) for intermediate solving process, or (A (ina))) for the answer to finish. All the action and action sequence is sampled from the LLM's conditional distribution:
121
122
123
124
125
126
= Act(s;) ~ Pol- st).
127
128
For the Search Query (gr) issued by the agent, the environment e, the information evidence retrieved from the search tools:
129
130
C Search(q:).
131
The next state 1 is formed by appending the combination of the new action sequence and environ- ment to the trajectory He:
132
133
134
9t11 = (Q, (an, a E ((o,.r)). intermediate step,
135
(5)
(Annal),
final answer.
136
137
In multi-hop question answering, the reward function R Jin al. 2025b Song et al. 2025 is typically sparse, rewarding the agent only upon completing the reasoning trajectory and producing the final answer Asmat- The reward is computed by comparing Agnal with the reference Agold using metrics like exact match (EM) or F1 score:
138
139
140
141
142
Rst, 0.)
{ Eval(Annal Agola) az is (Afénal);
(6)
143
otherwise.
144
The agent optimizes the expected terminal reward via policy gradient methods:
145
146
J( = E, [R(T)]:
147
148
where
(So, Go, co...., is the reasoning trajectory. However, treating the entire reasoning trajectory as a monolithic sequence for optimization introduces a structural vulnerability, known as catastrophic fragility. In reasoning trajectory (so, ag Co, .1.... @r 11-1,6T-1 -r 87), any failure at single step can disrupt the entire process, leading to an erroneous final outcome. For instance, an error step < T result from:
149
150
151
152
153
Decomposition Errors: The Generatequery action generates deviated sub-query ge-
154
155
Retrieval Omissions: The Search action fails to retrieve the relevant evidence ét-
156
Reasoning Errors: The Synthesize action produces an incorrect intermediate conclusion r..
157
158
The eгror at 8 contaminates all subsequent states (s142)..., 87), as each following action (Ol+1...., ar-1) depends on this contaminated history. This resembles a domino effect, where the failure of single link leads to entire system collapse. This structural fragility is the reason for the unreliability current caachh-uggmented LLMs when tackling complex, multi-hop problems.
159
160
161
3

---

Published as a conference paper at ICLR 2026
162
3
METHOD
163
164
3.1
REINFORCEMENT LEARNING WITH A SEARCH ENGINE
165
166
We extend reinforcement learning incorporate search engines into policy optimization. The
167
objective is
168
max Eg y-maa--a:R) (r, w)] - DDKI[ToT m; R) Tee (y my R)]
169
Tg
170
where Te is the policy, ref the reference model, and Ti the reward (Jin et al. 2025b). Inputs x contain
171
both natural language and retrieved results, enabling TT learn retrieval- reasoning integration beyond
172
prompt-based methods. For training we adopt Proximal Policy Optimization (PPO) (Schulman et al.
173
2017 yielding
174
175
JPPO (0) EgyDD (123))
(w)) min
Te( Meer 2;; R) Ac: Told (3/e Mo 2;; R)
I
176
177
(8)
178
clip
To Told (vc,,,;;).;;;))
1, 1,
179
180
with Told the previous policy, (w) masking retrieved tokens, and A, the advantage from GAE
181
(Schulman et al. 2015).
182
183
3.2
HOUND-BASER REASONING
184
185 We model reasoning as a sequence teraction pair (al,et): where at denotes the action and e, the retrieved evidence. If a ((o,,7)), rc), 91) the agent executes the sequence <observation> or </observation> - <sub answer> т, </sub answer> <search> qi </search>, and the query q is sub- mitted to a search tool to obtain evidence et. If a; = (Amal), the process terminates with the final answer. The policy action units can be defined on the original question and previous action unit in the dialogue history h; = {((0), Tı), qi, es) ]
structured rounds. Each round t produces an in-
187
188
189
190
191
192
Or ~ ( he) + rt ( Q (he, 01)) - qt = no( Q, (he, (or, + e = Search (g:).
193
(9)
194
This structured format allows the agent alternate between querying and reasoning, tightly cou- pling retrieval with generation. The episode terminates when the policy outputs (Afinal) like <answer> Annal </ </answer>.
195
196
197
198
3.3
REWARD DESIGN
199
Dense stepwise rewards are critical to prevent sparse supervision. ERL introduces two intermediate rewards, Rreaarc for sub-queries and 1uau gsub_answer for intermediate reasoning, in addition to the final reward panuswer.
200
201
202
203
Search reward. Let gold evidence D (4;4_ and retrieved set D(O) ( Define
204
TF-IDF cosine similarity s(d;; d Maintain coverage vector m/:
205
206
207
< = max sdd; )o), A max(c m 1- 100,, -
mf = max(m ..... A
208
209
(10)
210
Redundancy penalty is defined as Eq. [I] and the final search reward Eq.
12 as below. This
211
design encourages novel evidence retrieval while suppressing repeated queries.
212
4
213
pt p==
l(d)) -1'1),, J=1
Σ
= )
(11)
215
Rrearch Gt pd.
(12)
4

---

Published as a conference paper at ICLR 2026
216
Sub-answer reward. Let gold sub-answers A = {at) With FI overlap:
217
218
S (1(r),)))) uf max [1,,), 63 maxfut ",=",0,, = max of.
(13)
219
Reward:
220
Re raub_answer m,
(14)
221
222
This ensures that only genuine improvements to intermediate reasoning rewarded.
223
224
Final reward.
Ranswer (Mmaaa Agola) + F1 Agold).
(15)
226
The token-level attribution aligns RRpaach with </search>, R, prub_answw answer with < </bbeevvtioo>> and </sub_answer and Ranswer with </answer>.
227
228
229
3.4
ERASABLE REINFORCEMENT LEARNING
230
231
Rewards alone cannot prevent compounding errors. ERL introduces erasure operators that surgically remove faulty parts of the trajectory, enabling as shown in Figure [] We define a trajectory as (8o, -1,--., 8T). For any / < T, denote the truncated prefix of the trajectory up step t by
232
233
234
Toxt (su. $1 $ st).
(16)
235
236
We further introduce an erasure operator E, which modifies the action sequence according to different conditions in each round. Formally,
237
238
(Non)),
if the sub-answer Ta is incorrect,
239
240
((),
if the initial or subsequent search results are incorrect,
(17)
241
(Lor, qt), er, if the action sequence valid. $1+1 = Toxt + {s,ct(s:), Search(q:) = To. + Ela,, eg].
242
243
244
Different erasure conditions be explained with thresholds are introduced: for local errors and 3 for plan-level errors. And here goes the details:
245
246
247
Sub-Answer Erasure. If Reub_anawe answer < erase <observations <sub_ _ambb__sser>>> and all subsequent actions of round t, meaning any actions the current round are discarded:
248
249
250
St+1 Tu.s + (None)
St.
(18)
251
Subsequent Search Erasure. If Rrearch < a and t > 1, erase the query behavior <search> issued round and keep the correct <observation> with <sub_answer>;
252
253
254
Toat @ (2),
(19)
255
256
Initial Search/Plan Erasure. If 12qearch < the first action round of no observation or sub-an sub-answer action unit and erase the query behavior <search> (reset the trajectory):
257
258
T + To.0 @ (None) = s0.
(20)
259
260
261
262
EXPERIMENT
263
264
4.1
EXPERIMENTAL SETUP
265
266
Datasets and Evallution Metries We evaluate four multi-hop QA benchmarks: HotpotQA
267
(Yang et al. 2018). 2WikiMultihopA (Ho et al. 2020). MuSiQue (Trivedi et al.] 2022). and Bamboogle (Press et al. 2023), which span diverse domains and reasoning complexities. We report performance using canonical word-level and Exact Match (EM) metrics, while refraining from the of third-party L.M evaluators owing concerns regarding reproducibility and stability.
268
269
5

---

Published as a conference paper at ICLR 2026
270
271
272
273
Search
274
Information
275
Observation
276
Sub Answer
Torget
Chock
277
278
/searrch>
</observation>
</searc///rch>>>>
</observation>
Res.
279
Question
Plan
Final Response
Erase & Retry
Update
280
281
Figure 1: Overview of ESearch. Different colors and symbols are used to represent the interactive
282
behaviors S (Search), I (Information), O (Observation), and A (Sub Answer). In the answering process, there are three types of erasure and retry behaviors: (1) incorrect initial search results trigger initialization plan erasure; (2) incorrect subsequent search results trigger search erasure; (3) incorrect sub-answer triggers observation erasure. addition, history checking and target set are built for
283
284
285
286
searches and sub-answers respectively to evaluate value gains, which serve as the basis for erasure triggers and reward calculation.
287
288
289
290
Method
HotpotQA
2Wadd
MuNiQue!
Bamboogle'
HolpoiQA
2WiRI
MuSiQue'
Bamaboogle'
291
EMT
F7f
RMT
FJ"
EAST
FI1
EMT
F/1
FMT
F7T
FMT
F17
FMA
FI1
BMAT
F11
Qwem25--h-Bussee Bu-ne----BB
292
Search RJ base
0,272
0.361
0.248
0.296
0.081
0.146
0.176
0.270
0.348
0.431
0.381
0.445
0.120
0.184
0.280
0,400
293
Search-R Rl-itstmet
0.304
0.401
0.293
0.352
0.120
0.188
0.240
0.344
0.350
0.442
0.371
0.452
0.126
0.195
0.392
0.513
294
ZereSeurch base
0,281
0.377
0.253
0,311
0.096
0.164
0.165
0.256
0.324
0.414
0,392
0.473
0.152
0,237
0.261
0.522
0.267
0.239
0.288
0.085
0.145
0.393
0.299
0.357
0.453
0.355
0.441
0.114
0.176
0.421
295
R. R.Sench-iieenn--GGPPP
0.329
0.427
0.307
0.351
0.131
0.208
0:228
0.327
0.374
0.460
0457
0.519
0.142
0.227
0.504
0.644
R-Seuch-innoout-PP
0.289
0.381
0.277
0.328
0.124
0.187
0.260
0.355
0.308
0.495
0.496
0.558
0.152
0.234
0.4%
0.056
296
SSRL -inttnctt
0.314
0.405
0.250
0.348
0.093
0.156
0:316
0.267
0.346
0.424
0,365
0.461
0.114
0.195
0.344
0.453
297
SurpSearc--usee
0.329
0.434
0.319
0.395
0.181
0.273
0.328
0.419
0.345
0.464
0.434
0.542
0.196
0.291
0.502
0.631
StepSeanc--isirmet
0.345
0.452
0,330
0.385
0.174
0.261
0.344
0.452
0.394
0.470
0.402
0.496
0.150
0.240
0.520
0.636
298
[Search base
0.438
0,499
0.345
0.414
0.529
0.438
0.385
0.581
0.684
0.247
0.367
0.653
299
ESeurhh instruct
0.447
0.587
0.415
0,500
0.232
0.339
0.446
0.587
0,513
0.612
0,521
0.644
0.211
0,311
0.674
0,813
300
Owen2.5-76 B7--5--se-eBaessttrrucc
Search RI base
0,432
0.567
0,350
0.411
0.296
0.290
0.450
0.545
0.508
0.610
0,533
0.607
0.219
0.310
0.577
0.692
301
Search- R1-instruct
0.394
0.312
0.376
0.362
0.384
0.464
0.570
0.475
0.561
0,268
0.536
302
Rescanbhitoee
0.204
0.385
0.264
0.313
0.143
0:230
0.375
0.440
0.386
0.486
0.457
0.534
0.176
10275
0:488
0.582
Reseanch-intracct
0.362
0.354
0.628
0.544
303
0.375
0.481
0.207
0.356
0.201
0.286
0.417
0,532
0.411
0.529
0,525
0.593
0,211
0.207
0.505
0.634
304
Zenseanch-instraat
0.388
0.360
0.330
0.304
0.225
0.368
R Search instruct GRPO
0
0.500
0,546
0.401
0.179
0.260
0.400
0,517
0.376
0.168
0.:70
0.535
0.134
0,225
0.164
0.601
305
R-Serch-inemee-PP0
0.338
0.274
0.358
0:527
0.464
306
SSRL instruct
0,380
0.489
0,332
0.389
0.153
0.238
0.344
0.466
0.388
0.165
0,358
0.442
0.106
0,184
0.336
0.438
StepSeurch- hase
0.380
0.355
0.450
0.324
D.467
0.573
0.446
0.552
0.561
0.638
0.232
0.325
0.541
307
StepSeurch instruct
0,386
0.502
0,366
0.431
0.226
0.312
0.400
0.534
0.462
0.560
0.485
0.570
0.222
0,327
0.600
0,718
308
07434
05544
0.436
0.513
0.544
0.31T
03334
0.656
07500
5632"
0.66"
0.530
0.366"
00.02
00725
07755
EStarch-nsttmct
0.442
0.576
0.419
0.494
0.341
0.388
0.458
0,612
0.507
0.642
0,550
0.684
0.254
0.398
0.687
0,823
309
310
Table 1: The main results. "f" indicates offline retrieval, and "*" indicates online retrieval.
311
312
313
314
Baseline Method We employ various baselines to evaluate our proposed ESearch, including Search-
315
RI (Jin et al. 20256). Research (Chen et al. 2025), ZeroSearch (Sun et al. 2025), R-Search (Zhao)
316
et al. 2025a). SSRL (an et al. 2025). StepSearch (Zheng al. 2025a).
317
318
319
Implementation Details We conduct experiments using two model scales: Qwen2.5-3B-
320
base/instruct and Qwen2.5-7B --..-------aasnssrrcecee (Yang et al.) 2024). During training, we adopt E5 (Wang et al. (2022) as the retriever, with the document corpus built from the Wikipedia 2018 dump (Wiki-18) 18) KKrpukhhn et al. 2020). For offline evaluation, we maintain the same Wikipedia dump as the retrieval corpus to ensure consistency with the training setup. For online evaluation, we employ
321
322
323
the Google Search API the retrieval source.

---

Published as a conference paper at ICLR 2026
324
HotpoiQA
2Wiki!
MuSiQue'
Bambongle!
HotpotQA"
2Wiki
MuSiQue
Bamboogle"
Method
325
EMT
F71
EMT
FI1
EM1
F11
EM1
F11
EM1
F11
EM1
F11
EM1
F11
EM1
FJ1
326
Qwea2.5-7b-Ban
0.564
0.511
0.371
0.510
0.730
0.372
327
PPO
0.371
0.475
0.279
0.326
0.196
0.278
0.428
0.545
0.382
0.422
0.475
0.547
0.198
0.277
0.475
0.603
328
GRPO
0.350
0.462
0.267
0.344
0,203
0.292
0.398
0.514
0.401
0.497
0.499
0.568
0.208
0.292
0.489
0.623
Qwen2.5-30-BBa
329
ERL
0.415
0.548
0.499
0.236
0.345
0.414
0.435
0.586
0.581
0.684
0.367
0.633
0.797
330
PPO
0.264
0.372
0.265
0.322
0.106
0. 0.192
0.206
0.313
0.242
0.326
0.322
0,380
0.137
0.204
0.352
0.443
GRPO
0.258
0.367
0,254
0.321
0.113
0.188
0.225
0.312
0.237
0.319
0.326
0.382
0.134
0.199
0.345
0.437
331
332
Table 2: Accuracy performance of models trained by different RL algorithms." "t" indicates offline retrieval using the wiki-18 knowledge base, and "*" indicates online retrieval using Google Search.
333
334
335
1.0
0.25
336
337
338
0.35
0.32
339
340
341
342
0.93
0.00
343
tiining Strps
Baining Prape
Thain eq Steps
344
Figure 2: Training dynamics of different RL strategies.
345
346
347
4.2
MAIN RESULTS
348
349
Offline evaluation Table shows that ESearch sets a new SOTA on four multi-hop QA datasets,
350
consistently surpassing strong baselines with Qwen2.5; with three billion parameters, it gains +6.06%
351
EM and +9.94% F1 on average, rising to +4.78% EM and +6.56% F1 with seven billion parameters.
352
353
Online evaluation We evaluate models using continuously updated search engine instead of a static knowledge base. Online retrieval improves nearly all methods by providing fresher, more comprehensive information. ESearch consistently surpasses baselines, achieving +10.90% EM and +13.18% Fl for Qwen2.5-3B, and +5.98% EM and +7.88% F1 for Qwen2.5-7B. Across scales, ESearch delivers the largest relative gains, demonstrating superior adaptability and robustness in dynamic environments.
354
355
356
357
358
359
360
Comparison with classical reinforcement learning algorithms To further assess the effectiveness of ERL, we conduct a direct comparison with two classical reinforcement learning algorithms: PPO and GRPO, both of which rely solely on task-level success as the reward signal. Experimental results, summarized in Table and Figure demonstrate that ERL consistently and substantially outperforms PPO and GRPO on both the 3B and 7B models. Compared to PPO, GRPO demonstrates faster learning speed and reward acquisition during training but tends to suffer from instability and potential collapse in the later stages of Search Agent tasks. In contrast, ERL achieves higher learning efficiency than PPO while maintaining training stability.
361
362
363
364
365
366
367
368
369
5
ANALYSIS
370
371
To quantify the relative contributions of each component the ERL framework, we conducted a systematic ablation study. Table presen]s the performance of different component combinations. The full ERL framework achieves the best performance across all datasets, confirming our design principle that the three erasure mechanisms are complementary Plan-triggered erasure proves essential on highly structured datasets, with disabling it leading to a -2.05% Fl on 2Wiki, yet remains ineffective in addressing missing retrieval. Search-triggered erasure shows clear advantages for retrieval- inteesivv tasks, where disabling it results in a -2.40% Fl on Bamboogle, but it fails to remedy global reasoning errors. Sub-aenswer-riiggereed erasure benefits reasoning-intente tasks,
372
373
374
375
376
377
7

---

Published as a conference paper at ICLR 2026
378
379
0.313
380
25-
0.20
S4l Jaooee Stane
381
24
0.223
382
383
0373
22-
384
1005
0.033
385
386
350
355
300
250
100
3.0
387
0.200
388
09.77
0.00
389
390
1.4
0.225
391
0.855
0.30
392
3.2
393
005
0.835
394
395
302
155
206
100
35.5
200
404
31.0
250 000
396
Figure 3: Training dynamics in ablation experiments.
397
398
399
Method
HotporQA
2WIki
MuSiQue
Bamboogle' EMI
HetpotQA
2W1k!"
MuSIQue
Bamboogle
EMI
Fit
EM1
FI1
EM1
F11
F31
EM"
FF1
EMF
F1T
EM1
FI"
EM1
FI1
400
Swwen25-76-Biine Buse
FRI.
0.434
0.564
1,436
0,513
0.244
0.371
0.534
0.656
0.510
1.632
0.635
11730
0.265
0.372
0.622
0.799
401
plan
0.430
0.545
0,421
0.496
0.359
0.517
0.634
0.611
0.620
0.706
0.257
0.361
0,602
0.773
402
wioe
0.533
0,412
0.351
0.505
0.607
0.755
whee auñ cmaaser
0.392
0.509
0.393
0.463
0.220
0.335
0.487
0.593
0.461
0.570
0.579
0.659
0.241
0.336
0,563
0.731
403
QwenZs- 36--Bass
Bose
0.322
0.418
0.325
0.380
0.181
0.275
0.3%6
0.486
0.378
0.468
0.476
0.541
0.195
0.273
0.461
0.992
404
afw plam
0.348
0,451
0.349
0.410
0.195
0.297
0.428
0.525
0.408
0.506
0.514
0.584
0.215
0.301
0.498
0.639
014688
308
0.443
0.524
0.532
0.606
0.663
405
smsster
0.378
0.463
0.557
0.633
0.693
406
Table 3: Accuracy on 7b and 3b models. 'w/o' represent "without" while "o/w" for 'only with', """" indicates offline retrieval, and "*" indicates online retrieval.
407
408
409
410
with disabling it yielding -1 -1.80% F1 on Musique. Although it alleviates error propagation, it cannot fundamentally prevent erroncous reasoning from emerging.
411
412
413
Table and Figure present the performance of individual erasure mechanisms across different iteration numbers on the Qwen2.5533bIInstee with increasing iterations, indicating that planning can reduce localized structural mistakes but is insufficient for errors in longer reasoning chains. Notably, even with an imperfect initial plan, the model can still identify the next required information through further interaction with the external environment. Search-triggered erasure yields more pronounced improvements, especially on retrieval- intensive datasets, highlighting the importance of accurate search queries for maintaining reasoning
model. Plan-triggered erasure shows modest gains
414
415
416
417
418
419
420
421
Method
HotpotQA
2Wiki!
MuSiQue'
Bamboogle'
HotpotQA"
2Wiki"
MuSiQuee
Bamboogle
422
AM
F7
AM
Fi
EM
F1
EM
F1
EM
F1
EM
Fi
E.M
Fi
E.M
Fi
0.348
0.457
0.389
0.181
0.364
0.457
0.39%
0.476
0.501
0.151
01525
0.633
423
plaol
0.464
0.183
0:405
0.153
0.542
www
plan3
0.358
0.470
0.333
0.401
0.256
0.272
0.358
0.473
0.413
0.491
0.417
0.516
0.156
0.250
0.541
0.652
424
alw
plans
0.365
0.477
0.338
0.407
0.189
0.276
0.363
0.478
0.417
0.498
0.425
0.524
0.158
0.253
0.549
0.611
425
w/w
a/W scarobl
searel3
0.383
0,3633
0.478
0.502
(L337
0.355
0.427
0.408
(L188
0.198
0.275
0.390
0.361
0.382
0.478
0.502
0:421
0.438
0,499
0.523
0.448
10423
0.550
0.522
0.190
I.156
0.266
0.252
0.576
0.377
0.688
0735
426
0.44
0.451
0.301
0.462
0.581
0.174
0611
0.732
wWW
0.371
0.4%5
0.14
0.13
0.486
0:410
0.507
0.533
0.162
03533
427
www
eut 3
0.399
0.52)
0.371
0.445
0.207
0.302
0.398
0.523
0.456
0.545
0.465
0.573
0.175
0.277
0.597
0.724
alw
sub as
0.455
0.561
0.306
0.479
0.221
0.324
0.426
0.561
0.492
0.584
0.502
0.615
0.186
0.207
0.641
0.776
428
ERI.
0.447
RL587
0.415
01500
0.3
0.339
0.446
0.587
0.513
0.612
0.521
0.644
121
0,11 0.674
0313
429
Table 4: Qwen2.5 3- Instruct. 'o/w" for 'nnyy with', 'sub-answer' represents process supervision
430
rewards based on intermediate sub-answers." "4" indicates offline retrieval using the Wiki-18 18 knowledge base, and "*" indicates online retrieval using Google Search.
431
8

---

Published as a conference paper at ICLR 2026
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
Figure 4: Training dynamics of individual crasure mechanisms across different iteration numbers, In the figure, "a/w" (only with) indicates that only the current mechanism added to the base method.
450
451
452
453
fidelity. Sub-answer-triggered erasure is the most effective, providing consistent gains that approach the full ERL framework's 's performance as iterations increase, demonstrating that revising intermediate sub-ansue sub-answers significantly mitigates error propagation. Overall, the mechanisms follow clear hierarchy: sub-answer erasure > search > plan, emphasizing that error correction during reasoning has greater impact than error prevention. Regarding correction rates, ERL exhibits varied effectiveness across error types, correcting 2.01 % of decomposition errors, 6.53% of retrieval failures, and 9.6% of reasoning errors.
454
455
456
457
458
459
460
461
SeB-rreee seno twee Dnfteved Aaino
Soith SOnS Broend Ng13
Rays SoeCEN D'Pres 850
462
3.03
0.80
463
464
0.2
465
466
467
468
022
6:1
469
470
30
120
471
(a) Model performance over train- ing steps. The thresholds a to trig- ger the Sub-Answer erasure are com- pared between (0.1, 0.3)
(b) Model performance train- ing steps. The thresholds a to trig- ger the Search erasure are compared between (0.05, 0.10, 0.15, 0.20}
(c) Model performance over train- ing steps. The thresholds B to trig- ger the Plan erasure are compared between {0.1, 0.2, 0.3)
472
473
474
475
Figure Model performance over training steps under different erasure trigger thresholds.
476
477
478
To evaluate the stability of ERL, we investigated the sensitivity of three core erasure mechanisms, namely search, planning, and sub-answer modules, to their respective trigger thresholds. The threshold, denoted as or B, determines the aggressiveness of the erasure process. A higher threshold requires the module to maintain a higher level of confidence in the retained information, which consequently increases the frequency of erasure. We isolated the impact of each threshold by modifying the values of the target module while disabling the other erasure mechanisms. The resulting variations in Answer Fl scores during training are illustrated in Figure 5 Our analysis indicates that sensitivity is not uniform across different modules, reflecting their distinct roles within the cognitive chain.
479
480
481
482
483
484
485

---

Published as a conference paper at ICLR 2026
486
The search module exhibits the highest sensitivity. Lower thresholds, such as a E 0.05, 0.10, facilitate rapid convergence and lead to optimal performance. However, when the threshold is increased to 0.2, a sharp decline in performance is observed. This phenomenon suggests that the search module functions as an information bottleneck for the entire system. An overly aggressive deletion strategy during the retrieval phase leads to information starvation, in which the reasoning engine is deprived of the raw contextual information required to generate accurate sub-answers.
487
488
489
490
491
492 In contrast, the planning module demonstrates strong robustness. Varying 3 within the range of 0.1, 0.2, 0.3 yields nearly identical learning trajectories. This indicates that downstream search and reasoning processes possess sufficient flexibility to compensate for imperfections in high-level planning, thereby ensuring stable end-to-end performance.
493
494
495
496
The sub-answer-basse erasure mechanism shows moderate and regular sensitivity. As the threshold increases within 0.1, 0.2, 0.3, the model maintains a stable convergence trend, although the final Answer F1 scores exhibit a stepwise decrease. This reflects the role of sub-answers as intermediate reasoning nodes, where excessive erasure slightly impedes the effective accumulation of logical information.
497
498
499
500
501
502
RELATED WORK
503
504
Reinforcement learning has been widely applied to enhance retrieval-augmentt reasoning in large
505
language models. Search-R1 (Jin (2025b) and ReSearch (Chen et al., 2025) optimize multi-round query generation, R R1-Searcher (Song al. 2025) adopts a two-stage reward, and StepSearch (Zheng et al. 2025a)shapes trajectories with stepwise rewards, while DeepResearcher (Zheng et al., 2025b)
506
507
508 O² Searcher (Mei et al. 2025). and ZeroSearch (Sun al., 2025) target real webpages, localized environments, and retrieval simulation. MaskSearch (Wu et al.) 2025 hand EvolveSearch (Zhang et al. 2025a) improve multi-hop reasoning through pretraining or iterative self-evolution, and R- --eearch
509
510
511
(Zhao et al. (2025a) and DynaSearcher (Hao et al. (2025) integrate multi-reward signals with dynamic knowledge graphs. ParallelSearch (Zhao et al. 2025b). HybridDeeepSearhher (Ko et al., (025). and
512
SSRL (Fan et al., (2055) further advance retrieval via parallelization, adaptive strategies, or internal knowledge search.
513
514
515
516
7
LIMITATION & FUTURE DISCUSSION
517
518
The strength of the ERL framework lies in its structured cycle of identification, erasure, and regen- eration, which enables targeted correction of reasoning errors and significantly improves reliability, This sequential design inherently increases computational overhead and may struggle when mul- tiple heterogeneous errors occur simultaneously within a reasoning trajectory. In such cases, the framework often requires repeated iterations to separately repair failures retrieval, reasoning, and subsequent retrieval stages, which limits scalability and efficiency. Addressing this challenge calls for strategies that can recognize and resolve multiple concurrent errors a single corrective pass. Such an advance would require moving beyond localized error signals toward a global understanding of the entire reasoning trajectory, enabling coordinated error mitigation rather than piecemeal correction. Developing this global perspective is not only crucial for enhancing the robustness and efficiency of ERI., but also represents a broader step toward equipping search-augmented language models with genuinely resilient reasoning capabilities.
519
520
521
522
523
524
525
526
527
528
529
530
8
CONCLUSION
531
532
This paper introduces erasable reinforcement learning algorithm designed to automatically detect and correct decomposition, retrieval, and reasoning errors in complex multi-hop question answering. The method leverages joint signals from the quality of sub-search and sub-answer processes to identify error types, and erases the corresponding segments for regeneration when errors occur, thereby maximizing the utility of both the model and external knowledge. Experimental results demonstrate that our approach surpasses the current state of the art on multi-hop QA benchmarks including HotpotQA, MuSiQue, 2WikiMultHHppAA, and Bamboogle, validating its effectiveness. Future work may explore extending this mechanism to a broader range of generative tasks, or integrating it with online learning to further enhance the model's 's adaptive error-correction capability.
533
534
535
536
537
538
539
10

---

Published as a conference paper at ICLR 2026
540
ETHICS STATEMENT
541
542
This study uses only publicly available benchmark datasets (HotpotQA, MuSiQue, 2WikiMulti- HopQA, and Bamboogle), with knowledge sources limited Wikipedia and the Google Search API. No private or sensitive data are involved. The proposed Erasable Reinforcement Learning (ERL) substantially enhances multi-hop reasoning capabilities, offering positive value for applications such as information retrieval and educational question answering. However, we also recognize that stronger reasoning ability could be misused to generate deceptive or misleading content. We recommend that future research integrate alignment and bias detection mechanisms prior to deployment to mitigate such risks. Overall, this work adheres to established academic ethical standards, balancing capability advancement with responsible use, and aims to contribute to the development of trustworthy artificial intelligence.
543
544
545
546
547
548
549
550
551
552
553
REPRODUCIBILITY
554
555
We have taken extensive measures to ensure the reproducibility of our work. All datasets used in this study are publicly available benchmarks, including HotpotQA, MuSiQue, 2WikiMuliHHopAA, and Bamboogle. For retrieval, we employ both a fixed Wikipedia dump and the Google Search API, and describe the retrieval setup in detail to enable consistent replication. Implementation details, such as model architectures (Qwen2.5-3B) BBB/BB base and instruct), retriever backbone (E5), training hyperparameters, and evaluation metries (Exact Match and F1), are fully documented. To further facilitate reproducibility, we will release the training scripts, evaluation pipelines, and configuration files required to replicate all results reported in this paper. Random seeds and hardware specifications will also be provided to minimize variance across runs, Our methodology does not rely on proprietary or undisclosed components, ensuring that independent researchers can fully verify and extend our findings.
556
557
558
559
560
561
562
563
564
565
566
LLM USAGE
567
568
We partially used large language models (LLMs) exclusively for non-scientific writing assistance, specifically for language polishing, clarity improvement, and suggestions. No parts of the core methodology, experiments, or results were generated by LLMs.
569
570
571
572
REFERENCES
573
574
Perplexity AI. Introducing perplexity deep research, 2025.
575
ai ai/hub/blog/ hatgg/iitroouuuc httpdd/ddnn....pp..ee.eeeeeeeeee
researci Accessed: 2025-09-25.
576
Joost Broekens, Koen Hindriks, and Pascal Wiggers. Reinforcement learning as heuristic for action- rule preferences. In International Workshop on Programming Multi-Agent Systems, pp. 25-40. Springer, 2010.
577
578
579
580
Mingyang Chen, Linzhuang Sun, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, et al. Research: Learning to reason with search for Ilms via reinforcement learning. arXiv preprint arXiv:2503.1941) 2025.
581
582
583
Zhixun Chen, Yali Du, and David Mguni. All language models large and small. arXiv preprint arXiv:2402.12061. 2024.
584
585
586
Google DeepMind. Gemini deep research, 2025. URL htt/atttpp///gaaa..ooo/////////r respachsh Accessed: 2025-9-25.
587
588
Yuchen Fan, Kaiyan Zhang, Heng Zhou, Yuxin Zuo, Yanxu Chen, Yu Fu, Xinwei Long, Xuekai Zhu, Che Jiang, Yuchen Zhang, et al. Ssrl: Self-search reinforcement learning. arXiv preprint arXiv:2508.108: 2025.
589
590
591
592 Jiaxuan Gao, Wei Fu, Minyang Xie, Shusheng Xu, Chuyi He, Zhiyu Mei, Banghua Zhu, and Yi Wu. Beyond ten turns: Unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint xxXvv2550.007976 2025.
593
11

---

Published as a conference paper at ICLR 2026
594
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
595
596
597
598
Chuzhan Hao, Wenfeng Feng, Yuewei Zhang, and Hao Wang. Dynasearcher: Dynamic knowl- edge graph augmented search agent via multi-reward reinforcement learning. arXiv preprint arXiv:2507. 17365, 2025.
599
600
601
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprchensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 5609-6625, 2020.
602
603
604
605 Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025a.
606
607
608
609
Tzu-Hao Huang. Reinforcement learning framework for window hardware installation. 2022.
610
611
Yue Huang, Lichao Sun, Haoran Wang. Siyuan Wu, Qihui Zhang, Yuan Li, Chujje Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561 2024.
612
613
614
615
Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Huichi Zhou, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, et Deep research agents: systematic examination and roadmap. arXiv preprint arXiv:2506.1888 18096, 2025b.
616
617
618
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, al. Openai ol system card. arXiv preprint arXiv:2412.16770,, 2024.
619
620
621
622
Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. An empirical study reinforcement learning for reasoning-search search interleaved I1m agents. arXiv preprint arXiv:2505. 2025a.
623
624
625
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Serean Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training iims to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09556, 2025b.
626
627
628
629
Leslie Pack Kaelbling, Michael L. Littman, and Andrew W Moore. Reinforcement learning: A survey, Journal of artificial intelligence research, 4:237-285, 1996.
630
631
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 6769-6781, 2020.
632
633
634
635
636
Dayoon Ko, Jihyuk Kim, Haeju Park, Sohyeon Kim, Dahyun Lee, Yongrae Jo, Gunhee Kim, Moontae Lee, and Kyungjae Lee. Hybrid deep searcher: Integrating parallel and sequential search reasoning. arXiv preprint arXiv:2508. 19113, 2025.
637
638
639
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis. Wen-tau Yih, Tim Rocktäschel, et al. eetrieval-augmentt genera- tion for knowledge-intensinn nlp tasks. Advances in neural information processing systems, 33: 9459-9474, 2020.
640
641
642
643
644
Wenjun Li, Zhi Chen, Jingru Lin, Hannan Cao, Wei Han, Sheng Liang, Zhi Zhang, Kuicai Dong, Dexun Li, Chen Zhang, et al. Reinforcement learning foundations for deep research systems: A survey. arv preprint arXiv:2509.06733, 2025.
645
646
647
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arxiv 1701 007274,, 2017.
12

---

Published as a conference paper at ICLR 2026
648
Jianbiao Mei, Tao Hu, Daocheng Fu, Licheng Wen, Xuemeng Yang, Rong Wu, Pinlong Cai, Xinyu Cai, Xing Gao, Yu Yang, et al. Oe-searcher: A searching- based agent model for open-domain open-ended question answering. arXiv preprint 2tii:2550511688,,, 2025.
649
650
651
652
OpenAL.
Introducing deep research, 2025. URL httts:://ppeeei com/ index Accessed: 2025-9-25.
653
-elrooducingg-de----eeee
654
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, 2023.
655
656
657
658
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint 150.00430602338. 2015.
659
660
661
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
662
optimization algorithms. arXiv preprint
663
664
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibín Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. In Proceedings of the Twentieth European Conference on Computer Systems, pp. 1279-1297, 2025.
665
666
667
668
Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card. arXiv preprint arXiv:2601 1032677, 2025.
669
670
671
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI -sesercherr: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05552, 2025.
672
673
674
675
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of Ilms without searching. arXiv preprint (xXv:25505004885 2025.
676
677
678
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
679
680
681
682
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguisties, -554,, 2022.
683
684
685
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03531, 2022.
686
687
688
689
Weiqi Wu, Xin Guan, Shen Huang, Yong Jiang, Pengjun Xie, Fei Huang, Jiuxin Cao, Hai Zhao, and Jingren Zhou. Masksearch: A universal pre-training framework to enhance agentic search capability. arxiv preprint 2rXiv:2505 2025.
690
691
692
Yunjia Xi, Jianghao Lin, Yongzhao Xiao, Zheli Zhou, Rong Shan, Te Gao, Jiachen Zhu, Weiwen Liu, Yong Yu, and Weinan Zhang. A survey of lim-based deep search agents: Paradigm, optimization, evaluation, and challenges. arXiv preprint arXiv:2508.0566 006668,666866 2025.
693
694
695
696 An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv e-prints, pp. arXiv-2412, 2024.
697
698
699
700
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.0933 2025.
701
13

---

Published as a conference paper at ICLR 2026
702
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference empirical methods in natural language processing, pp. 2369,,380, 2018.
703
704
705
706
Ding-Chu Zhang, Yida Zhao, Jialong Wu, Liwen Zhang, Baixuan Li, Wenbiao Yin, Yong Jiang, Yu-Feng Li, Kewei Tu, Pengjun Xie, et al. Evolvesearch: iterative self-evolving search agent. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. pp. 13134-13147, 2025a.
707
708
709
710
711
Yuxiang Zhang, Yuqi Yang. Jiangming Shu, Xinyan Wen, and Jitao Sang. Agent models: Internalizing chain-of-action generation into reasoning models. arXiv preprint arXiv:2503.06580 2025b.
712
713
Qingfei Zhao, Ruobing Wang, Dingling Xu, Daren Zha, and Limin Liu. R-search: Empowering Ilm reasoning with search via multi-reward reinforcement learning. arXiv preprint arXiv:2506.04185 2025a.
714
715
716
717
Shu Zhao, Tan Yu, Anbang Xu, Japinder Singh, Aaditya Shukla, and Rama Akkiraju. Parallelsearch: Train your Ilms to decompose query and search sub-queries in parallel with reinforcement learning. arXiv preprint arXiv:2508 093003, 2025b.
718
719
720
Xuhui Zheng, Kang An, Ziliang Wang, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting Ilms search ability via step-wise proximal policy optimization. In Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, pp. 21816-21 21841, 2025a.
721
722
723
724
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojic Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 414 431, 2025b.
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
748
749
750
751
752
753
754
755
14

---

Published as a conference paper at ICLR 2026
756
A COST-EFFECTIVENESS OF INCREASING THE ERASURE-RETE BUDGET
757
758
759
Timais) per step
ComeectF1) by Stop
CerrecFF-) Accumiitted Timet))
760
761
64
762
763
764
356-
765
766
cinterown
767
100
768
100
13c 200 200
300
3 30
200
200
230
302
20000 63030 83303
R0000
309000
769
770
Figure 6: Training efficiency under different Sub-Answer retry budgets. (a) Per-step wall-clock time across PPO, GRPO and Sub-Answer Erasure under different retry budgets. (b) Model performance over training steps. (c) Model performance over cumulative training time.
771
772
773
774
775
776
Time(N) per Step
Conmett by Stap
Fonnect 13 Arrumuleted Timejs
777
040
04
0.2-
778
779
780
400
000
781
782
FEP
783
784
300
010
Lo00c13 3
100
120
203
200
300
30
323
120
200
200
300
20000
46008
63003
00000
100000
785
Step
[a]
[c]
786
787
Figure 7: Training efficiency under different Search retry budgets. (a) Per-step wall-clock time across PPO, GRPO and Search Erasure under different retry budgets. (b) Model performance over training steps. (c) Model performance over cumulative training time.
788
789
790
791
792
Tinels) per stop
ComeenFFl) by Stop
ConrectF:) by Accumulated Timets)
793
344
794
polix
0.19
795
1:0
colan
0.50
796
797
350
798
6:15
799
800
100
801
802
803
Figure 8: Training efficiency under different Plan retry budgets. (a) Per-step wall-clock time across PPO. GRPO and Plan Erasure under different retry budgets. (b) Model performance over training steps. (c) Model performance over cumulative training time.
804
805
806
807
808
To address the concern regarding the computational overhead introduced by longer rollout horizons and more frequent research queries, we conduct an additional experiment that systematically varies the maximum number of allowed erasure retries for each module (subanswer, plan, and search). Since
809
15

---

Published as a conference paper at ICLR 2026
810
the erasure mechanism is implemented sequentially, the wall-clock time measured under identical hardware directly reflects both GPU computation and external querying volume. This provides an interpretable and implementation-faithre estimate of the cost of expanding the search-and-res and-repair SuuAAsse---Frasee Plan-Frase, and Search- we train the model while capping the erasure-retry budget at (1,3,5}.
811
812
813
budget. For each erasure
814
815
(21)
816
817
All other training hyper-paramelers, random seeds, dataset order, and hardware remain strictly identical across runs. For each setting we record:
818
819
Per-step wall-clock time (s/step) - directly reflecting computation + querying cost.
820
Accuracy-per-a curve training progress with respect to optimization steps.
821
822
Accuracy-per-eeeee perttcc curve training progress normalized by cumulative runtime, measuring practical training efficiency.
823
824
This design allows us to isolate the marginal benefit of higher erasure budgets while quantifying their real-world cost. The experimental results show that applying erasure different stages produces significantly different effects, primarily reflecting the importance of each stage itself in chain-of- thought reasoning. Applying the erasure mechanism to stages such as "Sub-Answer, which has the greatest impact the final answer, leads to the most substantial and noticcable improvement in training efficiency. Although the computation time per step becomes longer due to the need for rethinking and responding, the rate of improvement in training metrics per unit of computation time actually increases compared to the original PPO.
825
826
827
828
829
830
831
832
Another noteworthy point is that, although GRPO is far more efficient per unit time than the original PPO and PPO optimization algorithms enhanced with the erasure mechanism, its performance ceiling clearly lower.
833
834
835
836
B
DYNAMICS OF ERASURE EVENTS ACROSS MULTI-ROUND REASONING
837
838
To further understand how the ERL framework behaves during training, conduct additional experiment that analyzes when and how often crasure events occur within a multi-hop reasoning trajectory. Specifically, for each training step. record the average number of retries triggered by the Sub-Answer, Search and Plan modules under different maximum reasoning depths (i.e., number of rounds). We evaluate four settings with round limits
839
840
841
842
843
E (1,2,3,4),
(22)
844
845
corresponding to increasingly deeper multi-step decomposition strategies. We find that the average frequency of erasing subanswers becomes more pronounced as reasoning deepens. This indicates
846
847
848
849
Aob-Annwwrrroo Retried Proquency Step
Search&Paan Retried Froquency by
850
851
25
2.5-
852
2.0
853
854
3.5
1.5
855
856
857
round33
mund-4
0.5
Surc-4
Runc-3
858
mind-2
round 1
859
860
861
(a) Average frequency Aub-Answer Erasures occur- rence per round with max retry set by
(b) Average frequency Search&Plan Erasures occur- rence per round with max retry set by 3.
862
863
Figure 9: Dynamics of erasure events across multi-hop reasoning rounds during training.
16

---

Published as a conference paper at ICLR 2026
864
that the later stages of the reasoning chain are more likely to produce incorrect subanswers and thus require erasure and retry. Difficult and multistep problems further demonstrate that the accumulation of carlier errors makes subsequent reasoning more challenging. The increased number of retries triggered by such cases fully illustrates the value of the ERI mechanism. A similar phenomenon is observed in the rewriting step during rounds 2, 3, and 4 of the Search process. In contrast, for Plan which specifically includes an erasure mechanism only in the first round, the erasure phenomenon tends to remain stable.
865
866
867
868
869
870
871
872
C
MAIN RESULT WITH CONFIDENCE INTERVALS
873
874
875
Mecteel
Hegilge
Natiger
Hepegg
Vodge
876
650
877
amprrs
878
07022 m
100,300
879
000/0000
135-004
Miliccs
656-14
6512-481
118-430
638-483
673-444
880
$18,310
$28,59
660.387
881
Table 5: The main results. indicates offline retrieval, and "*" "indicctes online retrieval.
882
883
884
We have incorporated statistical analysis of confidence intervals into the main experimental results. Table 5 presents the experimental results with confidence intervals, helping to confirm that the performance gains we report are significant and go beyond random noise.
885
886
887
888
D RELATED WORK
889
890
Recent research has increasingly explored reinforcement learning (RL) as means to improve the
891
retrieval and reasoning capabilities of large language models (LLMs) et al., 20256 Sun et al.
892
2025; Zheng et al. 2025a Zhao et al. 2025a Chen al. 2025 Zhao et al. 2025b Wu al. 2025 Hao et al. 2025 Zhang et al. 2025a Zheng et al. 2025b Song et al. 2025 Ko et al. 2025). A key
893
894
theme in this literature is the integration of retrieval into multi-step reasoning, often referred as search-reinforcemenn learning. We summarize related work along three major dimensions: coupling
895
896
retrieval with reasoning. reward design for retrieval optimization, and dynamic structured retrieval
897
strategies.
898
899
Retrieval-Reasoning Coupling Several approaches train LLMs to seamlessly integrate retrieval into reasoning trajectories. Search-R1 (Jin al. 2025b ab applies RL to enable models to au- tonomously issue queries during multi-step reasoning, with iterative retrieval interactions guiding trajectory refinement. R1-Searcher (Song et al.) 2025) introduces a two-stage training paradigm: a retrieve reward first encourages correct execution of retrieval operations independent of final answers, after which an answer reward incentivizes effective use of retrieved evidence to solve problems. ReSearch (Chen et al. (025) explicitly regards search as part of the reasoning chain, training LLMs to perform retrieval whenever necessary and incorporate results into subsequent steps. DeepResearcher (Zheng et al. (025b) pushes this line further by performing end-to-end training on real webpages, showcasing advanced behaviors such as planning, cross-source verification, and self-reflection.
900
901
902
903
904
905
906
907
908
909
910
Reward Design and Training Paradigms Another line of work develops specialized environments and reward functions to guide retrieval. O2-Searcher constructs a localized search environment with carefully designed rewards to address both open-domain and closed-domain tasks. ZeroSearch (Sun) cet al., 2025) reduces training costs by simulating retrieval while maintaining comparable effectiveness to real search engines. StepSearch (Zheng et al. 2025a) introduces fine-grained step-level rewards, such as information gain and redundancy penalties, within PPO (Schulman et al.) 2017) training to progressively refine search behaviors. MeskSearch (Wu et al. (225)) augments pretraining with retrieval-baa -aaee masked prediction tasks, teaching models to leverage search tools to fill textual gaps and thereby improving multi-hop QA. EvolveSearch (Zhang et al.) 2025a) integrates supervised
911
912
913
914
915
916
917
17

---

Published as a conference paper at ICLR 2026
918
fine-tuning (SFT) with RL in an iterative self-evolution framework, continually improving multi-hop retrieval without requiring annotated reasoning data.
919
920
921
Dynamic and Structured Retrieval Strategies Recent studies emphasize adaptive control over retrieval behaviors and the exploitation of structured query patterns. R-Search (Zhao et al. (025a) employs multi-reward RL to dynamically decide when to retrieve versus when to reason, while integrating multi-turn results to enhance answers for knowledge- and logic-intensive tasks. Dy-
922
923
924
925
naSearcher (Hao al., 2025) leverages dynamic knowledge graphs and multi-reward RL to maintain consistency in retrieval and improve output quality. ParallelSearch (Zhao et al. (2025b) identifies decomposable query structures and executes multiple subqueries in parallel. HybridDeepSeercheer (Ko) et al. 2025) combines parallel and sequential retrieval modes, selecting the most suitable strategy based on problem characteristics. Finally, SSRL (Fan et al. 2025) explores retrieval grounded in a model's 's internal knowledge base, thereby reducing dependence on external search engines.
926
927
928
929
930
931
932
E DATASETS
933
934
We selected four benchmark datasets designed based on multi-hop questions: HotpotQA Yang et al.,
935 2018). 2WikiMuliHopQAA (Ho et al. 2020), Musique(TTivedd et al. 2022). and Bamboogle (Press et al. 2023).
936
937
HotpotQA Yang et al., 2018): HotpotQA was introduced to address the limitations of earlier QA datasets, which mostly focused on single-paragraph reasoning and lacked explicit supervision for multi-hop reasoning. HotpotQA aimed to build a large-scale dataset requiring reasoning across multiple documents, while also supporting explainable predictions. To achieve this, they crowd- sourced over 112k question- guestion-answer pairs based on Wikipedia, ensuring that questions required integrating information from more than one article. A key innovation was the collection of supporting facts sentence-eeeeeee evidence for answers wwwwers-lloownnggg models not only find the correct response but also to explain it. Additionally, HotpotQA includes a novel class of comparison questions, which require systems to compare two entities on shared properties such as dates or numerical values. The dataset was split into train-easy (18,089), train-medium (56,814), train-hard (15,661), dev (7,405), and two test sets (7,405 each: distractor and fullwiki).
938
939
940
941
942
943
944
945
946
947
948
et al. 2020p: The 2Wiki dataset is a large-scale multi nulti-hop question answering benchmark created from Wikipedia and Wikidata. It aims to evaluate reasoning by requiring models to integrate information across multiple documents. Unlike earlier datasets, it provides explicit evidence paths in the form of triples, which both enhance interpretability and allow direct evaluation of reasoning skills. The construction process involved designing templates, applying logical rules, and filtering to guarantee multi-hop reasoning. Four question types are included: comparison, inference, compositional, and bridge-comparison, ensuring diversity and difficulty. In total, the dataset contains 192,606 examples, split into 167, 454 for training. 12,576 for development, and 12,576 for testing. This scale makes it significantly larger than many prior multi-hop QA datasets. Human performance remains much higher than model baselines, showing the dataset's 's value as a challenging benchmark for machine reasoning.
WWliMMuliHHoQQ((2iikii)
949
950
951
952
953
954
955
956
957
958
Musique Trivedi et al., 2022): The MuSiQue dataset was created to address the limitations of existing multi-hop question answering benchmarks. Musique proposed a bottom-up construction method: they carefully composed multi-hop questions from single-hop questions sourced from several Wikipedia-bate based datasets. The dataset consists of two main variants: MuSiQue-Ans, containing about 25,0002 2-4 hop questions, and MuSiQue-Full, which doubles this size by adding contrastive unanswerable questions, resulting in 50,000 samples. Specifically, MuSiQue-Ans is split into 19,938 training, 2,417 development, and 2,459 test questions, with balanced distributions across different hop lengths. These features make MuSiQue a challenging and less "cheatable" benchmark, pushing research toward genuine multi-hop reasoning.
959
960
961
962
963
964
965
966
967
Bamboogle (Press et al., 2023): The Bamboogle dataset was introduced to address the limitations of existing question answering benchmarks, where many compositional questions cannot be answered with a single Google query because the necessary information is dispersed across multiple sources. Unlike prior datasets that often focus single-hop fact retrieval, Bamboogle emphasizes multi-ho hop factual reasoning. It requires models to integrate multiple entities and relations to arrive at the correct answer. In terms of scale, the benchmark contains a test set of 125 questions, which are carefully
968
969
970
971
18

---

Published as a conference paper at ICLR 2026
972
annotated to evaluate the model's ability to identify and use intermediate entities (bridging objects) during reasoning.
973
974
975
976
F EXPERIMENT SETUPS
977
978
Our implementation builds upon Search-RI (Jin et al. 2025b) and StepSearch (Zheng al. 2025a
979
with training performed using Verl (Sheng et al. 2025). We evaluate two model variants, Qwen-2.5-
980
3B and Qwen-2.5-7B (Yang al.] (222)) We use the 2018 Wikipedia Wiki-18) (Karpukhin et al.] 2020) dump and E5 (Wang et al. (022) as the knowledge base and retriever. For training, we utilize the MuSiQue dataset processed through our training, while evaluation is conducted on the full test or validation splits of 2Wiki, Bamboogle, HotpotQA, and MuSiQue. Both EM and F1 are reported as evaluation metries. Training runs for 500 steps total. The learning rates are set to 5 10-7 for the policy model and 5 10 for the value model, with warm-up ratios of 0.285 and 0.015, respectively. Experiments are executed across two nodes equipped with 16 H800 GPUs. We configure the total, mini-batch, and micro-batch sizes as 512, 64, and 16. To improve memory efficiency, we apply Fully Sharded Data Parallel (FSDP) with CPU offloading, fixing the GPU memory utilization ratio at 0.7.
981
982
983
984
985
986
987
988
989
For rollout sampling, we set both the temperature and top_p to 1.0. The KL-divergence regularization coefficient (3) and clipping ratio are set to x 10-3 and 0.2, respectively.
990
991
992
G PROMPT FOR RESEARCH PLAN ON QUESTION ANSWERING
993
994
995
Template for ESEARCH.
996
997
You are an expert AI assistant with search engine access. When answering complex questions, you need to decompose them into sub-questions and reason step by step. For each sub-question: provide concise search terms between <search> and </search>; the search results will be placed between <information> and </informations; conduct thorough analysis and reasoming in <observation> and </observation>; then output concise conclusion in <sub _ns_anrsw and </sub annwer. If you find that all sub- questions have been solved, you should directly provide the final answer inside <answer> and </answer> without detailed illustrations. For example, <answer> xxx </answer>.
998
999
1000
1001
1002
1003
1004
1005
Question:(question)
1006
1007
1008
Figure 10: LLM interacts with external search engines and provides answers to prompt templates.
1009 The (question] will be replaced with the actual question content.
1010
1011
1012
H INCORRECT FORM
1013
1014
Esearch errors can be broadly categorized into three types. First, premature observations occur when the system concludes too quickly in the observation step without fully leveraging the available evidence, as shown Table [3] Second, retrieval errors occur when the system fails to retrieve the correct documents, often due to imprecise or poorly formulated queries, as shown in Table [4 Third,
1015
1016
1017
entity alignment localization errors arise when the correct document retrieved. Still, the model fails to identify and ground the right entity within it, as shown in Table [5] These error types are the main impact of failures in retrieval, entity alignment, and observation, undermining multi-hop question answering. In the actual training process, we observed that observation errors decrease steadily with training steps, while retrieval emror also decline but at a much slower rate compared to observation errors. This further reveals that the training is hindered by the limited capabilities of the locally deployed search engine based on Wiki-18.
1018
1019
1020
1021
1022
1023
1024
1025
19

---

Published as a conference paper at ICLR 2026
1026
1
COMPARE WITH TRADITIONAL METHODS
1027
1028
1029
HotpotQA
2Wiki
MuSiQue
Bamboogle
Method
1030
EM
F1
EM
F1
EM
F1
EM
F1
1031
1032
Direct Inference
0.167
0.214
0.263
0.308
0.014
0.095
0.038
0.099
1033
CoT
0.037
0.099
0.016
0.094
0.009
0.067
0.179
0.234
1034
IRCoT
0.077
0.135
0.137
0.197
0.058
0.141
0.221
0.305
1035
Search-ol
0.204
0.287
0.230
0.293
0.047
0.126
0.336
0.397
1036
RAG
0.285
0.366
0,192
0.271
0.089
0.148
0,303
0.364
1037
SFT
0.197
0.252
0.158
0.243
0.077
0.139
0.100
0.181
1038
R R1-base
0,239
0.294
0,262
0,317
0.070
0.127
0.246
0,303
1039
RI-instruct
0.194
0.279
0.239
0.327
0.085
0.151
0,204
0.297
1040
Esearch-base"
0.415
0.548
0.428
0.499
0.236
0.345
0.414
0.529
1041
Escarch-instruct"
0.447
0.587
0,415
0.500
0.232
0.339
0.446
0.587
1042
1043
Direct Inference
0.201
0.248
0.238
0.319
0.019
0.106
0.107
0.191
1044
CoT
0.079
0.165
0.127
0.184
0.035
0.094
0.214
0.299
1045
IRCoT
0.121
0.206
0.133
0,218
0.055
0.143
0.237
0.296
1046
Search-ol
0.206
0.257
0.189
0.246
0.045
0.132
0.281
0.366
1047
RAG
0.317
0.375
0.221
0.307
0.084
0.144
0.273
0.361
1048
SFT
0.233
0.287
0.277
0.328
0.080
0.138
0.124
0.178
R l-hasee
0.212
0.301
0.229
0.315
0.066
0.157
0.277
0.335
1049
R1-instruct
0.254
0.314
0.304
0.361
0.060
0.146
0,266
0,329
1050
Esearch-base"
0.434
0.564
0.436
0.513
0.244
0.371
0.534
0.656
1051
Esearch-instruct
0.442
0.576
0.419
0.494
0.241
0.358
0.458
0.612
1052
1053
Table 6: Comparison of ESEARCH with traditional non-reinforeement learning methods on four
1054
multi-hop Q&A datasets, reported with Word-level F1 and Exact Match (EM) scores using Wiki-18 as search engine. The best results are highlighted in bold.
1055
1056
1057
1058
J NUMBER OF RETRIEVED K DOCUMENTS
1059
1060
Table shows the effect of varying the number of top-K on the 3B model. A single document (A- = 1) leads to the lowest performance across all datasets, indicating insufficient evidence for multi-hop reasoning. Three documents (A: = 3) yield the most reliable improvements and deliver the strongest overall results. Increasing the retrieval to five (k: = 5) produces different outcomes: in some cases, such Bamboogle, performance is close to & = 3, while in others it is slightly degraded. Therefore, excessive retrieval may introduce irrelevant or distracting information, bring more confusion to the model. When & = 3 indicates the most robust choice, while is 5 can offer marginal gains only in particular datasets.
1061
1062
1063
1064
1065
1066
1067
1068
HotpotQA
2Wiki
MuSiQue
Bamboogle
1069
Top-K
EM
F!
EM
F1
EM
F1
EM
F1
1070
S-ee225-3b-Banl
1071
0.379
0.508
0.415
0.481
0.206
0.332
0.329
0.447
1072
3
0.415
0.548
0.428
0.499
0.236
0.345
0.414
0.529
5
0.389
0,518
0.406
0.491
0.245
0.343
0.424
0,573
1073
1074
0.409
0.546
0.366
0.457
0.192
0.316
0.373
0.496
3
0.447
0,587
0.415
0.500
0.232
0.339
0.446
0.587
1075
0.455
0.603
0.374
0.469
0.208
0.304
0.448
0.603
1076
1077
Table 7: Impact of the Number of Retrieved Documents (Top-k) on Answer Accuracy. (ESEARCH on Qwen2.5-3B with updated results). Bold indicates the best performance.
1078
1079
20

---

Published as a conference paper at ICLR 2026
1080
K
CASE STUDY
1081
1082
1083
Case study for bad planning recovery in 2-hops question during training
1084
Question: Who in the city where Chopin grew up has the power of legislative action?
1085
1086
Ground Truth: ['Warsaw City Council']
1087
Sub-answers: ("Warsaw, Poland', "Warsaw City Council']
1088
1089
ERL:
1090 To answer this question, we need to break it down into sub-questions:
1091
Identify the city where Chopin was born.
1092 2. Determine which organization has legislative power in that city.
1093 Let's address these sub-questions one by one.
1094 <search> Chopin birthplace city </search> <informations Doc <## Title: "Frédéric Chopin" ## Frédéric François Chopin was born on 1 March 1810 in Zelazowa Wola, village in the Duchy of Warsaw ... moved to Warsaw when he was seven months old
1095
1096
1097 Doc 2<## Title: "Chopin family" ##> The Chopin family moved Warsaw in October 1810, when Frédéric was seven months old grew up in Warsaw, attending school there Doe 3*## Title: "Early life of Chopin" " ##> Though born in Zelazowa Wola, Chopin was raised in Warsaw from infancy and considered Warsaw his hometown... </information> <observation> The city where Chopin was born Zelazowa Wola, a village the Duchy of Warsaw.</observation
1098
1099
1100
1101
1102
1103
<sub answer> Warsaw, Poland </sub answer>
1104
Sub-answer incorrect! It will trigger an erasure and regeneration of obervation!
1105
1106
Erasure and Regeneration:
1107
<observation> Although Chopin born in Zelazowa Wola, he grew up in Warsaw, which he considered his hometown. </observation>
1108
<sub answer> Zelazowa Wola </sub_answer>> answer>
1109
1110
Now that we know Chopin grew up in Warsaw, can move on the second sub-question. <search> legislative power Warsaw Poland government </search>
1111
1112
<information> Doc 1<## Title: "Warsaw City Council" ##> Warsaw City Council unicameral governing body of the city of Warsaw, the capital of Poland. It consists of 60 councilors elected in free elections for Yoar term and has legislative powers over local matters.
1113
1114
1115
1116
Doc 2<## Title: "Government of Warsaw" ## Legislative power in Warsaw vested in unicameral Warsaw City Council ("Rada Miasta"), " which comprises 60 members elected for four-year terms
1117
1118
Doe 3*## Title: "Warsaw" ## The city council of Warsaw exercises legislative authority within the city limits, passing local ordinances and budget decisions </information> <observation> The Warsaw City Council has legislative power in Warsaw. </observation> <sub answer> Warsaw City Council </sub_answer>>>>>>
1119
1120
1121
answer>
1122
1123
The organization that has legislative power in the city where Chopin grew up (Warsaw, Poland) <answer> Warsaw City Council </answer>
1124
1125
1126
1127
1128
Figure II: Case study demonstrating error recovery where initial planning focused on birthplace rather than where Chopin grew up, but observations corrected the understanding identify Warsaw as his hometown.
1129
1130
1131
1132
1133
21

---

Published as a conference paper at ICLR 2026
1134
1135
Case study for Esearch in 2-hops question
1136
1137
Question: Who is the spouse of the person who won the popular vote in the 2000 election?
1138
1139
Ground Truth: ('Tipper Gore"]
1140
Sub-answers: ('Al Gore', 'Tipper Gore']
1141
ERL:
1142
To answer this question, we need to follow these steps:
1143
Identify the person who won the electoral vote the 2000 U.S. presidential election.
1144
2. Then find out who that person's 's spouse is.
1145
<search> Winner electoral vote in the 2000 election </search>
1146
<information>
1147
No useful information related are retrieved!
1148
</informatios>
1149 Plan is incorrect! will trigger an erasure and regeneration of plan!
1150
1151 To answer this question, we need to break it down into sub-questions:
Erasure and Regeneration:
1152
1. Identify the person who won the popular vote in the 2000 election.
1153
Determine the spouse of that person. Let's address these sub-questions one by one.
1154
<search> popular vote in the 2000 election (Prussia)? </search>
1155
<information>> Doc 1<## Title: "2000 United States presidential election" ##>The 2000 United States presidential election was the 54th quadrennial presidential election...
1156
1157 Doc 2<## Title: "2000 United States presidential election" ##> slate of electors on December 12. should the dispu...
1158
1159 Doc 3<## Title: "2000 United States elections" ## The 2000 United States elections were held
1160
1161
</information> <observation> The person who won the popular vote in the 2000 election was Al cobe.r/obsrvvation> <sub answer> Al Gore</sub answer>
1162
1163
1164
1165
Now that we know the person who won the popular vote the 2000 election, we can move on to the second sub-question. <search> spouse of Gore</search>>
1166
1167
1168
<information> Doc |<## Title: "Pauline LaFon Gore" ##> Pauline LaFon Gore Pauline LaFon Gore (October 6, 1912 - December 15, 2004) was the mother of former United States Vice President Al Gore and the wife of Doc 2<## Title: "Tipper Gore" ##> Tipper Gore Mary Elizabeth ""Tipper"" Gore (""née"" Aitcheson; born August 19, 1948) is an American social issues advocate who was Second Lady of the United States from 1993 to 2001, and the wife of Al Gore, the 45th Vice Doc 3<## Title: "Tipper Gore" ##> former editor of ""National Geographic"" Tipper Gore is the author of number of books including: She has also contributed to the following books: .../information>
1169
1170
1171
1172
1173
1174
1175
</information>
1176
<observation> Doc 3 clearly mentioned that the spouse of Al Gore is Tipper Gore</observation>
1177
1178
<sub guub_answer>Tippee Gore</sub_answer
1179
1180
Therefore, the spouse of the person who won the popular vote in the 2000 election iscanswer> Tipper Gore </answer>
1181
1182
1183
1184
Figure 12: Step-by-step reasoning for a 2-hop question identifying the spouse of the winner of the
1185
popular vote in the 2000 U.S. presidential election. The example demonstrates how breaking the problem into sub-questions and using evidence from multiple sources leads to the correct answer
1186
1187
22

---

Published as a conference paper at ICLR 2026
1188
1189
1190
1191
1192
1193
Case study for Esearch in 2-hops question (with incorrect observation)
1194
1195
Question: Who wrote "The Road to City A", with City A being the location where KFAB broadcasts?
1196
1197
1198
Ground Truth: ['Robert Ludlum']
1199
Sub-answers: ('Omaha', 'Robert Ludlum']
1200
Solution:
1201
To answer this question, we need to break it down into qubstiusssoon:
1202
1. Identify where City A is, where KFAB broadcasts.
1203
Determine who wrote "The Road to City with City A being the location identified above.
1204
1205 Let's address these step by step: <search> KFAB broadcasts </search>
1206
1207
<information> Doe I Title: KFAB ##> KFAB (1110 AM) is a 50,000 watt clear channel news and talk radio station licensed Omaha, Nebraska... </information>
1208
1209
1210
<observation> KFAB broadcasts in Omaha, Nebraska.</observatis ssub_answer>Omal Nebraska</sub_ntt answer>>
1211
1212
Now that we know City A, we move to the actor: <search> The Road Omaha actor </search>
1213
1214
<information> Doc 1 l
Title: KFAB ##> (no relevant info about the book or author)
1215
Doc 2*## Title: Paul Newman ##> Paul Newman was American actor, film director, and philanthropist, known for his roles in classic films such as...
1216
1217
</information>
1218
Search is incorrect! It will trigger erasure and regeneration of Search!
1219
1220
Erasure and Regeneration:
1221
<search> The Road to Omaha author </search>
1222
<information> Doc |*## Title: KFAB ## (no relevant info about the book or author)
Doc 2<## Title: Robert Ludlum ##> Robert Ludlum was an American author of 27 thriller novels, best known as the creator of Jason Bourne from the original The Bourne Trilogy... </information>
1223
1224
1225
<observation>> Based on observation, Doc 2 has the correct answer, Robert Lud- lun</observation>>
1226
1227
<sub muub_answer>Robee Ludlum </sub_
1228
1229
Based on the analysis, the writer of "The Road to Omaha" is <answer> Robert Ludlum
1230
</answer>
1231
1232
1233
Esearch makes an incorrect observation: although the first hop (KFAB - Omaha) is correct, the retrieval step for the author did not return documents mentioning Robert Ludlum. Lacking the correct evidence, the system hallucinates "Jack Elliott" as the author, yielding wrong final answer.
Figure 13:
1234
1235
1236
1237
1238
1239
1240
1241
23

---

Published as a conference paper at ICLR 2026
1242
1243
1244
1245
1246
1247
1248
1249
Case study for Esearch in 2-hops question (with incorrect retrieval)
1250
1251
Question: Who released the Fantasy Land Tour 2004 album, performed live in the same city where Alchip is headquartered?
1252
1253
1254
Ground Truth:
["S.H.E'] Sub-answers: ('Taipei', 'S.HE")
1255
1256
Solution:
1257
1258 To answer this question, we need to break it down into sub-questions:
1259 1. Identify the city where Alchip is headquartered.
Determine which performer released the Fantasy Land Tour 2004 album and performed live in that city.
1260
1261 Let's address each sub-question:
1262
<search> Alchip headquarters city </search>
1263
<information> Doc I Ic### Title: Alchip ##> Alchip is a fabless semiconductor company founded in 2003 and headquartered in Taipei, Taiwan... </information>
1264
1265
<observation> Alchip is headquartered in Taipei, Taiwan.<<observatte <sub awwwwnawwwaaaaa Tnaser>Taapei</subh Satpeiii/uubanswer>>>> answer>
1266
1267
1268
Now that we know the city, we move to the performer: <search> performer who released the Fantasy Land Tour 2004 album and performed Taipei </search>
1269
1270
1271
<information> Doc I Title: JI World Tour ##> J1 World Tour was the debut concert tour by Taiwanese singer Jolin Tsai... Doc 2 Title: Play World Tour ##> Play World Tour was the fourth concert tour by Jolin Tsai...
1272
1273
1274
1275 Doc 3<### Title: Castle (Jolin Tsai album) ##> Castle the sixth studio album by Jolin Tsai, released in 2004... </information>
1276
1277
<observation> The performer is Jolin Tsai L
1278
<sub Luub_answer-lolin Tsai</sub_ saai</subaanswer>> answer>
1279
1280
Based on the analysis, the performer who released the Fantasy Land Tour 2004 album, recorded live Taipei, is <answer> Jolin Tsai (INCORRECT) </answer>
1281
1282
1283
1284
1285
Figure 14: Esearch makes an incorrect entity alignment: although the retrieval step surfaced the correct document mentioning Fantasy Land Tour 2004 by S.H.E. the system failed align the entity
1286
1287
properly and instead associated the album with Jolin Tsai. This shows a case where the key evidence was present but reasoning and entity grounding went wrong, leading misleading yet internally consistent answer.
1288
1289
1290
1291
1292
1293
1294
1295
24

---

Published as a conference paper at ICLR 2026
1296
1297
1298
1299
1300
1301
1302
Case study for Esearch in 2-hops question (with incorrect entity alignment)
1303
1304
Question: Banir is located in which country, and in what year did the country achieve unification?
1305
1306
1307
Ground Truth: ['1963']
1308
Sub-answers: I'Malaysia', "1963']
1309
Solution:
1310
1311 To answer this question, we need to break it down into sub-questions: Identify the country Banir is located
1312 2. Determine the year of the country's s unification.
1313
1314 Let's address each sub-question:
<search> Banir location </search>
1315
<information> Doc I<## Title: Banior ## Banior is a village and gram panchayat in Nalhati I Block in Rampurhat subdivision of Birbhum District in the Indian state of West Bengal... Doe 2<## Title: Banagher ##it Banagher is a town Republic of Ireland, located in the midlands, the western edge of County Offaly in the province of Leinster, on the banks of the River Shannon...
1316
1317
1318
1319
Doe # Title: Banhpur ##> Banhpur village in Bulandshahr District, Uttar Pradesh, India. It is situated near Kuchesar Fort... </information>
1320
1321
1322
<observation> Banir is located in Ireland (INCORRECCT,</obserraiil aoopaaswee>reeladdeeee answer>
1323
1324
1325
Now that we know Banir is in Ireland, we need to determine the year of Ireland's unification: <search> Ireland unification year </search>
1326
1327
<information> Doc I<## Title: "United Ireland" ##> United Ireland refers to the proposition that the whole of Ireland should be single sovereign state... Partition since May 1921...
1328
1329 Doe 2<## Title: "Partition of Ireland" ##> The partition of Ireland divided the island into Northern Ireland and Southern Ireland on May 1921...
1330
Doc 3 Title: "Irish Free State" ##> The Irish Free State was established in 1922 under the Anglo-Irish Treaty...
1331
1332
</information>
1333
<observation> Ireland was united in 1922 (INCORRECT)</obeeee </observation> <sub looo_answer-aaee 1e22</sub_ansssteeeeee _nnsbbr-aaerr>r-_a
1334
1335
1336
Based the analysis, the year of the country Banir is located in (Ireland) happening unification is <answer> 1922 (INCORRECT) </answer>
1337
1338
1339
1340
1341
Figure 15: Esearch also produces an erroneous observation reasoning chain: Banir was incorrectly matched Banagher (Because the candidate document doesn't contain the target answer, the model makes close spelling eтrоr and misinterprets "Banagher" as Banir) instead of Banir (Malaysia), leading to the wrong unification year (1922 instead of 1963).
1342
1343
1344
1345
1346
1347
1348
1349
25

---

Published as a conference paper at ICLR 2026
1350
Case study for Esearch in 3-hops question
1351
1352
Question: Who established the first committee of correspondence 1772 in the capital of the state where Wellesley College is located, and why?
1353
1354
1355
Ground Truth: ['Samuel Adams"]
1356
Sub-answers: ('Massachusett'', 'Boston', 'Samuel Adams']
1357
ERL:
1358
To answer this question, need to break it down into sub-questions:
1359
Identify the state where Wellesley College is located.
1360
Determine the capital of that state.
1361
3. Find out who established the first committee of correspondence in 1772 in that capital.
1362
Understand the reason for establishing the committee.
1363 Let's address each sub-question:
1364
1365
<search> Wellesley College location </search>
1366
<information> Doc 1<## Title: "Wellesley, Massachusetts" ##> Wellesley is a town in Norfolk County, Massachusetts.... It is best known as the home of Wellesley College...
1367
1368
Doc 2<## Title: "Wellesley College" ##> Wellesley College is private women's 's liberal arts college located west of Boston in the town of Wellesley, Massachusetts...
1369
</information>
1370
<observation> Wellesley College is located in the state of Nassachuetts-</bbsevtttoo <sub_answaer answer> Massachusetts </sub_answer answer>
1371
1372
1373
1374
Now we know the state, we can find the capital: <search> capital of Massachusetts, United States </search>
1375
1376
<information> Doc 1<### Title: Massachusetts ##> Boston the state capital and largest city in Massachusetts...
1377
1378 Doc 2<## Title: Massachusetts ##> The capital of Massachusetts is Boston... </information>
1379
<observation> The capital of Massachusetts Boston.</observation>
1380
<sub answer> Boston c'sub_answen
1381
1382
1383
Next, we need to find out who established the first committee of correspondence in 1772 in Boston and why:
1384
1385
<search> first committee of correspondence Boston 1772 established by </search>
1386
<information> Doc 3*## Title: "Committees of correspondence" ##> In Massachusetts, in November 1772, Samuel Adams and Dr. Joseph Warren formed committee... in relation to recent British decision have the salaries of the royal governor and judges be paid by Crown rather than the colonial assembly... </information>
1387
1388
1389
1390
<observation> Samuel Adams established the first committee of correspondence in Boston in 1772.c/obsrrvation> <sub answer> Samuel Adams </sub answer>
1391
1392
1393
1394
1395
The reason for establishing the committee was to rally opposition and organize resistance against British reforms, particularly the Crown paying governors and judges directly, which undermined colonial seff-governance.
1396
1397
1398
1399
Therefore, the person who established the first committee of correspondence in 1772 in Boston, the capital of Massachusetts, was <answer> Samuel Adams </answer>
1400
1401
1402
1403
Case study showing step-by-step reasoning with sub-questions leading to the identification of Samuel Adams the founder of the first committee of correspondence 1772 in Boston.
Figure 16:
26

---

Published as a conference paper at ICLR 2026
1404
1405
Case study for Esearch on 3-hops question
1406
1407
What is the weekly publication in Jeanne Flanagan's 's birthplace that issued by the school attended by the author of "America-Lite How Imperial Academia Dismantled
Question:
1408
1409
Our Culture"?
1410
1411
Ground Truth: Yaaele Herald"]
1412
Sub-answers: ('David Gelernter', 'Yae University', 'New Haven', Yale Herald"]
1413
1414
Solution:
1415
To answer this question, we need to break it down into sub-questions:
1416
Identify the author of the book "America-Lite: How Imperial Academia Dismantled Our Culture".
1417
Determine the school attended by this author.
1418
3. Find the weekly publication in Jeanne Flanagan's birthplace that is issued by the school identified step 2.
1419
1420
Let's address each sub-question:
1421
<search> author the book "America-Lite" How Imperial Academia Dismantled Our
1422
Culture" </search>
1423
<information> Doe 1<## Title: America-Lite ##> America-Lite America-Lie How Imperial Academia Dismantled Our Culture (and Ushered in the Obamacrats) is a 2012 book by David Gelernter, published by Encounter Books...
1424
1425
1426
</information>
<observation> The author of the book "America-Lite: How Imperial Academia Dismantled Our Culture" is David Gelernte.</observatiion <sub answer>David Gelernter</sub answer>
1427
1428
1429
1430
Now that we know the author, we need to find out which school he attended: <search> school attended by David Gelernter </search>
1431
1432
<information> Doc 1 I### Title: "David Gelernter" ##> patriotism and the traditional family, writing "Gelernter is Jewish, and it is not likely that non-Jew would airily argue that obnoxious leftist Jews have taken over elite higher education.."
1433
1434
1435
</information>
1436
<observation> David Gelernter attended Yale University-z/observtion <sub_ mub_answers>ae Yale University</sub__nse answer>
1437
1438
Finally, we need to find the weekly publication in Jeanne Flanagan's 's birthplace (which is Boston, Massachusetts) issued by Yale University:
1439
1440
<search> weekly publication in Boston, Massachusetts issued by Yale University </search> <information> Doc 1<## Title: "The Daily Campus" ##> The Daily Campus The Daily Campus, founded in 1896. is student-run newspaper at the University of Connecticut that has a circulation run of 10,000 copies weekdays during the school year... </informatio>>
1441
1442
1443
1444
1445
<observation> The weekly publication in Boston, Massachusetts issued by Yale University is The Yale </oradd//bbsrrvatio>>> <sub awwwaan Yale Herald</sub answer> ansswr
1446
1447
1448
1449
Based on the analysis, the weekly publication in Jeanne Flanagan's 's birthplace (Boston, Massachusetts) that issued by the school attended by the author of "America-Lite: How Imperial Academia Dismantled Our Culture" (Yale University) is <answer>Yall Herald</answer>
1450
1451
1452
1453
1454
1455
Figure 17: Complex multi-step reasoning requiring identification of author, educational background, and geographic publication details.
1456
1457
27

---

Published as a conference paper at ICLR 2026
1458
1459
1460
1461
Case study for Esearch in 4-hops question
1462
1463
Question: The director "The Mystic Masseur" was born in a city that is in the same country Kattalai, city known for its famous temple. In which state of India is Kattalai located?
1464
1465
1466
1467
Ground Truth: ('Tamil Nadu']
1468
Sub-answers: ('Ismail Merchant', 'Bombay', 'India", "Tamil Nadu')
1469
1470
Solution:
1471 To answer this question, we need to break it down into sub-questions:
Identify the director of The Mystic Masseur.
1472
2. Determine the country where the director was born.
1473
Find out which state Kattalai, a city, located in that country.
1474
1475 Let's address each sub-question:
<search> director of The Mystic Masseur </search>
1476
<information> Doc |### Title: "The Mystic Masseur" ##> The Mystic Masseur The Mystic Masseur is 2001 Merchant Ivory film based on the novel of the same name by V. S. Naipaul. It is one of relatively few films directed by Ismail Merchant, who is better known as the producer in the Merchant Ivory partnership...
1477
1478
1479
1480
</information>
1481
<observation> The director of The Mystic Masseur is Ismail Merchant.c/observation <sub answer>Ismail Merchant</sub answer>
1482
1483
Now we know the director, we need to find out in which country he was born: <search>> Ismail Merchant birth country </search>
1484
1485
<information> Doc 1<## Title: "Ismail Merchant" ##> Ismail Merchant Ismail Merchant (25 December 1936 25 May 2005) was Indian-born film producer and director. He worked for many years in collaboration with Merchant Ivory Productions... Born Ismail Noor Md. Abdul Rahman in Bombay (Mumbai) on Christmas Day 1936...
1486
1487
1488
1489
</informatio>>
1490
<observation> Ismail Merchant was born in India.c/observation> <sub suu_annswer>ndiaa//aa >niie>/bee>>i///baasasne>>>>>>>> answer>
1491
1492
Finally, we need to find out in which state Kattalai, a city, is located in India:
1493
<search> Kattalai, India state </search>
1494
<information> Doc 1 *## Title: "Kattalai Amman Temple" ##> Kattalai Amman Temple Kattalai Amman Temple is Hindu Temple situated in Kanyakumari District the southern part of Tamil Nadu, India. The temple is located in the village, surrounded by mountains and canals...
1495
1496
1497
1498
</information>
1499
<observation> Kattalai the state of Tamil Nadu, India.</observation>>
1500
<sub answer> answer>Tamml Nadu, India</sub answer>
1501
1502
Based the analysis, the state in India where Kattalai, city, is located is <answer> Tamil Nadu, India </answer>
1503
1504
1505
1506
Figure 18: Esearch can efficiently handle a 4-hops reasoning question: after gathering relevant information across multiple hops, the model completes the reasoning process just three search queries.
1507
1508
1509
1510
1511
28


Reviews:
[{"review_id": "NKn3Wt8Vau", "reviewer": "Reviewer_knbt", "summary": "This paper addresses critical reliability issues faced by search-augmented LLMs when performing complex multi-hop reasoning tasks. It identifies three primary failure modes: decomposition errors, retrieval errors, and reasoning errors. To mitigate these issues, the authors propose ERL, a framework that identifies, erases, and regenerates faulty reasoning steps, thereby preventing error propagation. Extensive experiments on multi-benchmarks demonstrate substantial performance improvements with ERL, achieving new state-of-the-art results.", "questions": "Please check the weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces ERL, a novel approach to addressing the inherent brittleness in search-augmented LLMs. The key innovation lies in explicitly identifying, erasing, and regenerating faulty reasoning steps, a method previously unexplored in multi-hop reasoning contexts. This creative combination of reinforcement learning with targeted error correction significantly advances existing frameworks.\n\n- Validates the framework across multiple challenging multi-hop reasoning datasets, HotpotQA, MuSiQue, 2Wiki, and Bamboogle, demonstrating generalizability.\n\n- This framework enhances the robustness and real-world applicability of search-augmented language models, delivering state-of-the-art results on multiple benchmarks. Its dynamic error-correction capability ensures improved reliability and adaptability in practical scenarios.", "weaknesses": "- The main results are point estimates without confidence intervals, so it’s unclear whether reported gains exceed noise.\n\n- The manuscript does not present empirical measurements or benchmarks of runtime, memory use, or computational resource demands. The lack of quantitative efficiency analysis weakens the practical argument for real-world applications.\n\n- Certain equations contain notation ambiguities or inconsistencies, such as unclear definitions or insufficient explanations for symbols.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U3hVtbXRC8", "reviewer": "Reviewer_jujf", "summary": "This paper proposes Erasable Reinforcement Learning (ERL) for search-augmented LLM agents: the agent detects faulty steps in multi-hop reasoning (decomposition, retrieval, or reasoning), reverts those segments, and regenerates them in-place with dense stepwise rewards, reducing error propagation. On HotpotQA, MuSiQue, 2Wiki, and Bamboogle, ERL-trained models beat strong baselines as new SOTA in both offline and online settings", "questions": "- I would expect longer rollout time and more research queries thus make it both cost more and takes a longer time for each unit update/iteration, but I am not sure about the overall effect.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "* Novel training strategy: Introduces *Erasable RL* with three erasure signals—plan/init search, subsequent search, and sub-answer—to cut off faulty steps and retry, directly targeting error propagation.  \n* Dense, well-aligned rewards: Designs stepwise rewards for retrieval and intermediate reasoning, with token-level attribution, improving credit assignment beyond sparse final answers.  \n* Strong empirical results & diagnostics: The experiments shows the proposed approach achieves SOTA across four multi-hop QA datasets in both offline and online settings, with consistent gains at 3B/7B and ablations showing each erasure component is complementary.", "weaknesses": "- Unclear implementation details. Which dataset is used for training? And also, it is unclear on how the grounding gold evidence and gold sub-answers set is constructed. Furthermore, the dependence on gold set may be infessible for many real-world applications, unless a general and lightweight approach can be provided.\n- Concerns on efficiency and scalability : The erase–regenerate loop adds computational overhead and may require repeated passes when multiple heterogeneous errors occur; training notes also show retrieval quality is bottlenecked by a local Wiki-18 setup, suggesting a practical bottleneck beyond the algorithm itself.  Also for real-world challenge setup, it is likely the search cannot provide answer, which means the model will struck in the loop, unless the training data and search resource ensure valid information can be provided.\n- Writing issue, e.g. L160 Jin et al. (2025b) > (Jin et al., 2025b). These issue occurs very frequent spanning intro, sec 4.1, and related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DiYkRrVMLi", "reviewer": "Reviewer_BrTp", "summary": "This paper proposes Erasable Reinforcement Learning (ERL), a novel framework designed to enhance the robustness of multi-hop reasoning in search-augmented LLMs. The key insight is that current search-augmented LLMs suffer from decomposition, retrieval, and reasoning errors, where a single failure can derail the entire reasoning chain. ERL addresses this by introducing an erasure mechanism that detects faulty reasoning steps, removes them, and regenerates new reasoning from the last correct state.", "questions": "1. How does ERL’s computational cost compare to PPO or GRPO during training?\n2. Can ERL generalize beyond QA tasks, for example, to long-form reasoning or tool-using agents? If so, what modifications would be required?\n3. How sensitive is ERL to the thresholds ($\\alpha$, $\\beta$) used to trigger erasures? \n4. How does ERL scale with reasoning depth (e.g., beyond five hops)? \n5. In real-world settings without gold evidence, how could dense rewards such as $R_{search}$ and $R_{subanswer}$ be approximated?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of erasable reasoning is inspired by human-like self-correction, effectively addressing a key weakness in multi-hop reasoning systems.\n\n2. ERL demonstrates consistent and notable improvements over strong baselines across multiple datasets.\n\n3. The paper provides thorough ablation studies (plan/search/sub-answer erasure), comparisons with PPO and GRPO, and both offline and online evaluations.\n\n4. The writing is clear and mathematically rigorous, with precise formulations of the MDP setup, reward decomposition, and erasure operators.", "weaknesses": "While novel in its integration, ERL primarily extends existing reinforcement learning frameworks with a modular erasure component. The conceptual depth may be viewed as incremental rather than fundamentally new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SL1Tlxv5d3", "reviewer": "Reviewer_gvRH", "summary": "This paper proposes Erasable Reinforcement Learning (ERL), a new framework for enhancing the robustness of search-augmented LLMs in multi-hop reasoning tasks. The key idea is to explicitly identify and erase faulty reasoning or retrieval steps within a reasoning trajectory, then regenerate subsequent steps from the last correct state. The method introduces three erasure mechanisms (plan, search, and sub-answer erasure) triggered by reward thresholds on intermediate steps. ERL is trained using Qwen2.5-3B/7B models, evaluated on different benchmarks. Experiments show consistent SOTA gains and improved stability over PPO/GRPO. Ablation studies confirm the complementary roles of the three erasure modules, with sub-answer erasure contributing most.", "questions": "1. What is the training cost relative to PPO/GRPO baselines (e.g., GPU hours given the same steps)? \n\n2. How sensitive is ERL to the choice of thresholds hyperparameters? \n\n3. Have you tried applying ERL to other reasoning-centric tasks, such as math or coding?\n\n4. Are there qualitative/quantitive analysis or visualizations showing where erasure occurs in the reasoning chain, and how it affects the final outcome?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The “erasable” mechanism introduces a fresh perspective distinct from prior step-wise or reward-shaping RL methods. It explicitly operationalizes fault detection and local correction, which is both intuitively appealing and empirically validated.\n\n2. The multi-component reward (for search, sub-answer, and final answer) effectively mitigates the sparsity of typical RL signals, and the formulation is clear and well-motivated.\n\n3. The component-wise ablation and comparison with PPO/GRPO support the claim that ERL provides stable optimization and interpretable corrective behavior.", "weaknesses": "1. The iterative erase-and-regenerate cycle introduces non-trivial computational overhead, which the paper acknowledges but does not quantify. \n\n2. The erasure triggers and dense-reward coefficients appear manually tuned; there’s no sensitivity analysis showing robustness to these hyperparameters.\n\n3. The evaluation is limited to search QA-style benchmarks. It remains unclear whether ERL generalizes to other agentic reasoning tasks (e.g., math, tool use, code reasoning).\n\n4. More clearer illustrative examples or reasoning traces would make the method more intuitive to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)