Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers
Authors: Zhengliang Shi
Lingyong Yan
Dawei Yin
Suzan Verberne
Maarten de Rijke
Zhaochun Ren
Venue: NeurIPS 2025 poster ()
Keywords: information retrieval
large language model
retrieval-augmented generation
Primary area: deep_learning
TL;DR: We propose ExSearch, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process.

Abstract: Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose ExSearch, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning.  To enable LLM with this capability, we adopts a Generalized Expectation-Maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated data, progressively improving itself for search. We further theoretically analyze this training process, establishing convergence guarantees. Extensive experiments on four knowledge-intensive benchmarks show that ExSearchS substantially outperforms baselines, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce ExSearch-Zoo, an extension that extends our method to broader scenarios, to facilitate future work.

Iterative Seff-Incentivization Empowers Large Language Models as Agentic Searchers
Zhengliang Shi' Lingyong Yan Dawei Yin
Suzan Verberne" Maarten de Rijke* Zhaochun Ren Shannong University, Qingdao, China Baidu. Inc, Beijing, China 3 Leiden University, Leiden, The Netherlands
Universitty of Amsterdam, Amsterdam, The Netherlands (zhengliang. shii, lingyongy Jegmail com yindawei@acm org [s. verberne, 2. wer.nneliac...eeeenniii.......
leidenuniv.ni M.deRijke@uva. nl
Abstract
Large language models have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains challenge due the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose EXSEARCH, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self- incentivized process. each step, the L.M decides what retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning. To enable L.LM with this capability, EXSEARCH adopts generalized anpectationmmaximizztinn algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM them with re- re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated trajectories, progressively improving itself for search. We theoretically analyze this training process and establish convergence guarantees. Extensive experiments on four benchmarks show that EXSEARCH outperforms baselines substantially, e.g., +7.8% improvement on exact match score. Motivated by these promising results, introduce ExSEARCH-Zoo, extension that extends method to broader scenarios, to facilitate future work. Code available on O EXSEARCH.
Introduction
Information Retrieval (IR), one of the most fundamental data mining techniques, aims to understand
complex queries and extract relevant information from external sources for users 15 Recently,
large language models (LLMs), which exhibit remarkable language understanding and reasoning
abilities, have been widely integrated into to enhance traditional techniques 22
g9
For example,
some work uses LLMs for query rewriting [6] document re-ranking [67 to augment search engines, while other work uses LLMs to summarize information and generate accurate responses (78)
Despite the progress made by LLMs in IR applications, enabling LLMs to effectively seek accurate knowledge in complex downstream tasks remains challenge 4
Specifically, many real- world tasks, such as multi-hop question answering, require iterative and dynamic retrieval, where directly issuing complex query composed of multiple sub-queries often results low retrieval coverage 30 33 As retrieval is typically imperfect [82 87). even for simple one- hop query, the retrieved results contain irrelevant content, leading to misleading context S8 Thus, it is crucial to teach LLMs how to interact with retrievers and reflect on retrieved content as reasoning unfolds.
16 72
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2.5-8-253388
3.2--3.2-38
Mistral-713 Binsttl-iBinsttuee
16
85-
05
Daat
36
Training
22
Sxt
Trubiing Sxt
Evaloztico Set
Evaloation Set
Evaluatinn Set
27
10
Training Dendion
Training Rendion
Training Itntion
Qween: 2.5-70
Elama 3.1-BD
Mistral 240 small-2501
Mards
65-
Mards
68
Mands
80
Tratting Set
42
Traiting Sit
Tratiing Set Essliaation
Pealuction
Peslistion
20
30
30
Trinnig Itcration
Training Itrration
Training ltrrtionn
Figure I: Performance on HotpotQA dataset when applying our EXSEARCH to different LLMs.
To address these limitations, previous work typically cascades information-seeking pipelines (c.g., query decomposition [43 knowledge extraction (52]), and trains LLMs for specific stages with synthetic data independently 40 88 However, aligning different retrieval stages with end-to-end supervision remains active research area.
In this paper, we propose EXSEARCU, an exploratory search framework that empowers an LLM as agent with fine-grained reasoning and search capabilities through self-incentivized process. Specifically, EXSEARCH formalizes three core actions: (i) thinking: generating a query based on the evolving search trajectory; (ii) search: triggering retriever, and (iii) recording: extracting fine- grained evidence to support subsequent reasoning. Given an input task, the LLM interleaves these actions to progressively explore a search trajectory, which is sequentially composed of sub-queries, retrieved documents, and supporting evidence. The final answer is generated based on the entire trajectory, effectively grounding the generated response in external knowledge.
To incentivize the LLM with this agentic search, EXSEARCH adopts the generalized expectation- maximization (GEM) algorithm, which alternates between two main steps: trajectory exploration (E- step) and re-weighted trajectory learning (M-step). The key innovation is treating search trajectories as latent variables, training the LLM to reason and search end-to-end. In the E-step, we approximate the current distribution over search trajectories through importance sampling [69]. For each input task, the LLM generates candidate trajectories, each automatically assigned an importance weight based on how well it supports the correct answer. In the M-step, we reweight these trajectories to construct an evidence lower bound (ELBO), which is then maximized to update the LLM parameters. This step enables the L.M to learn from its own generated data, encouraging it to generate more supportive trajectories and accurate answers. By interleaving the EM steps, we form a self- iiff-incentivize loop that progressively optimizes the model to search relevant knowledge and reason over it.
Our method tightly integrates LLM with IR and diverges from prior work by enabling unified LLM for dynamic document retrieval, evidence extraction, and answer aggregation through self-improving process. Extensive experiments
wide range of knowledge-intente intensive benchmarks demonstrate the improvement of EXSEARCH over strong basclines. To further understand its advantages, we provide thcoretical analysis, which illustrates that the proposed seff-incentivized framework ensures stable convergence, as briefly shown in Figure I These promising results motivate us to extend comprehensive resource that extends EXSEARCH by two dimensions: (i) diverse backbone LLMs across different model families (LLaMA, Qwen) and scales (7B, 24B parameters); and (ii) extended actions, i.e., document re-ranking, to enrich the action space in vanilla EXSEARCH.
EXSEARCH more diverse scenarios. We therefore introduce ExSEARCH-Zoo
Our contributions are summarized as follows:
We propose EXSEARCH, an exploratory search framework that empowers an LLM for agentic retrieval, evidence extraction, and answer aggregation through a self ielf-incentivizatiin process.
2. We provide theoretical analysis Extensive experiments on four benchmarks also demonstrates the superiority of our method.
training process and establish convergence guarantees.
We introduce ExSEARCH-Zoo, an extended resource that generalizes EXSEARCH more scenar- ios, facilitating future research.

---

Trajectory
Trajectoy (() d)..)
Sampling
think
search
record
Importance Weighting
Mibquaery
documents
exidence
Trajectories
LUM
docs
indience
LLM
neeativv
Re-weighting
w(t)
Loss
Update
Updated
(t))
Pannser [x.z]
Weights
Model g
step: Trajectory Exploration
M stop: Re- Weighted Trajectory Learning
Figure 2: Overall framework of the Dopectation-Mxxmizaaiee process EXSEARCH. The E-step samples search trajectories and assigns each weight based on its likelihood leading to correct answer. The special token End marks the completion of reasoning. The M-step trains the LLM, encouraging the LLM to generate more supportive search trajectories and accurate answers.
2 Method: EXSEARCH
In this section, we first introduce how the proposed method, EXSEARCH, models the iterative and dynamic search strategy as an agentic information-seeking [6 92 process. Next, we explain how the LLM-based search agent in EXSEARCH can be incentivized to learn this search pattern through self-improving process. The theoretical analysis of this learning process is presented in $
2.1 Search process in EXSEARCH
Our method draws inspiration from Exploratory Search 44 a well-established paradigm that con- ceptualizes searching as a dynamic, iterative process, involving the continuous refinement of queries based retrieved intermediate results. Accordingly, EXSEARCH simulates this process through an agentic flow, interleaving several actions: thinking, search, and recording, which progressively explore search trajectory. Specifically, in thinking, EXSEARCH determines the next retrieval action by formulating a query I; based on the current trajectory. In search, an external retrieval module R is triggered to obtain the top-K relevant documents d, R(x,) dr,j j E [K] for the query Ty Finally, in recording, EXSEARCH reflects on the retrieved documents and extracts fine-grained evidence e; to support the subsequent retrieval stage. Formally, we model the LLM-explored search sequence {(x; de ea) i E [=]]], with a joint likelihood:
trajectory
p x; 0)
II 1=1
((z,, d,, ez z< 0) =
p (z) x,, ==;=):;))>...... (es 24 R(x:):0), (1)
where the 0 denotes the LLM parameters. After such an interleaved search and reasoning process, the LLM generates the final answer y by aggregating the information from z, denoted as: y ~ p(y
z: f). The system prompt used in EXSEARCH is provided in Appendix F.I
2.2 Iterative self-incentivization for EXSEARCH
Figure presents the overall framework. EXSEARCII empowers the LLM with agentic search capabilities by alternating between the E-step and M-step, based on the generalized expectation- maximization algorithm [45]. Below, we begin by deriving the training objective of ExSEARCH. We then detail the iterative training process, demonstrating how the LLM progressively improves through trajectories generated by itself.
Deriving the learning objective. Given the user's input task define the training objective in EXSEARCH as optimizing the LLM to generate the correct answer y after gathering useful information, which can be modeled as the joint probability of the search and generation process:
Preliminary:
log p(y
x;;
0) log Σ p(u. 2;; 0).
Here = ((r,, di i [[=]]] which denotes trajectory interleaving the thinking, search and recording introduced in [ However, marginalizing over all possible trajectories is generally intractable. To address this, we treat z as a latent variable and derive a variational lower bound. Specifically, we introduce proposal distribution g(z xx) to estimate the sampling space of and
3

---

apply Jensen's 's inequality to the marginal log- likelihood in Eq. 2
ply. 2 xz
ply, x x; 0)
p(y. 2 xx;
8)
log Σ <(</))
((2|x)) log
I (xmg(()
log q o(=/ar)
(3)
((2(x)
g( (r)
We refer to this expectation Ex-g(=/=) [log p(v. 2 as the variational evidence lower bound (ELBO)
of log p(y]2;; 0). The bound (2) becomes tight if and only if the proposal distribution matches the true posterior: ((=[x)) R p(z v, 0). Therefore, we can naturally improve the log p(y|:: 0) from the perspective of Expectation-Maxiiizall
In each iteration, the E-step estimates the distribution p(z ly, 0) over trajectory x while the subsequent M-step updates the LLM parameters 0 by maximizing the ELBO under the sampled trajectories.
E-step: Trajectory exploration. In the thth training, to achieve the boundedness of Eq. (3) we assume (z([2)) = p(z x, y; 8*) and rewrite the ELBO as follows. The H(-) denotes the entropy.
ELBO E Ez-p(ell.)000)) pog = x;; 0)] + H(p(= x. v; 0²)
entropy p( 8s, 0t)
The entropy H(p 2z;; 0²)) is non-related to 0 and can be denoted constant c. As pointed by prior work [58] 669 933, sampling from the exact posterior (c.g., p( (z lr, y: 0')) is typically intractable. Therefore, we apply an importance sampling strategy [14 69]. The main conception is instead sampling from simple distribution (i.e., p( (x;0% 0*) in this work), and assigning each sample with an importance weight. This corresponds to producing the trajectory as formulated in Eq. TT by the
current LLM with parameters ge, With this strategy, we rewrite Eq.
as:
ELBO (x=p(()000)) [w](2) log = x:; 0)] + e,
(5)
where wxz) = p(z x. y. 0')(((z (²) denotes the importance weight. Using Bayes' theorem, we can decompose the posterior as: p(= x. y, 8') p(z 8*) p(y z, 0¹), which allows the importance weight to be further simplified as w (2) x p(y 2, xr; 0²). This weight directly reflects how well the search trajectory z supports the L.M in generating the correct answer y.
M-step: Re-weighted trajectory learning. In the M-step, we update the model parameters by maximizing the ELBO in Eq. (5) formulated as: 0 arg max, ELBO. By applying the product rule to decompose the joint likelihood p(y, æ|æ; 0), we can split the ELBO into two terms as follows, which corresponds to two objectives: (i) learning to search via Lxi (ii) learning to answer via L..
0 arg max E - p(=((00)))))
[w(z)) p(y, 2; 0)]
(6)
arg max x> wx(2) log p(= O) + wx2) log xr 2;
0
learning to reason
learning to answer
Therefore, we can understand and interpret the training process in the M-step from both retrieval
and answer aggregation aspects. In more details, the term LR == w(2) (z) log p(z z x; 0) encourages
the model to generate high-quality search trajectory = {(x; d, e;) i E [ to gather relevant documents. According to Eq.
[ Lx can be calculated as:
LR Σ |=|
p(r; =,;; 0) + log d; m;; 0);
(7)
where Zzg = [I1z, dizi } denotes the trajectory up to ith step. Similarly, the L. = log plu | I, z; () trains the model aggregate all useful information from z to produce the final answer. The overall optimization in this M-step can be achieved via stochastic gradient descent, which is highly compatible to existing computation library like Pytorch. The gradient Vo with respect 0 can be computed as:
V VoELBO(8) - -==-p())=00) (wx + Cx)..
(8)
Pseudo-algorithm. To clarify the overall procedure in EXSEARCH, we present Algorithm I In the E-step, the LLM generates search trajectories on its own and evaluates each one with an importance weight w( reflecting how well the trajectory supports generating the correct answer. In the M-step, the L.M is trained on these trajectories using a weighted loss, learning to generate supportive search paths and accurate answers.
in

---

Algorithm 1: Training process in EXSEARCH, which alternates between the E-step and M-step. Training iteration N; Maximal step T.
Input:
Initial LLM 00; Training data D = {(*', , for f to N do
// E-step: Trajectory Exploration for example (z,y) in training set do
while no end and < T do
Generate sub-query æ; p(x; z,
thinking: generate sub-query & search: retrieve documents > recording: extract key evidence > append actions into trajectory
Retrieve document candidates d; = R(x,)
Extract evidence e; p(es) 24, d, d; 0*
2 f 2 U (I, d,, e.)
Compute importance weight: w(2) - ply =0)
// M-step: Re-Weighted Trajectory Learning
CR(I, y; = E [==p(=)2;01) [w(z) log 1;; x;])]
define the loss for reasoning
Ca(2, y; 0) == (x-p(a)m:0)) [u(z) log 2, 2;)]
> define the loss for answer optimize through gradient
gc+1 arg max E (way)>/ [Cx y; 0) + (I,
if no improvement on validation set then
L Stop training
Early Stop
Output:
3 Theoretical Analysis
During the training of EXSEARCH, we alternate between the E-step and M-step to progressively improve the LLM. Below, we prove the training convergence by showing the non-decreasing opti- mization after each iteration and analyzing the upper-bounded property of the learning objective.
Lemma 3.1 (Non-decreasing optimization). After training in the tth (t E Z+) iteration, the overall learning objective log p(y x. f satisfies p(y xx, +*+ log ply x. 0*).
Proof. At tth iteration, start with E-step to sample trajectories z from the current LLM 0f via an importance sampling strategy. As described in Eq. (5). the ELBO can be written as:
ELBO 8*) =z=p(=(2001) (w(z) log p(y, x:; 0)] +
In the subsequent M-step, we update A by maximizing the ELBO: g++1 = arg max, EL 0*). By construction, this ensures ELBO(8++1 2 ELB 8¹). Furthermore, using the definition of ELBO from Eq. (3,, we observe:
loge(u | 1; 11; 2 ELBO(Ø*+1 000) ELBO( 0*) log ply x;; 0²)
(9)
Thus, we establish that log p(y la; g++!) 2 log p(y ar; 8*). demonstrating the non non-decreasing nature of the optimization process. Next, analyze the boundedness of the learning objective. Since p(y|æ; 0) E [0, 1]. it follows log p(y|<;; 0) E (-o,,00 O). Therefore, the sequence [log Plyl:; 0') non-decrgasing and upper-bounded. We then apply the following convergence theorem from [5)
Theorem 3.2 (Monotone convergence theorem). If sequence {an} is monotonic (either non- decreasing or non-increasing) and bounded, then it converges to a finite limit.
Applying Theorem 3.2
the non-decreasing and upper-bounded nature of (logr z; 0') ensures
that the sequence converges to finite value, proving the training convergence of EXSEARCH. We also provide more analysis in Appendix C.3 to rethink and understand the training process.
O
Experimental Setup
Datasets and evaluation metrics. Following prior work 37
57
88
90 we conduct experi-
ments on range of well-established benchmarks: Natural Questions (NQ) R6
HotpotQA [84),
MuSiQue [70], and 2WikiMultiooQQA (2WikiQA) (20). Table Appendix summarizes their
key statistics. We use three metries from KILT 49 F1, Exact Match (EM), and Accuracy (Acc).
5

---

Table I: Comparison between our proposed EXSEARCH and baselines, with the best and second best results in bold. Blue lines represent the results of EXSEARCH using the same backbone LLMs as the most commonly-used baselines. Green lines represent analysis results for applying EXSEARCH to various LLMs. indicates no open-source code can be used to run the experiments. indicates cases where the model produces long-form answers, which struggle align with the short-span ground truth format. In these cases, we suggest using Acc. as a more reliable metric.
Tasks
NQ
HolpoiQA
MuSiQue
2WiRiQA
Avg.
Metries
F1
EM
Acc.
Fi
EM
Acc.
F1
EM
Acc.
EM
Acc.
F1
EM
Acc.
Direct reasoning without retrieval-ougmented generation
Deepseek.R 1-667BB
49.45
35,71
43.83
46.98
35.83
37,80
17.34
10.22
12.69
52.18
43.83
50.66
41.49
31.40
36.24
GPT-4o 58
48.76
35.75
43.03
54.13
36.52
51,59
29.07
18.92
22.97
51.31
40.45
53.07
45.8
32.90
42.66
GPT-3.5-turbo
17
42.11
38,60
40.60
34.90
24.57
81
22.73
14.14
16.29
33.90
30.40
32,45
31
26.93
30.30
Qwen2,5-72B
45.68
30.12
45.67
38,80
29.20
32.00
20.40
11
11
12.54
42.70
34,40
33.61
36.90
26.28
30.96
1lama-3,3-70B
[k]
48.70
36.00
36.00
49.10
37.80
39.20
23.60
14,80
16.00
54.20
46.00
50.00
43.90
33.65
35.45
Mistral-8x7B 25
40.87
40.10
39.60
25.19
16.40
25 80
11.60
6.80
14.80
30.21
27.05
29.67
26.97
22.59
27
QwQ-32R [8
33.09
23.00
32.20
33.34
25
40
26.60
18.85
9.00
9.00
40.90
34,
36.40
31.55
22.95
26.05
Qwen-2.5-32B 33
33.10
23.00
34.20
34.74
25.40
27.60
18.90
8.50
10.03
36,34
29.80
34.40
30.77
21.68
26.58
Advanced retriesal-ougmeented generation
RankRAG (llam--3.1-70m [8]
54.20
55,40
42,70
43.90
38, 20
ChaiQA (llama-2-70BB) [2]
34.54
37.41
44.60
17.05
16.64
19.24
31,90
26.80
32.56
32.02
25.12
30.65
Recomp (Flan-
78
42.67
37.47
40.32
42.72
38.72
41,55
24.96
17.34
21.46
38.26
32 17
36
37.15
31.43
34.86
Retrubust (llama 13B) [87
43.82
37.03
39.56
40.54
35.59
18.11
19.61
38.65
39.77
35.41
34.43
InstructR AG (llama33, 3-8)) 77
39.21
37.82
37.58
37.31
36
35.31
25.88
14.94
20.45
40.01
44.57
38.91
35.60
33.52
33.06
DDR (llama 3.1 8BB) 400
30.51
31.71
13.54
10.57
14,21
38.40
35.44
37.41
31.10
28.70
31.14
RankRAG (1lama-3. 3. 1-8B) [8
50,60
46.70
35.30 30
36.90
31.40
ChatQA (llama-2-7B)
[2]
34.54
23.64
37.41
44.60
33.40
33.40
17.05
16.64
19.24
31.90
26.80
32.56
32.02
25.12
30.65
Iterative retrieval augmentedd generation
GeaGround (GP1)335) 64]
50.31
40.24
43.60
52.26
45.31
47,27
27
36
18.34
20.24
50.21
42.31
45.
45.04
36.55
39.18
IRCsT (GPT:3.5)
43.21
58.40
45.50
30.50
19.01
22.87
36.54
44.86
37.24
DSP (CP1333)) [2]
42,25
29.10
42.00
47.10
34.67
42.73
19.88
10,80
13:40
44.52
39.64
44.43
38.44
28.55
35.64
SearChain (GPT:3.5) 3D]
0
0.00
9.22
0.00
43.69
5.75*
36.49
Iter-RetGien (GPIT-3.5)
[6]
28.30
41.04
44.10
21.04
17.69
20.19
36,00
42.17
31.52
31.11
Verify and 3.P.
26.68"
12,44
0.00
27.43
5.87"
0.00
10.01
0
32.68
17.86
27.62
Gen-Ret-Gen (CPP)-3.5)
1
46.66
38.06
48.88
49,59
37.69
45.03
25.94
13.24
17.82
40.26
29.43
39.23
40.61
29.60
37.73
Search-al (QwQ:32B)
53.31
43.51
16.64
19.42
42.61
45.41
44.14
37.62
Search-RI (Qwen-2,5-7B)
28
54.26
42.21
$1.35
58.04
46.51
52.46
30.03
21.21
23.37
52.62
49.64
50.43
48.74
39.89
44.02
SEL.F-RAG (llama 7B)
42.50
21.50
21.50 50
9.43
7.10
27.33
23.52
20.80
30.01
20.99
24.90
Ours-Qwen-22 2.5.77
56.37
47.07
$1.75
62.59
50.35
54.32
29 68
22.03
24.34
57.14
$2.62
54.37
51.45
43.02
46.20
Ours Lla-alama-333
50.76
60.72
30.83
20.98
24.65
54.62
47.48
54,21
50.35
39.94
45.80
Extended analysis on various backbone LLMs
Ous-Qwen-225-338
46.23
36.76
39.12
54.32
42.22
46.08 19.44 13.76 13.94
43.39
37.24
44.78
40.85
32.50
35.98
Ous--Lama--33-3B8
41.42
33.49
37 44.12
33.53
36.14 17.64 11.23 11.82
36.28
42.22
36.23
28.63
31.34
56.83
45.13
$2.05
39.65
50.35
54.78 78 30.32 23.47 24.98
53.93
47.38
54.82
50.18
41.58 46.66
Ours-Mistrl-2501-244
47.62
56.39
67.03
54.51
35.84
23.68
28.54
53.19
61.59
44.75
51.63
Baselines. We compare EXSEARCH with a range of baselines, categorized into three groups based on their use of retrieval strategies: (i) Direct reasoning without retrieval: These methods pro- duce the answer to the input query by prompting the LLM to reason over its parametric internal knowledge, without an external retriever. This includes few-shot prompting off- tffethe-shelf LLMs. i.e., DeepSeek-RI [7] GPT-4o 48 GPT-3.5 (47) Qwen2.5 33 QwQ-32B [68] LLLMA- [13], and diistral-8x77 7 26 (ii) Advanced RAG: These methods retrieve relevant documents, followed by filtering or re-ranking, and incorporate useful documents into the L.M'' s context for answer generation.
We include: RankR AG (88) ChatQA 42 RetRobust 87 I. RAG-DDR
40 (trained by DPO (54)) )
and InstructRAG (74). (iii) Iterative RAG: These methods allow LLMS to interact with the retriever iteratively. We include the widely-used methods: GenGround (64) DSPy (2). SearchChain (80), Iter-RetGen [61] Verify-and-
[94), Gen-Ret-Gen (0. Search-ol [39] Search-RI (28) (trained by DPO (60), and Relf-RAG 4 We implement the above baselines following their official code. Full model descriptions are included in Appendix D.3 Following the most commonly used recipe for baselines, we set the size of retrieval documents to 10 for all advanced RAG baselines and 5 for all agentic RAG baselines (since they can iteratively retrieve), as well as for our method. Following RankRAG S8), for baselines without publicly available code, we report results from their original papers but only for reference (marked as "_" for metrics that were not reported in the original paper).
Implementation details. We apply the proposed method to various backbone LLMs, such as Qwen- 2.5-7B- 83). Following prior work [64] 80). we use the Wikipedia passage dump from December 20, 2018, the retrieval corpus and adopt C0 (59) for document retrieval. To

---

Table 2: Recall @ @ (K=3,5) for our method (w/ 8B Table 3: Ablation study where remove each
LJama3.1 and 7B Qwen2.5) and strong baselines. component from the vanilla EXSEARCH.
Tasks
HotpotQA
Musiqque
2W3RKD4
Tasks
HolpotQ4
2WiKIQA
Metries
R@3
R@5
R
Metrics
F1
EM
Acc.
F1
EM
Acc.
Co1BERIV20
14.27
17.91
29.13
Ours Qwen2.5-BB
62.59
50.35
54,32
57.14
52,62
54.37
Revomkins
thinking
54.23
43.46
51.45
45.08
42.36
45.35
MonoTS
71.37
78.61
14.77
18:99
soarch
43.34
39.61
42.25
42.15
41.46
Rank
RanlZoppe
00:36
71.65
52.86
$7.88
17.82
16.90
15.56
20:23
22,65
30.55
32.83
wio w(s)
reconding
57.23
43.36
46.35
47.45
52.16
53.35
46.55
50.49
49.35
QUery decoppatiioonn
Ours- Ioma3 1-8B
60.72
47.59
53.59
$4.62
47.48
S4:21
Semth-l
66.26
71.76
45.71
65.66
20.37
30 28
32.81
36.22
vio thinking
52.11
41.31
45,21
48.43
42.13
47
Search-r cl 26
68.21
70.00
42.71
63.41
18.45
25:48
27.73
34.52
sourch
48.25
39.21
11.86
41.24
35,32
39.34
Oum-Qumm255:71
71:20
79:42
64.06
66.94
35.62
40
33.94
36.39
reconding
Ouss Llum33.1 1-188 XB
72.21
78.70
62.52
63.49
24,60
41.22
33.68
36.05
ur(=)
54.45
43.56
49.23
49.61
42.51
49.54
equip the LLM with the basic skills required to follow our iterative think-search-recordd pattern, we initially generate 1,000 pseudo-examples and fine-tune the model on them, following cold-start setup as in prior work [TO] (66] (96). Data collection details are provided Appendix [D.] We also investigate the effect of data scale in the cold start on final performance (see $5.44. For the subsequent EM training, we set the number of iterations N to 5 and report model performance at each iteration.
We use DeepSpeed ZeRO 3 [56] with a learning rate of 2 10 More details are provided in D.5
5
Experimental Results
5.1 Overall evaluation
Surpasses reasoning LLMs without retrieval. As shown in Table 1 EXSEARCH substantially outperforms large-scale LLMs that rely solely internal knowledge across all benchmarks. Com- pared with GPT-4o and LLaMA-3.3-700 (both using large-scale parameters), our method achieves improvements of +5.63 and +7.55 in average F1, respectively, despite using much smaller 7B backbone model. These results highlight the limitations of closed-book reasoning and demonstrate the effectiveness of our interleaved search and reasoning approach in leveraging external knowledge.
Outperforms RAG baselines. EXSEARCH substantially outperforms strong RAG baselines, includ- ing both advanced RAG (e.g.. RankRAG) and iterative variants (c.g.. Search-ol). For example, on HotpotQA, improves Fl from 55.40 (RankRAG-709) and 53.31 (Search-ol 132BB to 62.59, using the 7B Qwen2.5 model. Even compared Search-R1, trained by the cost inst-nine PPO algorithm, our method achieves slightly higher performance (Avg. F1: 51.45 vs. 48.74) while using a much simpler self-improving loop based on the expectation-maxiine these improvements is that EXSEARCH enables the LLM to flexibly refine the query (thinking) and explicitly extract fine-grained evidence (recording), enabling more effective knowledge utilization.
framework. One reason for
Improves retrieval performance. To better understand the benefits of our agentic search method, we further evaluate the retrieval performance. We conduct experiments on four benchmarks and report recall scores in Table 2 which reflects how well the retrieved documents cover the ground truth. We follow prior work 36) and detine retrieved document as correct if it contains the answer to the input question. The results reveal several key findings: (i) using the retriever alone yields low recall; (ii) conventional re-ranking methods like MonoT5 show relatively limited improvement in multi- hop QA scenarios; and (iii) our method, through iterative reasoning, achieves substantially higher recall. These results indicate that EXSEARCH, by expanding the search reasoning exploration, can enhance retrieval performance in addition to improving end-to-end answer generation correctness.
5.2 Ablation studies
EXSEARCH consists of three core actions: thinking, search, and recording. To validate the effective- ness of each component, we independently remove each action and evaluate the resulting variant. Table presents the results for the following:
w/o thinking: Removing thinking reduces our method to a standard retrieve-then eerreeeethenn-eeeraate pipeline, leading to a substantial performance drop across all datasets. This validates the importance of reasoning and iterative query refinement in enabling more effective retrieval.
7

---

Qwen-2.5-3 73 on Trrining stt
Qwen-2.5 on Evaluxtion Set
Lama on Training
Llama 3.1 53 on Evaluntion Set
HotpotQQ
MusiQQee
2WikiQA
50
35
75
Exaact March
-0
Bxaet March
30
65
60
25
23
655
50
20
45
15
Triining Derstion
Thininn Iteration
Triniig Berstion
Thining Iteration
Figure 3: Training convergence of Qwen-2.5-7B and Llama-3.1 3.18-B.1-8B. where we report the Exact
Match score for checkpoints in each iteration.
(ii) w/o search: Without search, the method becomes equivalent to an iterative self-ask process without external knowledge. We observe consistent decline across all datasets, validating the necessity of integrating retrieval into the reasoning process.
iiii)
w/o recording:
Disabling recording prevents the model from extracting key evidence for each sub-query, resulting in average performance drop of approximately 8%, as shown in Table This suggests that deliberately and explicitly recording fine-grained evidence can enhance the model's understanding of external documents, which is consistent with prior findings in interpretability-fooul
studies, such as Physical LLMs [3]
(iv) w/o w=)): Removing the weighting function w(z) reduces our method to a naive self- training setup, where the model is directly trained on its generated data. Results in Table show substantial performance drop, e.g., the F1 score on the HotpotQA dataset drops from 60.72 to 54.45. This highlights the critical role of w(z) in guiding the model to prioritize trajectories that contribute to correct answers, rather than treating all sampled actions equally.
5.3 Training convergence
The theoretical analysis in $ demonsttattss the non-decreasing improvement property during our training process. To empirically verify this property, we evaluate model checkpoints after each training iteration on four experimental benchmarks. Figure shows shows the results, with more details provided in Appendix [.6] We observe that the performance on the training set consistently improves iterations and typically stabilizes, validating the expected non-decreasing convergence behavior. These findings align well with our theoretical expectations and further demonstrate the practical efficiency of our optimization strategy. On the evaluation sets, the model typically reaches peak performance within the second or third iteration, indicating rapid convergence in practice.
5.4 Investigating the cold start: Is more warm up necessary?
In our main experiments (Table D.
ExSEARCH is initialized with 1,000 syn- thetic examples for warm-up empirically, following prior work [IO] (6). To ex- plore the necessity and impact of warm-up data, we train separate models with varying amounts of supervised fine-tuning (SFT)
Exact Match Evalustion Set
Exact Mutch on Training Set
S1
Exarct Match
wi (600) cusplo sel Ko-ted comayla
K- ouzplo
36.
K-ste commeia
R. csnngles
Exscc
wilk 520 cumplees
K--to cuansier
K--to caameiir
data (denoted K) and apply EXSEARCH to each model independently. As shown in Figure we observe three key trends: (i) Warm-up helps: Initial training equips the model with basic reasoning capabilities and improves final performance; (ii) More data not always necessary: Qwen2.5-
K-iio somyyls
Keitnn
27
20-
complies (Zen)
Training Iteration
Training Iteration.
Figure 4: Performance for Qwen2.5-7B that is initially empowered by different amounts of warm-up data.
7B, even without any warm-up (zero), already outperforms strong baselines; (iii) 100 examples offer
a good trade-off: Using 100 SFT examples strikes a balance between performance and the cost of data synthesis (approximately 2$). We provide additional results and analysis in Appendix D.6

---

5.5
Case studies
To gain deeper insights, we conduct qualitative case studies using Qwen2.5-7B, with concrete cases provided in Appendix FE3 We observe two key strengths: (i) The model integrates reasoning and retrieval steps. dynamically selecting relevant evidence as part of its reasoning process; (ii) By interleaving reasoning with retrieval, our method helps the model ground its generated answers in factually relevant knowledge. We also manually examine 200 random failure cases and identify the following two errors: Under-searching: In 3.5% of cases, the model is misled by plausible-looking but incomplete evidence, retrieves fewer documents than needed, and stops too early; and Over- searching: In 7.5% of cases, the model retrieves more documents than necessary but still misses key information, failing to recognize when to stop. These observations highlight the need for better search-depth control and suggest improving trajectory-level stopping criteria in future work.
6 ExSEARCH-Zoo: An Extended Suite
So far, we have validated the effectiveness of EXSEARCH. The promising results in motivate us to extend our method more scenarios. We therefore introduce EXSEARCHI-Zo0, resource extending our method in two dimensions: diverse backbone models and extendable retrieval strategy.
Diverse backhone models. We apply EXSEARCH to wide range of LLMs and analyze the generalizability across LLMs of varying sizes and model families. Table and Figure ssoww the experimental results, which exhibit clear scaling-law pattern. In more detais, we highlight two key observations: (i) there is consistent performance improvement as the model size increases; and (ii) even models small 3B parameters achieve competitive performance when trained by our method. These suggest that EXSEARCH is broadly applicable and scales across different models.
Extended retrieval strategy. In EXSEARCH, the LLM explores a search trajectory by itera- tively thinking, searching, and recording. How- ever, to answer more complex queries, we may need customize a more specific retrieval strat- egy beyond these three actions. To demonstrate the extensibility, we introduce an example by adding additional document selection action, extending the retrieval process into the think search + select >> record pattern. Document selection allows the model to prioritize relevant evidence and discard distracting content, tech- nique widely used the IR field. Specifically, the LLM reads the retrieved documents and au- toregressively generates a ranked list of selected
Esac Maxt or Svalatio Sot
Aeraaa or Evalatio
$2.6510
54
St $5.
58.4
S1
Stamt Marth
24:
NQ
DerpongA Marque 2Wargg
12
MQ
Dorpott Munige
Oas Vazila
Oua w FEteesedReonnikng
Figure 5: Performance of EXSEARCH when ex- tended with the document re-ranking action.
identifiers (e.g.. [1] [2] [3]). similar list-wise re-ranking [51 67]. This variant modifies
only the E-step by adding the selection action to the trajectory, while the overall EM training remains self-improving process. Figure 5 shows consistent improvements, with accuracy gains of +1.1 and +1.5 on MuSiQue and 2WikiQA, respectively. These results suggest that more fine-grained customized actions can be seamlessly incorporated into EXSEARCH. A detailed derivation and experimental results are provided in Appendix
7 Related Work
Information retrieval with LLMs. As one of the most fundamental data mining techniques, information retrieval (IR) primarily focuses on understanding input queries, extracting relevant information from external data sources, and providing accurate responses to users [18] 34. recent years, large language models (LLMs), pre-trained on large-scale corpora, have been integrated into IR systems enhance conventional retrieval methods [5
leveraging their language processing abilities, such as LLM-based query rewriting. More recent work also uses LLMSs to aggregate long- context retrieved information and generate concise answers for users, a process known as retrieval- augmented generation (RAG) [15). However, most existing approaches integrate LLMs into IR by
77 8T

---

cascading traditional retrieval pipelines and optimizing LLMs for specific stages. For example, Self- RAG 4 fine-tune the LLMs by document relevance judgment and answer generation: ADEIE [52] fine-tunes LLMs for knowledge extraction; and other studies align LLMs for improved query decomposition 28 40). In contrast, we model the entire retrieval process through a dynamic and iterative agentic framework, where LLMS are seff-incentivized using the
algorithm. We also discuss further distinctions in Appendix B
Reasoning by LLMs. LLMs have demonstrated strong capabilities in long-term planning 8 86]. often decomposing complex tasks into sequence of sub-tasks and solving them step by step using their internal parametric knowledge
73
3. This ability, typically referred to as reasoning 50 53 79
or deliberate thinking !
29
85
underpins many recent advancements in LLM performance.
48 and DeepSeek-RI [17], which are post-trained using 62 have achieved impressive results on structured tasks like mathematical reasoning 41 and code generation [27 (55. However, LLMs suffer from outdated internal knowledge and the tendency to hallucinate factually incorrect content. Such foosed-book reasoning often underperforms in knowledge-intensiv inneee-- tasks like multi-hop question questtio
For example, models such as 4PT-4o
reinforcement learning (e.g.. PPO
60 or GRPO
answering. where models need access and integrate factual information [16, [3]. In this work, we tightly integrate retrieval with the LLM reasoning process. Unlike prior methods that directly augment LLMs with documents relevant to input queries [55] 74], our approach allows the model to acquire external evidence dynamically as reasoning unfolds and learns this pattern autonomously.
8 Conclusion and Future Work
In this work, propose EXSEARCH, a novel framework that empowers LLM to become an exploratory search agent, capable of iteratively reasoning over context, searching for target information, and reflecting retrieved documents. EXSEARCH incorporates self-incentivized loop, where the LLM learns to reason and search from its own generated data, progressively enhancing its expertise
effective search agent. We provide theoretical analysis of the advantages of EXSEARCH and conduct experiments on wide range of datasets to demonstrate its effectiveness. Motivated by its strong performance, we introduce EXSEARCH-ZoO, comprehensive resource that extends our vanilla method in two directions: (i) backbone LLM diversity and (ii) enriched action spaces (e.g., document re-ranking). Further experiments also validate the extensibility of our framework. We suggest future work to (i) extend EXSEARCH beyond text-only reasoning to include multi-modal reasoning with vision inputs, and (ii) apply EXSEARCH auglenugeetted agents, thereby supporting more real-world tasks.
Acknowledgements
This research was (partially) supported by the Dutch Research Council (NWO), under project 20. 183, and
numbers 024.004.023 NWA. 1389. 2889.20.11......
and the European Union under grant agreements No. 101070212 (FINDHR) and No. 101201510 (UNITE).
Views and opinions expressed are those of the authors only and do not necessarily reflect those of their respective employers, funders and/or granting authorities.
References
[1] Abdelrahman Abdallah and Adam Jatowt. Generraoo--etrivverreeennttooo approach for open- domain question answering. arXiv preprint arXiv:2307.11278, 2023.
[2] Omar Adjali. Olivier Ferret, Sahar Ghannay, and Hervé Le Borgne. Multi-level information retrieval augmented generation for knowledge-basse based visual question answering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.1433 /4316, 2023.
[4] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection.. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
10

---

[5] John Bibby. Axiomatisations of the average and a further generalisation of monotonic sequences. Glasgow Mathematical Journal, 1974.
[6] Donald Case and Lisa M Given. Looking for information: survey of research on information seeking, needs, and behavior. Emerald Group Publishing, 2016.
[7] Haolin Chen, Yihao Feng, Zuxin Liu, Weiran Yao, Akshara Prabhakar, Shelby Heinecke, Ricky Ho, Phil Mui, Silvio Savarese, Caiming Xiong, et al. Language models are hidden reasoners: Unlocking latent reasoning capabilities via self-rewarding. arXiv preprint arXiv:2411.0428. 2024.
[8] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09550 2025.
[9] Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint 22Xiv:2211.125885 2022.
[10] Yiqun Chen, Lingyong Yan, Weiwei Sun, Xinyu Ma, Yi Zhang, Shuaiqiang Wang, Dawei Yin, Yiming Yang, and Jiaxin Mao. Improving retrieval- augmenttd generation through multi-agent reinforcement learning. arXiv preprint arXiv:2501. 1,528,, 2025.
[11] Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models strong language models. arXiv preprint arXiv:2401.0133 2024.
[12] Peter Dayan and Geoffrey E. Hinton. Using expectation -itiittiiionmmiiiiazaaiiionon for reinforcement learning. Neural Computation, 1997.
[13] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21778 2024.
[14] Víctor Elvira and Luca Martino. Advances in importance sampling. arXiv preprint arXiv:2102.05407, 2021.
[15] Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491-6501, 2024.
[16) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-auge -agmenteed generation for large language models: survey. arXiv preprint ar 10997, 2023.
[17] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-ri. Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 1ttdi:::5001129444 2025.
[18] Kailash A Hambarde and Hugo Proenca. Information retrieval: recent advances and beyond. IEEE Access, 11:76581-76604, 2023.
[19] Peter Henderson, Joshua Romoff, and Joelle Pineau. Where did my optimum go?: An em- pirical analysis of gradient descent optimization in policy gradient methods. arXiv preprint arXiv:1810.00.52. 2018
[20] Xanh Ho, multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, 2020.
Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing
[21] Jian Hu, Siyang Jiang, Seth Austin Harding, Haibin Wu, and Shih-wei Liao. Rethinking the implementation tricks and monotonicity constraint in cooperative multi-agent reinforcement learning, arXiv preprint arXiv:2102.03479, 2021.

---

[22] Lei Huang, Weijiang Yu. Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 2023.
[23] Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In The ICLR Blog Truck 2023, 2022.
[24] Yifan Ji, Zhipeng Xu, Zhenghao Liu, Yukun Yan, Shi Yu, Yishan Li, Zhiyuan Liu, Yu Gu, Ge Yu, and Maosong Sun. Learning more effective representations for dense retrieval through deliberate thinking before search. arXiv preprint arXiv:2502. 129744 2025.
[25] Albert Q Jiang. Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04000 2024.
[26] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, ArXiv, 2023.
[27] Juyong Jiang. Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint Stti::2000.0051133 2024.
[28] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Iims to reason and leverage search engines with reinforcement learning. arXiv preprins arX3v:2503.09516 2025.
[29] Daniel Kahneman. Thinking, fast and slow. Farrar, Straus and Giroux, New York, 2011.
[30] ScongKu Kang, Bowen Jin, Wonbin Kweon, Yu Zhang, Dongha Lee, Jiawei Han, and Hwanjo Yu. Improving scientific document retrieval with concept coverage-based query set generation. In Proceedings of the Eighteenth ACM International Conference Web Search and Data Mining, 2025.
[31] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, 2020.
[32] O. Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, et al. Dspy: Compiling declarative language model calls into state-of-the-art pipelines. In International Conference on Learning Representations, 2024.
[33] Takyoung Kim, Kyungjae Lee, Young Rok Jang, Ji Yong Cho, Gangwoo Kim, Minseok Cho. and Moontae Lee. Learning to explore and select for coverage strieval-augmeen generation. arXiv preprint arXiv:2407.0113 051888 2024.
[34] Mei Kobayashi and Koichi Takeda. Information retrieval on the web. ACM computing surveys (CSUR), 2000.
[35] Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! In The International Conference on Learning Representations, 2019.
[36] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 2019.
[37] Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, et al. Retrieval- generation for snowledge-inteniin nip tasks. In Proceedings of the 34th International Confer- ence on Neural Information Processing Systems, 2020.
12

---

[38] Xiaoxi Li. Zhicheng Dou, Yujia Zhou, and Fangchao Liu. Corpusim: Towards unified language model on corpus for knowledge-intensinn tasks. In Proceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, 2024.
[39] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366, 2025.
[40] Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Hao Chen, Ge Yu, Zhiyuan Liu, et al. Rag-ddr: Optimizing aarmenel-augmented generation using differen- tiable data rewards. arXiv preprini arXiv:2410. 13509, 2024.
[41] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl. arXiv preprint arXiv:2503.23383, 2025.
[42] Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. arXiv preprint vrXiv:2401.1022. 2024.
[43] Xinbei Ma, Yeyun Gong, Pengeheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval- augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[44] Gary Marchionini. Exploratory search: from finding to understanding, Commun. ACM, 49.
2006. ISSN 0001-0782.
[45] Todd K Moon. The expecaaionnmaxxiiizatoo algorithm. IEEE Signal processing magazine, 1996.
[46] Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with pretrained sequence-to-sqqeeee model. In Findings of the Association for Computational Linguistics: EMNLP 2020, 2020.
[47] OpenAl. Introducing ChatGPT, 2022.
[48] OpenAl. GPT-4, 2023.
[49] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. KILT: benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
[50] Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint arXiv:2407. / 2024,
[51] Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankvicuna: Zero-shot listwise document reranking with open-source source large language models. arXiv preprint arXiv:2309.1508S 2023.
[52] Yunjia Qi, Hao Peng, Xiaozhi Wang. Bin Xu, Lei Hou, and Juanzi Li. Adelie: Aligning large language models on information extraction. arXiv preprint arXiv:2405.05000 2024.
[53] Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey. arXiv preprint arXiv:22/2.09599, 2022.
[54] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024.
[55] Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context retrieval-augmennee language models. Transactions of the Association for Computational Linguistics, 11, 2023.
13

---

[56] Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020.
[57] Ruiyang Ren, Yuhao Wang, Yingqi Qu, Wayne Xin Zhao, Jing Liu, Hao Tian, Hua Wu, Ji-Rong Wen, and Haifeng Wang. Investigating the factual knowledge boundary of large language models with retrieval augmentation. In ACL. 2023.
[58] Zhaochun Ren, Zhi Tian, Dongdong Li, Pengjie Ren, Liu Yang, Xin Xin, Huasheng Liang, Maarten de Rijke, and Zhumin Chen. Variational reasoning about user preferences for conversa- tional recommendation. In proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval, 2022.
[59] Keshav Santhanam, O. Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. In North American Chapter of the Association for Computational Linguistics, 2021.
[60] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06444 2017.
[61] Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retreetal-augmenteed large language models with iterative retrieval-genen generratonn synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, 2023.
[62] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
[63] Zhengliang Shi, Weiwei Sun, Shuo Zhang, Zhen Zhang, Pengjie Ren, and Zhaochun Ren. Rade: Reference-assisted dialogue evaluation for open-domain dialogue, ArXiv, 2023.
[64] Zhengliang Shi, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. seneratt-then-ground in atrieval-augmenter generation for multi-hop question answering. arXiv preprint arXiv:2406. 14891,
[65] Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J. Liu, James Harrison, Jaehoon Lee. Kelvin Xu, Aaron T Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. Trans. Mach. Learn. Res., 2023.
[66] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. RI Rl-searhher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.08 05592, 2025.
[67] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[68] Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, 2024.
[69] Surya T Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics, 2, 2010.
[70] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. In Transactions of the Association for Compulational Linguistics: TACI., 2022.
[71] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with -fain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, 2023.
14

---

[72] Shuting Wang, Xin Yu, Mang Wang, Weipeng Chen, Yutao Zhu, and Zhicheng Dou. Richrag: Crafting rich responses for multi-faceted queries in strieval-auggeent generation. arXiv preprini arXiv:2406. 12566, 2024.
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. ArXiv, 2022.
[74] Zhepei Wei, Wei-Lin Chen, and Yu Meng. InstructRAG: Instructing retrieval- -aggmentdd generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representations, 2025.
[75] Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with -eet------ arXiv preprint wrXiv:2407.19544 2024.
[76] Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023.
[77] Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. Large language models for generative information extraction: A survey. Frontiers of Computer Science, 18(6): 18637,,
[78] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving compression and selective augmentation. arXiv preprint urXiv:23/0.04408, 2023.
Ims with
[79] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang. Yunke Zhang. Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng. al. Towards large reasoning models: A survey of reinforced reasoning large language models. arXiv preprint XXiv:2501.09686 2025.
[80] Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Search-in in-the-chain: Towards accurate, credible and traceable large language models for knowledge nnowledgeinntessiee tasks. In wwW, 2024.
[81] Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou. Unsupervised information refinement training of large language models for generation. arXiv preprint arXiv:2402.18150 2024.
[82] Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. arXiv preprint 2rXiv:2401.1588. 2024.
[83] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.151 15115, 2024.
[84] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018.
[85] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36, 2023.
[86] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long thain-of-thought reasoning 1lms. arXiv preprint arXiv:2502.03373, 2025.
[87] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making aegienel-augmmented language models robust to irrelevant context. arXiv preprint arXiv:2110.01558 20233
[88] Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. Rankrag: Unifying context ranking with retrieval-augmented generation in Mms.. arXiv preprint 47. 2024.
15

---

[89] Zhenrui Yue, Huimin Zeng, Lanyu Shang. Yifan Liu, Yang Zhang, and Dong Wang. Retrieval augmented fact verification by synthesizing contrastive arguments. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
[90] Hamed Zamani and Michael Bendersky. Stochastic rag: End-to-end retrieval- drrieval- generation through expected utility maximization. Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, 2024.
[91] Junzi Zhang, Jongho Kim, Brendan O' D'Dongguu, and Stephen Boyd. Sample efficient reinforce- ment learning with reinforce. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[92] Weinan Zhang, Junwei Liao, Ning Li, Kounianhua Du, and Jianghao Lin. Agentic information retrieval. arXiv preprint arXiv:2410.09713,, 2024.
[93] Xiaoyu Zhang, Xin Xin, Dongdong Li, Wenxuan Liu, Pengjie Ren, Zhumin Chen, Jun Ma, and Zhaochun Ren. Variational reasoning over incomplete knowledge graphs for conversational recommendation. In Proceedings of the Sixteenth ACM International Conference Web Search and Data Mining, 2023.
[94] Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. Vedify-and-edit: A knowledge-enhanced chain-of-though framework. arXiv preprint arXiv:2305.02200 2023.
[95] Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, Benjamin Negrevergne, and Gabriel Synnaeve. What makes large language models reason in (multi-turn) code generation? arXiv preprint arXiv:2410.08105, 2024.
[96] Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, and Lei Sha. Atm: Adversarial tuning multi- agent system makes robust retrieval-ane eerienel-augmeented generator. arXiv preprint arXiv:2403 18111, 2024.
16

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have provided detailed abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitation of this work.
17

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
We have demonstrated the full proof in 'Method' Section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have reported the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We have released the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cc)
hiaiic/guuiess/oeeeuubiisssss wonoooiiissiooooiccc for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
/ //nips.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
19

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
The full settings are provided in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not conduct statistical significance tests. (However, we experiment with different settings for some hyperparameters and find the results to be relatively stable. See Appendix for details.)
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have reported the detailed implementation.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: More details are provided in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: See Appendix for more details.
21

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets (datasets, code, large language models) used in this paper are publicly available and properly cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswil thcode. thicode cwm/catoottotoo datassts has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
The code and data in this work are well documented in open-source Github.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
22

---

Answer:
[No]
Justification:
for We have no crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [No]
Justification:
This work focuses on large language models instead of human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: LLM is used only for grammar refinement.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non stansadd components.
Please refer LLM policy what should or should not be described.
23

---

Appendix
Contents
A
Practical Considerations and Societal Impact
24
Limitations
24
A.2 Ethics statement
25
A.3 Societal impacts
25
B Comparison with Prior Work
26
Comparing with previous iterative retricval method
................................
26
Comparison with previous telf-trainnng methods
................................
26
Detailed Theoretical Analysis
27
Skeleton derivations for EXSEARCH
................................
27
Convergence analysis
................................
28
Rethinking what the LLM learns within EXSBARCH
29
D
More Experimental Details
33
D.
Experimental datasets
33
D.2
Evaluation metrics
33
D.3
Baselines]
33
D.4
Data collection for warm- Up
34
D.3
Implementation details and hyperparameters
36
D.6
Supplementary experimental results
38
EXSEARCH-Zoo: Extending EXSEARCH Diverse Scenarios
Diverse model families and seales
42
Extended retrieval strategy
F
Prompt and Case Study
43
F.1
System prompl EXSEARCH
................................
43
F2
Human evaluation
44
F3
Case studies
45
A Practical Considerations and Societal Impact
A.1 Limitations
Despite our efforts to improve the LLM's ability to reason and search, our work has several limitations. Some of these limitations are shared across many existing RAG systems. We highlight them only as directions for future work but also in the hope of inspiring broader exploration within the community to advance the development of more effective retrieval-augmentte reasoning systems.
First, our method currently focuses on textual inputs and outputs. Extending search multimodal scenarios, e.g., incorporating images structured tables, remains important direction for future work. We plan to integrate EXSEARCH with large vision-language models and extend the text-only retrieval documents
multimodal corpus.
Second, in line with prior work, we deliberately avoid hardcoding heuristics such as fixed query decomposition rules predefined in-context learning demonstrations. However, similar to most existing approaches, our model performs retrieval at every reasoning step, regardless of necessity. Since LLMs pre-trained large web data have parameterized extensive world knowledge, for some

---

simpler queries, this fixed retrieval strategy may be redundant. An important next step is to develop more adaptive strategies that allow the model to decide when retrieve based on context, rather than always retrieving at each step.
Third, our training and evaluation are based on rule-based, answer-centric metrics such Exact Match and Fl score. While effective for benchmark evaluation, these metrics may not fully capture performance in more open-ended exploratory tasks, especially for long-form generation. As mentioned in our paper, in future work, we aim to explore more open-domain setups and alternative supervision signals beyond standdt answers, such as LLM-as MaMM----the-juugee
Overall, the proposed EXSEARCH makes progress in the RAG research area by effectively teaching LLMs to interleave reasoning and search within self improving process. However, it remains an initial step. We helieve addressing the above limitations will be crucial for building more general and robust search-augmented reasoning systems.
A.2 Ethics statement
The research conducted in this paper centers around the development of search framework. The proposed method enables language models to dynamically retrieve and reason over external information. In the process of conducting this research, we have adhered to ethical standards to ensure the integrity and validity of our work. All the questions used in this study were obtained from existing benchmarks, ensuring high level of transparency and reproducibility in our experimental procedure. To support our retrieval system, we used an open spen-source corpus, specifically Wikipedia. This ensures that our research utilizes publicly accessible and freely available data, minimizing potential bias and promoting fairness.
We have made every effort to ensure that our study does not involve human subjects, private data, or any content that may cause harm to individuals or social groups. No part of this work includes deceptive practices or intentional misuse of information. We are committed to conducting and presenting this research with integrity and social responsibility. We intend to release our code and implementation details to support open research, following the NeurIPS submission policy, and aim to facilitate further study in the information retrieval and retricval- -ugmeeende generation arcas.
A.3 Societal impacts
Our work introduces EXSEARCH, a maasoning-augmenter search framework that interleaves multi- step reasoning with dynamic document retrieval. The primary goal is to improve the factual accuracy and interpretability of LLMs open-domain question answering and knowledge-intensive tasks. The model presented in this work requires explicit grounding in external sources, and our framework emphasizes verifiable retrieval and intermediate reasoning steps, which improve transparency. These properties support downstream mitigation efforts, such traceable generation or retrieval auditing. Additionally, we note that our current implementation does not involve user data or personalized profiles, thus mitigating privacy concerns.
This approach offers several key benefits. Improved Transparency: By grounding answers in external sources and providing intermediate reasoning steps, EXSEARCH allows for better traceability of the model's eecrisionmmaking process, which is crucial for understanding and verifying outputs; Enhanced Trustworthiness: The framework's 's reliance on verifiable external retrieval reduces the risk of generating hallucinated information, contributing to more reliable and factually accurate responses; and Broader Applicability: With its focus on factual grounding and reasoning, EXSEARCH can be applied to a wide range of mnowledge-intensing applications, including scientific research, legal analysis, and education, where accuracy and clarity are essential.
While this research foundational and not tied to specific applications or deployments, we ac- knowledge the broader risks associated with enhancing the factual accuracy and coherence of LLMs. Specifically, the improved text generation capabilities of LLMSs may be misused in malicious contexts or scenarios, such as: (i) generating more convincing disinformation; (ii) fabricating plausible but incorrect content by selectively retrieving or combining real documents; and (iii) enabling better- targeted persuasive text (e.g., phishing, political propaganda). In summary, while EXSEARCH could be misused in unintended settings, its design principles, combined with the potential for verifiable and auditable generation, offer avenues for responsible deployment. While we do not foresee imme-
25

---

diate or direct societal harm, we encourage future work to explore safeguards, such as automated monitoring and ethical auditing mechanisms, high-stakes applications.
B Comparison with Prior Work
In this work, the proposed method enables L.MSs to perform interleaved reasoning and search (also referred to as agentic search in this work) and optimizes their capability for this pattern through a self-incentivized framework. Below, we compare our method with previous iterative retrieval methods and self-learning methods in detail.
B.1 Comparing with previous iterative retrieval method
In this paragraph, we systematically compare EXSEARCH with previous work that incorporates iterative retrieval techniques, especially those integrated with LLMs, highlighting our fundamental innovations. Most previous iterative retrieval approaches rely on a modular multi-stage pipeline. where distinct models are trained separately for sub-tasks such as query rewriting, retrieval, and evidence aggregation. For example, For example, Self-RAG 4 and RankRAG [8 train LLMs for document relevance judgment; ADELIE [52] adopts LLMs for knowledge extraction; while other work 2 4 cascades multiple specialized components in sequence. Despite their progress, this modular design overlooks the end-to-end optimization, leading to potential misalignment among the training objectives in different stages. More recently, several works attempt to prompt powerful, extremely large models (i.e., GPT-3355 or Qwen-32B) to iteratively interact with retrieval engines through in-context learning [39]6]] GT.. However, they are limited by using predefined demonstrations in-context learning examples. While simplifying system design, these approaches overlook improving the LLM's intrinsic reasoning capability for adaptive retrieval and generation. In contrast, EXSEARCH trains a unified LLM to reason over the evolving context, adaptively decide what information to retrieve, optionally re-rank retrieved documents, extract fine-grained supporting evidence, and generate the final answer, all within end-to-ent learning framework. Unlike previous work, EXSEARCH aligns the training signals of these actions through a coherent learning objective, improving the LLM in an end-to-end manner.
The most contemporary work to EXSEARCH Search-RI (28), developed independently around the same time. Search-R1 applies the proximal policy optimization (PPO (60) algorithm to en- courage LLMs to issue multiple search queries during reasoning. However, EXSEARCH differs from Search-RI in three critical aspects: (i) Reasoning Process: EXSEARCH enables richer retrieval actions, including query generation, optional re-ranking of retrieved documents, and fine-grained evidence extraction, while Search-RI only conducts iterative query decomposition without reflection or re- re-ranking; (ii) Training Algorithm: EXSEARCH treats search trajectories as latent variables and optimizes a variational evidence lower bound via Generalized Expectation- Sipectation-MMxximizztoiii algo- contrast, Search-R R1 adopts an online on-policy reinforcement learning framework, which often suffers from sample inefficiency and training instability [19 21 23); and (iii) Reward Signal: EXSEARCH introduces a trajectory- level training signal, evaluating the quality of the entire search trajectory based the likelihood of generating the correct answer, rather than relying solely on binary antcome-based reward as in Search-RI.
rithm 45 achieving stable and progressive self-improvemmnn.
B.2 Comparison with previous self-training methods
Recent studies have explored self-training frameworks where a model is iteratively trained on its generated data 75 For example, some work 7 65 allows the model to first generate solution and fine-tune it on the generated trajectories, showing promising results on mathematical reasoning tasks. Similar ideas have also been applied in combination with REINFORCE Leave-One-On
(RLOO) methods 7 35 91. More recently, other work [1] 75 proposes self-rewarding methods, where the LLM itself is used via LLM-as-a-Judge prompting to provide its own rewards during an iterative DPO 54 training process. While effective in closed-book settings, these approaches overlook a critical limitation: the inherent limitation of parametric knowledge in LLMS [15] [6]. Without external retrieval, the model cannot dynamically access relevant information when it is missing from its internal memory. contrast, this work focuses on agentic search, which aims to enable the LLM to interleave dynamic retrieval within the reasoning process and further reflect on the
26

---

Table 4: Main notation used in this work.
Symbol
Description
The initial input query.
The ground-truth truth answer corresponding to the initial query
The parameters of the large language model (LLM).
R
The external retriever.
The index of the i-th step in the reasoning and retrieval process. The sub-query generated at the i-th step.
The set of documents retrieved in the i-th step, i.e., R(x;) (d.,) / [K]}. The fine-grained evidence extracted from the retrieved documents d,.
The index of the t-th training iteration.
The full reasoning trajectory, consisting of interleaved sub-queries, retrieved documents, and extracted evidence, i.e., 2 {(T,, e,) [|=[]]
retrieved content at a fine-grained level (See the $ 2.1 in the main body of the paper). Additionally, provide theoretically grounded analysis of the advantages of our method, as discussed in this Appendix C.2 and Appendix C We also introduce an extended resource, EXSEARCH-ZON which supports multiple model families and richer reasoning actions.
C Detailed Theoretical Analysis
Due to space constraints in the main body of this paper, we include the detailed version of the theoretical derivations and analyses of EXSEARCH here. Below, first formulate the search and reasoning process in EXŠEARCI. We then introduce how the generalized expeccaionn-maximizatoo technique leveraged improve the LLM's capability in EXSEARCH through self-improving loop (& &C..T. Additionally, we prove that the resulting training procedure is convergent. Table lists the main notation used in the paper.
C.1 Skeleton derivations for EXSEARCH
Reviewing agentic search procedure. In this work, the proposed EXSEARCH is inspired by the Exploratory Search paradigm [44), which models information-seeking dynamically unfolding process where search queries are iteratively refined based on intermediate results. ExSEARCH simulates this by interleaving three core actions:
thinking: The LLM generates a query æ; based on the current context x and the accumulated search trajectory z<i. formulated
p(z: x, z<x; 0)
(10)
ii search: A retrieval module R retrieves the top- K documents d; relevant the query 24;
d, = R(x;)
(11)
(iii) recording: The LLM reflects on the retrieved documents d, and extracts evidence e; conditioned on ø and d,:
-= d;;
(12)
In this step, the model focuses solely on the current sub-query and its associated documents, reducing computational cost by limiting the context the most relevant information.
Formally, the reasoning-an augrmente search process modeled as a sequence z = =(x;; di 6x) ii
[|z[]), with the joint likelihood:
p(= x; 0)
II p (1r, ei) xx, Z2;;0)
(13)
After the interleaved search and reasoning process, the LLM aggregates information from z to generate the final answer y p(y 2;
27

---

Training objective. The goal of EXSEARCH is to improve the LLM's ability to generate the correct answer y after reasoning. The training objective is formulated as:
log
0) log Σ. ply+ z (2:)
(14)
Here, 2 = [1T e;) i E [|=|]] represents sequence of the thinking, search, and recording
actions. In EXSEARCH, introduce proposal distribution g(z æ) approximate the sampling space of x and apply Jensen's inequality to the marginal log- likeliihoood in Eq. 14
log Σ, g(2
p
x; 0)
<(2
x))
P(U. 2
x;; 0)
{(z log
log
(15)
q(
(x)
p(y:
ax; 0)
log
x))
The right-hand side represents the variational evidence lower bound (ELBO) of logr(y x; 0),
and the bound becomes tight if q(= x) approximates the true posterior distribution p(z y, y, 0). Thus, we can iteratively estimate p(= y, () and maximize this ELBO using the generalized expectation-maximiztiin algorithm to progressively improve the LLM.
E-step: Trajectory exploration. In the E-step, we estimate the distribution over reasoning tra- jectories by sampling from the LLM 0. the 1-th iteration, we approximate the distribution 1(2 x)) Rs p(= x, y; 0). yielding the following form for the ELBO:
ELBO =
Eg-p(ela..y00) (log((y= x; 0)] + H(p(= x, v2,;("))
(16)
Here, the entropy term H(p(2 x. y;; ()) is constant with respect to 8. Since direct sampling from the posterior is intractable, apply importance sampling [14 69], where the distribution p(z x;; 0²) is casier to sample from, and each sample is assigned an importance weight:
p(z x. yı 0*)
w(x)
(17)
p(= x;; 0*)
This allows to rewrite the ELBO as:
ELBO = Ex-p(=1=;00) log x; 0; + e,
(18)
where is constant.
M-step: Re-weighted trajectory learning. In the M-step, we update the model parameters by maximizing the ELBO from Eq. [8 The objective becomes:
0 arg max Ez=p(=)2;0)) (wxx) log z æ; 0)]
(19)
The overall training process is performed using stochastic gradient descent, with gradicnts computed as:
V voELBO( = - -E (u())VoC + Call
(20)
C.2 Convergence analysis
Below, we analyze the convergence behavior of EXSEARCH using the generalized expectation- maximization algorithm, providing more detailed explanation than in the main body of the paper. We show that the training objective logp(v x;; 0) is non-decreasing after each training iteration and progressively converges to stationary point due to its upper-bounded property. To provide tighter characterization of convergence, interpret the optimization gap a KL divergence between importance- weeighte sampling and the true posterior.
Lemma C.1 (Monotonic improvement). each iteration t Zt, the training objective of the LLM satisfies:
log ply x;; ((+1) 2 log xx; xx;0'.
(21)
28

---

Proof. Reviewing the EM-style training in our method, the main concept involves introducing tractable evidence lower bound (ELBO) and progressively improving optimize log p(y) z; 0). In
the E-step of the 1-th iteration, we sample trajectories from the current model as ~ p(= x;; 0*)
and assign each a weight (x)) = p(=(a;01) & ply I, 2; 8*). We define the ELBO as:
p(=l=-y)0')
ELBO 0*) == E ===((=);))) (w)) log p(y, = a;; 0)] + c. In the M-step, update the model by maximizing this ELBO:
(22)
g++1 = arg max ELBO
(23)
0
which guarantees that: ELBO 8*)
ELBO(6",0') Meanwhile, since the ELBO indi-
cates the evidence lower bound for the marginal distribution logp(y 2; 1; it holds that:
log ply x;; 0++ 2 ELBO(0++1 0²), and ELBOO 0²) logply x;; 0*). By combining these two equations, we have:
logply x; 0 log p(y x;;0).
(24)
Therefore, we have completed the proof of non non-decreasing improvement for each training iteration. Lemma C.2 (Boundedness). The sequence {log 2;; {-) is upper-bounded.
Proof. Since plv | x; E [0, 1]. we naturally have logp(y ply
x; < 0 for all 0.
Theorem C.3 (Convergence of EXSEARCH). By Lemma C.I and Lemma C.2 the sequence (logply x; k²,0%} is non-decreasing and upper-bounded. By the Monotone Convergence Theorem 15), converges a finite limit.
Remark C.1 (Tightness via KL Divergence). The ELBO can be interpreted as a tight bound of log p(y x;; 0) with the following identity:
logp(y x;; 0) ELBO 0') + KL(A"()) l p(z 0),;
(25)
where (2) x wr(2) pp(z x:; 0*) the induced sampling distribution. Therefore, under mild regularity conditions (e.g., bounded support, continuity of log-likelhhood) as 4" approaches the true posterior, the KI. term vanishes, and the ELBO becomes tight. This strengthens the convergence result and characterizes the optimization gap.
C.3 Rethinking what the LLM learns within EXSEARCH
In C.I and main body of our paper, we demonstrate that the answer log likelihood (i.e., p(y|æ, z; 8)) serves as an end-to-end training signal in our self-incentivized training process, guiding the model to reason and search. Given the primary goal of information retrieval, especially in downstream tasks like sarrivall-augeentee users and evaluate its performance using correctness-orienteed metrics, such exact match score accuracy, thereby ensuring the factuality. However, a natural question arises:
generation, we typically train the model to generate accurate answers for
Does our metrics, such as accuracy, and why does it work or not?
framework also maximize the commonly used downstream
In addition to the strong empirical results shown in our experiments, we provide a more interpretable and theoretical analysis in this section. Below, first briefly highlight the learning objective introduced in $ C denoted as the vanilla objective, which uses w(z) o p(y x, 2; 0) as the training signal. We then introduce goal-oriented learning objective, where the LLM is trained to maximize an evaluation metric using a similar Expectation-Muximizaee these two objectives and show their consistency in model training and optimization, illustrating why the vanilla objective aligns with maximizing the expected metric and downstream task performance.
algorithm. Finally, we relate
C.3.1 Reviewing vanilla learning objective
The vanilla learning objective is defined as ELBO = =z=p(()2;00) (og x;; 0)). Here, w(2) is the weighting function given by w(z) = p(= x.yr0')
p(z p(=|=;0*) This weighting function is derived from the ratio of the posterior distribution p(= x. 0*) ) to the prior p(z x;; 0'), which reflects how well
the trajectory supports the final answer y.
29

---

C.3.2 Introducing goal-oriented objective
We now replace the weighting function w(z) with evaluation metric r(y) (also widely known the reward function), which evaluates the quality of the final answer y. Formally, given an evaluation metric r(y) that rates the quality of answer y, goal is to optimize the model parameters @ to improve the expected performance. We define the expected learning objective under the model as:
[()) = Eyr-p( ( = Σ r(v)p(y x)
(26)
In EXSEARCH, since the model first generates reasoning trajectory and then outputs an answer y, we can rewrite Eq. 26 as:
( = E grrpl 00 [r[u)] = Er(v) r(y) Σ E(z,vl x))
Ir(v)((, 1 1
(27)
2,9
Here, p(y, 2 x) denotes the LLM generating a reasoning path x followed by a final answer y. Marginalizing over all possible (z, y) is typically intractable due to the large action space of the LLM. We now derive a variational surrogate for optimizing such goal-oriented objective through a tractable lower bound.
Proposition 1 (Variational lower bound proxy for metric maximization). Given non-negative evaluation metric function r, let I(0) == E = u|=;)) [r(y] be the expected metric. We can introduce a proposal distribution g(z,)) y) over the (z, u) space to construct more tractable evidence lower bound:
ELBO q) = Eg(z, y x)) log r(y)p(z. y x;
(28)
g(2,3 x)
2.9
which is a function only related to q and 0. satisfies:
arg max ELBO arg max [(0) S c. (KL(q po))
(29)
B
8
where is constant. Under mild assumptions, the boundedness (=) holds when q" R2 r- Po-
We provide the detailed proof for this proposition in $ C33.5 This proposition indicates that we can
optimize the ELBO proxy improve the (0),, following similar Sopeecation-Maximizatte algorithm as introduced in vanilla EXSEARCH (& C.I. In more details, this involves alternating between the following steps: (i) E-step: sampling (z, y) trajectories; and (ii) M-step: updating the model parameters.
E-step: Sampling trajectories. To approximate the true posterior distribution q", we sample (z, y) from the current model p(z, y importance weight is formulated as pla.y = = r(y), which is obtained using the evaluation metric.
x;; 8*) using importance sampling strategy. The corresponding ("(ww(l))
ELBO (( (lgg
x; 0)] +
9" (2
E(s.))
log log pepp(z)) x; +
(30)
p(z.3 z; 0t)
E(e.))
(r()) logp(z,y x: 0]] +
M-step: Update the model parameters. the weighted lieliiiioooo::
Given the weighted samples, we update 0 by maximizing
0 arg max (oy log) logp(z,y 1x;;) x;;
(31)
8
By alternating between the above E-step and M-step, we can train the LLM to maximize the given metric. This can be seen reward- reward-weighted generalization of expectation-maxiiizan [12). This formulation naturally integrates downstream evaluation metric (via r(y)) into a likelihood-based training framework.
30

---

Exact Match in Evaluation Set
Accuracy in Evaluation Set
51
55
Exact Match
47
51
Accuracy
43
47
Vanilla
Vamilla
39
Exact Match
Exact Match
43
Accuracy
Accuracy
2
Training Iteration
Training Iteration
Figure 6: Comparing vanilla training process in our method with variants trained using Exact Match or Accuracy the training signal.
C.3.3 Relating vanilla learning objective with goal-oriented objective
In the goal-oriented objective, the evaluation metric r(y) directly evaluates the quality the answer y return training signal. Similarly, the training signal w(() p(y x., 2; represents the probability of generating a correct answer, and the higher the probability of generating the correct answer, the greater the expected metric. Therefore, have w((2) x r(y).
Reviewing the optimization objectives in the vanilla objective and the goal-oriented objective: vanilla objective, we have 0 = arg maxe [=<p(=(x=00) [w[2) log p(u, x;; 0)]. In the goal-oriented objective, we have Ô arg max, J (0) arg maxę (z,,))>(( 01) (r(()) log p(z, x;; 0)]. Thus, when w(z) x rly). we have 0 x= 0. That is, under mild assumptions, the two optimization objectives are theoretically equivalent.
In summary, for a non-negative metric or reward r(-) that encourages the model to generate high- quality, correct answers, training with the vanilla objective also maximizes that the evaluation metric in downstream tasks.
C.3.4 Experimental results
To validate the thcoretical analysis presented above, we implement the goal-oriented objective to train the LLM. Following prior work, such as DeepSeck-B R [17], we adopt two rule-based metries, namely Exact Match (EM) and Accuracy (Acc.), as the function r(y) in Proposition]] [ These metrics are commonly used in open-domain QA and provide direct, interpretable supervision.
Figure(6] the performance of models trained using both the vanilla objective and the extended gral-oriented objective. We observe that models trained with different objectives (e.g., answer log-likellng ligllikel specific metrics) converge to similar performances instance, the curves labeled vanilla and w/ Exact Match on the left side of Figure booth achieve score of 50 in exact match after 5 iterations of training. A similar trend can also he found on the right side of Figure 6 where both the curves labeled vanilla and w/ Accuracy achieve a score of in accuracy. This observation supports and aligns with the thcoretical analysis presented above.
the corresponding metric. For
Additionally, we observe that the model trained using the vanilla answer likeeiihd converges faster and achieves the best average performance across both exact match and accuracy metrics. We analyze that there are two potential reasons. First, the p(y) z, 2; f) is proportional to exact match and accuracy, which enhances the probability of generating correct answers, thus improving both metrics. Second, compared to rule-based metric that only evaluate the final result, ply z, æ;; 0) provides denser and continuous signal. This offers more effective guidance on how intermediate steps in the reasoning process influence the final outcome, allowing the model fun-tune its reasoning trajectory toward the optimal solution.
31

---

C.3.5 Detailed derivation for Proposition 1
Let r(v) 2 0 be an evaluation metric, and define the expected objective as [(0) = [(xw))p(= w/: [r()]] For any proposal distribution q( y) over reasoning-answe trajectories, we can construct tractable evidence lower bound (ELBO) as:
ELBO(O, = Σ u(2.y ax) log r(v)p(z. y |x;0)
(32)
((2 x)
To prove Proposition ! we show that there exists a constant c > such that:
arg max ELBO g) arg max [(0] S c. (KL(q r po))
(33)
where r. pe denotes the unnormalized target distribution ("(z,0) v) x r(u)p(=, y 1;
Proof. We first apply Jensen's inequality to construct the ELBO:
T(0) = Σ r(y)p(z,y x; = E(2z.8 rl))p(z,3 2; q(z, y x)
(34)
Eg(2. x)) log (y)p(zz, y 2; ((2 y xx)
2 exp
exp exp(OOLO()
2,y
Taking logarithms (which preserves ordering due to its monotonicity), ELBO(6. q) and note that arg max, J(0) arg max, log J(0) due to the monotonicity of log(-). To relate the maximization of ELBO(0, q) and [()), we begin by denoting 8
obtain log (0)
arg max, log [() arg J(0) and == arg max ELBO q). We aim bound the dis- tance ||0- - 01|-
We assume the objective function log [(0) is L-smooth, i..., it has L-Lipschitz continuous gradients This implies that for any 01,02, .2:
T. [10: 1-
log I [(1) S log ((2) + log ((02) 02) 2
(35)
Applying this inequality with 01 H" and 0₂ 0, and using the fact that VELBO the maximizer 0, we can write:
L 1100 =
log I [00) < log [(0) + log J [0) (0
0)+
0112
(36)
L 1 01/
S log J(0)
where the last inequality assumes that the inner product term vanishes at first-order optimality. We now relate log J (0) and ELBO(0, q) via:
log (() ELBO q) KL || r(y) p(z, v x;; 0)) + log
(37)
where 2 2= r(u)(2() 1; 0) [(), so the gap equals:
log ((0 EL q) = (L())l
(38)
with q" (z,y) x r(y)p(2,)) x; 0).
Putting it all together, we can obtain the following equation:
- - s L -110- ô11 + KL(alla")
log
(39)
2
By rearranging this, we have:
|10~ - 0 2 L
2 -
2 S -K((lIrr-p))
(40)
32

---

Taking square roots gives:
100 - 011 S c. (KL(q]Ir pp))
(41)
where V2: This completes the proof.
In summary, this proposition suggests that we can maximize ELBO as a surrogate for J. Below, we examine when this bound is tight and show that optimizing ELBO under specific posterior q" is equivalent to maximizing the expected objective.
Lemma C.4 (Tightness of the lower bound). The lower bound Eq. 34 becomes tight if and only if the proposal distribution satisfies:
g(z, x) = (z, x) x r(y) p(z,y) æ;
(42)
Proof. Equality in Jensen's 's inequality holds when the log argument constant over 2, y:
r(v)p(z, y x; const.,
Vz, y-
(43)
9(2, x)
Solving for q(z,g) y ()) yields:
9" (z, y x) r(v)p(2, y 2;6)
Z Σ 2,y
(())p(2,811
(44)
Le., the normalized weirhe-weighted joint distribution.
Lemma C.5 (Optimality of ELBO maximization). If 9" (z. y x) x (r())((),)))) 0) then:
arg max ELBO((,, (") = arg max log (())
(45)
Proof. From Lemma C.4 if = q², then:
ELBO(0, q²) log [()) = arg.ax q²) arg max (())
(46)
Thus, maximizing the ELBO is equivalent to maximizing the expected objective.
D
More Experimental Details
D.1
Experimental datasets
Following prior work [37 38 88 90]. evaluate method wide range of knowledge-
intensive benchmarks, including Natural Questions (NQ) [36], HotpotQA [84), MuSiQue [70], and
2WikiMulrihopQA (2WikiQA) 20 Table key statistics of these experimental datasets.
D.2 Evaluation metries
for evaluation: F1, Exact Match (EM), and Accuracy (Acc.). Exact Match (EM) checks whether the predicted string exactly matches the ground truth. Accuracy checks whether the ground truth answer is included in the generated answer, often referred to cover-Exact Match. Fl Score measures the overlap between the generated answer and the ground truth answer. It represents the harmonic mean of token-level precision and recall between these two sequences.
Following previous studies 57
80 88
we use following metrics from KILT 49
D.3 Baselines
For comprehensive evaluation, we compare EXSEARCH with a range of competitive baselines, categorized into three groups based on their use of retrieval and reasoning integration strategies: (i) direct reasoning without retrieval; (ii) advanced retrieealauugmented generation; and (iii) iterative retrieval-augmented generation.
33

---

Table 5: Statisties of our experimental datasets, where we provide the amount of training and evaluation dataset, the average length of imput query (word) as well as the retrieval corpus.
Experimental
Training data size
Query length (Train)
Evaluation data size
Query length Retrieval (Evaluation)
benchmarks
corpus
Nature Question 36
58,622
9.21
6,489
9.16
Wiki2018
Hotpot QA
84
90,185
17.85
7,384
15.63
Wiki2018
MusiQue QA
70
19,938
15.96
2,417
18.11
Wiki2018
2WikiMultiHopQA 20
167, 454
12.74
12,576
11.97
Wiki2018
Direct reasoning without retrieval. These methods rely solely on the LLM's internal parametric knowledge reason over the input and generate answers, without incorporating any external infor- mation. We evaluate several recently released models, including both closed-source models, such as GPT-4o 48 and GPT-3.5 477, as well as strong open- models, such as DeepSeek-RI [17), Qwen2.5 [83] QwQ-32B [68] LLaMA-3.3-7000 [33. and Mistral-8x7B [26]. All of these models ex- hibit strong performance on wide range of natural language processing tasks.
following and chain-of chain-of-thought reasoning capabilities, achieving remarkable
Advanced -aumentl generation. These models retrieve relevant documents from an external corpus, followed by optional document filtering mechanisms such as re-ranking or summarization, and concatenate the useful information into the LLM's input context for answer generation. Specifically, we evaluate several widely used approaches, including: (i) ChatQA [42] and RankRAG (88), which unify various knowledge-intensitn document re-ranking, question answering) into a single framework, training LLMs on large-scale datasets; (ii) InstructRAG [74], which inserts retrieved documents into the LLM's input and trains the model to generate chains of thought to identify useful content for answer generation; (iii) RetRo- bust [87]. which trains the LLM to generate accurate answers conditioned on documents containing both correct and distracting information; (iv) Recomp 78), which uses extractive or abstractive sum- marization modules filter out irrelevant content from retrieved documents, employing large model (e.g., Flan-U1 UL.2, 20B 3 to generate answers from the remaining information. In our experiments, we use the abstractive summarization module, as serves as more competitive baseline than its extractive counterpart.
tasks (e.g.. www.lleeeggoundee. dialogue,
Iterative retrieval-augmeented generation. These approaches allow LLMs to actively interact with retrieval modules needed. For comprehensive evaluation, we include the following methods: (i) GenGround [64], which allows the LLM to iteratively retrieve external documents and refine its generated answer; (ii) DSPy (32). programming framework that enables an LLM decompose input queries and call external retrievers through structured prompting; (iii) SearchChain [80. which guides the LLM to generate a chain of queries and invoke retrieval each step; (iv) Iter-RetGen [6]] and IRCoT (71), which prompt the LLM to interact with the retriever in an iterative, few-shot manner, (v) Verify-and and-Edit [94), which adaptively determines when to stop retrieval and finalize the answer based on the generation logits of the LLM; (vi) Generator-Retrieeer Generrrtr [ (abbreviated as Gen-Rei-Gen), which instructs the L.LM to answer question using both -eeer-ege and retrieved documents concatenated context. All of the above methods are implemented on GPT-3.5, following their officially released reproducible settings. In addition, we include the following open- source baselines: (i) Search-ol [39] which prompts the LLM to interleave query decomposition and document retrieval steps iteratively; (ii) Search-RI (28), which trains the LLM to use external search engines via outcome rewards and Proximal Policy Optimization (PPO (60); (iii) Self-RAG [4), which trains an LLM to retrieve documents on demand, assess their relevance, and generate final answers. This method is fine-tuned on 1 170k synthetic examples generated by proprictary LLM.
D.4 Data collection for warm-Up
Data collection procedure. To construct high-quality training trajectories that reflect realistic search behavior, we design a two-stage process: (1) first, we use GPT-4o to simulate step-by-step reasoning traces in the form of interleaved sub-queries; (2) then, each sub-query is paired with
httts://wugginnfae..oooogggo.ee
34

---

retrieved document, simulating the retrieval process of real system rather than directly relying on officially annotated golden documents.
To synthesize pseudo training trajectories, we leverage existing datasets, such as HotpotQA [4], generate pseudo training data for method, it is widely used multi-hop QA dataset similar to the setting of our agentic search method. In HotpotQA, each example consists of complex multi-hop query, final answer, and set of officially annotated sub-queries along with their corresponding supporting documents. For each question, we prompt GPT-4o to simulate a step-by- step reasoning process that interleaves sub-query generation, document retrieval, and intermediate evidence extraction. Specifically, we instruct the model to act an intelligent search agent. Given the original multi-hop question, its gold answer, and the full of supporting Wikipedia passages. the model is asked to restructure the reasoning path into an interleaved sequence of three special operations:
<THINK> to denote generated sub-query;
<SEARCH> indicate the citation ID of the passage used to answer the sub-query; and
<RECORD> provide the answer the sub-query.
The output continues in this format until the final answer is reached, which is prefixed by <F <Final> tag. An example is included in the prompt to illustrate the expected format. This approach generates high-quality, interpretable reasoning trajectories that align with the structure of our method. All synthesized outputs are included the supplementary material for reference.
You are an intelligent search agent that can simulate the question- answering process based on my question and answer
Given open-domain query about Wikipedia, I have already marked the correct answer at end the question and provided all the reference Wikipedia passages needed to answer the question. Your task is to reformat
provided question and references into a detailed question-ant iiciionansseer answering process.
Specifically, there should be three types of special tokens your output:
1.
<THINK>, followed by a sub-query
2. <SEARCH>, followed by the citation ID
3. <RECORD>, followed by the answer to the sub-query
Since this is a multi-hop question, your output should interleave the THINK>", <SEARCH>", and *<RECORD>" tokens until you reach the final answer
Please start with special token "Finna>>' followed by the final answer.
Here is a concrete example to demonstrate the output format: eeeaampee
Question: Which magazine started first, Arthur's Magazine or First for Women? (Answer: Arthur Magazine)
Reference:
[1] Arthur's Magazine | Arthur's Magazine (1844-1846) was an American literary periodical published in Philadelphia in the 19th century Edited by T.S. Arthur, it featured works by Edgar Poe, J.H. Ingraham, Sarah Josepha Hale, Thomas into "Godey's 's Lady 's Book".
Spear, and others. In May 1846, was merged
[2] First for Women First for Women is women magazine published by Bauer Media Group in the USA. The magazine started in 1989. It is based Englewood Cliffs, New Jersey. In 2011, the circulation the magazine was 1,310,696 copies.
Your Output:
<THINK> When did the magazine "Arthur's 's Magazine" start?
<SEARCH> [1]
<RECORD> 1844
<THINK> When did the magazine "First for Women start?
<SEARCH> [2]
<RECORD> 1989
35

---

<Final> Arthur's Magazine
Starting below, for the question "(question)", please complete your output following the above requirements.
Question: (question)
Reference:
(golden doc)
Your Output
After obtaining the simulated reasoning trajectories from GPT-4o, we pair each generated sub-query (<THINK> entry) with a retrieved document. To maintain consistency with our main experiments, we use the same retrieval setup, i.e., 2o as the retriever and the 2018 Wikipedia dump the retrieval corpus. For each sub-query, we first locate its corresponding gold supporting document (provided the official HotpotQA dataset) and it as a reference. We then apply ColBERTv2.00 to retrieve the most similar document from the full corpus based on its proximity to the gold reference. This ensures that the retrieved documents closely reflect what real system would retrieve while remaining grounded in the original supervision signal. The resulting data consists of interleaved sub-queries and paired retrieved documents, effectively mimicking real multi-hop retrieval trajectories. All synthesized trajectories and retrieval pairs are included in the supplementary material.
Cost for data collection. The primary cost of our data synthesis arises from using GPT-4o to transform comaa-annotated sub-query paths into simulated search trajectories. following the pattern used in EXSEARCH. In our main experiment, we generate 1,000 examples, with an average input length 5,095.2 2, tokens and an average output length of 732.20 tokens. Based on OpenAl's GPT-4o [3] pricing the total cost for constructing these 1,000 examples is approximately $12.73 for input and $7.30 for output, totaling around $20. Thus, the cost per example 1000 = 0.02. Token counts are directly obtained from OpenAl's API call messages. See the official OpenAl API documentation for more details.
D.5 Implementation details and hyperparameters
During the training stage, we trained the models with a learning rate of 2 x 10-6 using DeepSpeed Zero for efficient distributed optimization. The batch size was set to 4 for the 3B, 7B, and 8B models, and reduced to 2 for the 24B model due to memory constraints. We applied linear warm-up (10% of total steps), followed by a cosine learning rate scheduler. All experiments used BF16 mixed-precision precision training with sequence length cutoff of 8192 tokens. For each training example, we sampled 2 search trajectories (We also experimented with varying the sampling number, choosing {1, 2, 4, 6, 8}, but observed no significant difference in final performance). Table 6 summarizes the hyperparameters our implementation. The models used our experiments can be directly downloaded from HuggingFace, open-source platform for machine learning and deep learning.
During the inference stage, the maximal step T for the thinking search recording iteration is set to 5.
Table 6: Experimental settings for model training.
Batch size
Learning rate
Cutoff length
Gradient accumulation
Model
Scheduler
Qwen-2.5-3B-inen
10
8192 tokens
Cosine
16
Dwen-2255-BBinnstru
4
2 x 10
8192 tokens
Cosine
16
Hlama-3.2-3B-il
x 10
8192 tokens
Cosine
Llama-3, Jimma-.--8B-instruue
x 10
8192 tokens
Cosine
16
Mistral-7B--instmeet--...33
4
2 10-6
8192 tokens
Cosine
16
Mistral-24B-small-5500
x 10-6
8192 tokens
Cosine
16
36

---

The training process of EXSEARCH is relatively straightforward to implement. Skeleton Py PyTorch code for EXSEARCH is demonstrated below.
import torch import torch. forch.nn.fio functional as
def wetpuute_ccuslll.iigtttt (ref_mdell, tokenizer, trajectory, answer);
Compute the reward weight w(z) p(answer trajectory).
This function estimates how likely the reference model (at step t) would produce the correct answer y given a sampled reasoning trajectory z.
Args:
ref_model: The frozen language model at current iteration. tokenizer: The tokenizer corresponding the model. trajectory: list of messages (e.g., in chat format).
answer: The gold answer string.
Returns:
weight (float): Estimated likelihood p(y x, z), scalar reward.
"*"
input_ids
return_tensors=""pt" itnprnnorrs""""iiitttiiiiiii input_iids
label_ coken,zerransswr,, return_tensors="pt")) input_ids
with torch. no_ggaa(()
logits ref ____mmodllllg (inputt_i____d logits(: clabel_ids.sizen :]
log_probs log__oooooff sof tmax dim=-1)
answer_ _oopp = torch torch.gather [log_probs, 2, unn.iiieesssueeee answer_loga answer_logo squeeze(-1)
weight answer saawwrr_lop.soe
exp() item()
# Likelihood as scalar
return weight
def W.step_Jearninngree_.ddd input_ids, label_ids, weights):
Compute re-weighted cross-entropy loss for a given trajectory.
This function performs forward computation of the M-step by applying scalar reward to the log-likelihood loss, encouraging trajectories that lead to correct answers.
Args:
ref_model: The language model to be updated. input_ids: The sampled trajectories (token IDs), shape (B, L) label_ids: The gold answers (token IDs), shape (B, L)
weights: Pre-computed weights, shape (B)
Returns:
loss (Tensor) scalar loss value used for gradient update.
ref_mode (input_ids) logits[: :-1, :] labels = label_ids[:, 1:]. to]
logits
loss_fct fct = torch. www..rrssfirrooooooeeeereedddddd
reduction="mean")
loss loss fos_ffct v-e
labels. view(-1))
Apply pre-computed weights
weights torch.exp(() exp (weights - weights.max()) / weights.sum()
37

---

Qwan-25--7 73 on Triiiin stt
Qwen on Evaluxtion Set
Lama on Training
Llama 3.1 on Evaluntion Set
HotpatQQ
MusiQQee
2WikiQA
71
80
66
score
70
spore
Fl score
60
55
22
X
50
40
15
45
Triining
Thining Iteration
Triining
Tmining Iteration
Figure 7: Training convergence of 2 and Llama-3. 1--ma-3.1-8B. where we report the F/ score for checkpoints each iteration. The Oth iteration indicates the initial warm-up training.
loss = loss * weights. meagh))smeen() # Weight the loss by the average of the weights return loss
Table 7: Precision @ K (K 3, for method (w/ 8B Llama3. and 7B Qwen2.5) and strong baselines.
Tasks
NQ
HotpotQA
MusiQue
2WikiQA
Avg.
Metrics
P@3
P@5
P@3
P@5
P@3
P@5
P@3
P@5
P@3
P@5
C0.
41.36
35.97
28.71
25.20
6.41
5.44
17.96
16.10
23.61
20.68
Re-ranking
MonoT5 [46
42.44
36.73
31.48
26.86
5.85
19.27
16.93
24.95
21.59
BGE 776
49.55
40.29
34.50
28.67
7.70
6.50
20.45
18.00
28.05
23.36
Vicuna 51
42.68
36.97
29.49
26.71
7.09
6.02
18.75
17.71
24.50
21.85
RankZepyhr 51
41.54
37.77
30.76
27.35
8.98
7.82
18.87
16.57
25.04
22.38
Query decomposition
Search-ol 39
42.42
37.01
42.01
34.11
17.23
13.13
25.34
18.24
31,75
25.62
Search-rl
28
40.17
36.86
27.12
32.67
4.07
8.30
16.25
23.49
21.90
25.33
Ours- 7
43.50
37.58
44.47
37.49
18.85
15.91
29.60
23.33
34.11
28.58
Ours- 1-8B
43.56
37.49
43.76
36.91
19.33
16.30
28.22
22.20
33.72
28.22
D.6
Supplementary experimental results
Supplementary results for retrieval performance. the main body of our paper, we have report the recall @ K score for the proposed EXSEARCH and strong baselines. Below, we supplements the performance in terms of precision @ K score in Table 7 These results further indicate that EXSEARCH, by dynamically expanding the search reasoning unfolds, can also improve retrieval performance in addition to enhancing end-to-end answer accuracy.
Supplementary results for training convergence experiments. The theoretical analysis in $ C.2 guarantees that our training objective is non-decreasing across iterations. To empirically validate this, we evaluate model checkpoints after each iteration on both the training and test sets. In the main body of our paper, we report the performance in terms of the exact match scores. In this appendix, we further supplements the performance in terms of F1 metrics in Figure ( for more comprehensive comparison. We observe consistent performance improvement over iterations, eventually stabilizing, confirming the expected non-decreasing convergence. On evaluation sets, models typically peak by second or third iteration, demonstrating rapid practical convergence. These results align with our theoretical guarantees and highlight the efficiency of our approach.
Supplementary results for EXSEARCH with cold start. To investigate the role of warm-up supervision, we train separate models with varying amounts of supervised fine-tuning (SFT) data, denoted by K, and apply EXSEARCH to each independently. The exact match (EM) scores has been reported in the main body of our paper as the primary evaluation metric. In this appendix, corresponding
scores are presented in Figure &as as supplementary result. all models benefit from iterative self-training, with larger K values leading to faster convergence and better final performance. Remarkably, even with only K 100 or no warm-up supervision (K 0), the models still achieve substantial gains, demonstrating the robustness of EXSEARCH in
observe that
38

---

Score in Evaluation Set
Score Training Set
62
75
55
65
w/ 1000 examples
w/ 1000 examples
F1 score
F1 score
w/ 900 examples
w/ 900 examples
48
55
w/ 800 examples
W/ 800 examples
w/ 700 examples
w/ 700 examples
41
w/ 600 examples
w/ 600 examples
w/ 500 examples
w/ 500 examples
34
w/ 100 examples
35
w/ 100 examples
w/ Zero examples
Zero examples
2 Training Iteration
2
4
Training Iteration
Figure 8: Fl score for EXSEAARCIIQQwen2255-77 that was initially empowered by different amounts of warm-up data.
low-resource and cold-start scenarios. Based on the cost analysis in $D.4 synthesizing each example costs only $0.02 average. Thus, empirically, in our experiment, synthesizing 100 examples incurs an approximate cost of $2.
Supplementary results for human evaluation. Considering the potential bias of automatic met- rics 633, we conduct a human evaluation with three educated individuals assessing the correctness of 100 randomly sampled cases from five benchmarks, using two-scale rating for correct; 0 for incorrect). Each query is paired with the corresponding golden documents and ground truth answers from the original datasets, which serve as references for the human evaluators. We ask at least two annotators to evaluate the same case repeatedly. If there is discrepancy between two annotators, ask third annotator to recheck it. The results are presented in Table where we found that our method achieve the highest correctness, further indicating its effectiveness. In our human evaluation, the overall Kappa value is 0.771, demonstrating substantial agreement among the annotators. This indicates the reliability of the evaluation process.
Table 8: Human evaluation 100 randomly sampled cases.
GPT-4o
InstructRAG
Search-ol
Search-R1
EXSEARCH
Correctness
48/100
40/100
46/100
50/100
54/100
39

---

52
Evalunton Performance
Evslation Perforamce
52
Evalustoo Perfirmance
Evalustion Performance
March
33-
28.
28
28
K-Zero examples
w/ K-100 examples
K-500 exmmlees
K-600 examples
Training lteration
Training Itenation
Training Iteration
Training Iteration
75
Training Performance
Training Performance
Trining Perfomiance
Training Perfermance
51
51
59
27
K-Zero examples
w/ K-100 examples
K-500 examples
K-600 examples
0
Training Iteration
Thaining Itenution
Training Iteration
Training Iteration
Evaluntion Performanee
Evalanion Performince
Evalustion Perfffrance
Evalustion Performance
-1
51
44-
37
37
K- examples
w/ K-800 examples
X-400 exumples
K=10000 eumples
50
30
50
Training lteration
Training lieration
Training lteration
Training lleraion
82
Training Performance
Training Performince
82
Triining Perfompance
Training Ferformaace
71
60-
K- 700 examples
wt K-800 examples
wi X-900 examples
w/ X- 1000 eumples
0
Training lteration
Training lerratonn
Training lleraton
Training lteration
Figure Exact match score for EXSEARCII-Qwen-225575 which initially empowered by varying amounts of warm-up data, during the iterative training process in EXSEARCH.
40

---

62
Evalunton Perfermance
Evalation Performmce
62
Evaluston Perfirance
Evalnstion Peffrrmane
55
41-
34
K-Zero examples
w/ K~100 examples
34
K-500 axamples
K-600 examples
Training lteration
Training Itenation
Training lteration
Training Iteration
Training Performance.
Training Penformance
Triiing Perfomiance.
Training Benfermance.
K-Zero examples
w/ K-100 examples
wi' K-500 exumples
K-600 examples
2 Thaining Itenution
0
Training Iteration
Training Iteration
Training Iteration
Evaluttion Perfermance
Evaluntion Performane
Evalustion Perfirmence
Evalustion Performance
65-
65
Seore
60
60
60
55
50
K0700 examples
w/ K-600 exmmlee
K-400 axmmpees
K=00000 eumples
Training ltrration
Training lieration
Training lteration
Training lleaaion
Training Performance
Training Performance
Triiing Perfompance
Training Perfermance
78
71
64
64
57
57
57
50
KKK700 examples
wt K-800 examples
50
wi X-400 examples
50
(0-0000 eumples
2 Training lerraionn
0
Training lteration
Training lleration
Training lteraion
Figure 10: Fl score for EXSEARCII-QNen-22.5 which initially empowered by varying amounts of warm-up data, during the iterative training process in EXSEARCH.
41

---

Table 9: Experimental results for applying our method to various LLMs.
NQ
HotpoiQA
MuSiQue
2WikiQA
Avg.
Metrics
F1
FM
Acc.
F1
FM
Acc.
F1
FM
Acc.
F1
EM
Acc.
F1
FM
Acc.
Ours Qwen 2.5-3B
46.23
36.76
39.12
54.32
42.22
46.08
19.44
13.76
13.94
43.39
37.24
44.78
40.85
32.50
35.98
Ous--wee--25-77
56.37
47.07
51.75
62.59
50.35
54.32
29.68
22,05
24.34
57.14
52.62
54.37
51.45
43.02
46.20
Ours-Llama-3.3-33
41.42
33.49
35.17
44.12
33.53
36.14
17.64
11.23
11.82
41.73
36.28
42.22
36.23
28.63
31.34
Ours-Llama-3.1-58
55.21
43.71
50.76
60.72
47.59
53.59
30.83
20.98
24.65
54.62
47.48
54.21
50.35
45,80
Ours-Mistral-7BBno -Bnintree
56.83
45.13
52.05
59.65
50,35
54.78
30.32
23.47
24.98
53.93
47.38
54,82
50.18
11.58
46.66
Ours-Mistal-2501-2000
59.89
47.62
56.39
67.03
54.51
59.98
35.84
23.68
28.54
60.81
53.19
61.59
55.89
44.75
51.63
E ExSEARCH-Zoo: Extending EXSEARCH for Diverse Scenarios
Extensive experiments wide range of Knowledge-intensiv benchmarks demonstrate the state-of- the-art performance of EXSEARCH, as detailed in the main body of the paper. This strong performance motivates extend EXSEARCH more diverse scenarios. We introduce EXSEARCH-ZO0, comprehensive framework that enhances EXSEARCH along two key dimensions: (i) Diverse backbone LLMs across model families (LLaMA, Qwen, Mistral) and scales (7B-24B parameters); (ii) Extended actions, such as document re-ranking, to enrich the existing reasoning actions (thinking, search, and recording). Below, we describe how EXSEARCH extended along each of these dimensions.
E.1 Diverse model famiilies and scales
We apply EXSEARCH to a range of LLMs with varying parameter sizes and model families. As shown Table 9. the results exhibit clear scaling-law pattern. More specifically, we highlight two key observations: First, there is consistent performance improvement as the model size increases; Second, even models with as few 3B parameters achieve strong performance when augmented with our method. These findings suggest that EXSEARCH is broadly applicable and scales favorably across different model families.
E,2 Extended retrieval strategy
In EXSEARCH, when faced with complex information needs, the LLM iteratively infers missing knowledge, evaluates retrieved evidence, and adapts its search strategies as new information acquired. This behavior is formalized as a reasoning-interllavee search trajectory consisting of three core actions: think - seek record, as introduced in $ [..]] Although this pattern general, for more complex tasks, may also need to introduce additional actions improve the end-to-end performance. To illustrate the extensibility of EXSEARCH, we introduce an additional document re-ranking action to the vanilla EXSEARCH framework. The re-ranking step acts as a filtering mechanism, allowing the model to discard irrelevant content and focus on cleaner, more useful evidence before generating intermediate reasoning steps.
Example: Re-ranking as a reasoning action. We introduce a document re- re-ranking step between retrieval and evidence selection, resulting in four-step reasoning pattern: think - seek rank - record. Specifically, we implement this re-ranking following generative re-ranking techniques [51 671. where the LLM reads the retrieved documents and autoregressively generates ranked list of selected identifiers (e.g., [1] > [2] [3]).
The updated reasoning process consists of: (1) generating sub-query z;; (2) retrieving candidate documents d, using the retriever R; (3) re-ranking the retrieved documents to select the most relevant ones, denoted as d; and (4) reflecting on the selected documents d, by extracting an intermediate answer e, to the sub-query Iz-
Formally, we represent the full trajectory sequence of triplets {(x,, di, ) and define its likelihood conditioned on input æ and parameters 0 as:
x
p(2 xx; 0)
p (2x,, di, es)
ze:;
(47)
p(z: Zzs: 0)
p(d, d, x. R(x:): ple; 22, d):)).
thinking
retricval and ro-ranking
recording
42

---

FI Score Evaination
Exact Match Evaluation Set
Accuracy Evalcation
62.6 62.643.7
54.3 55.8
$5.455.9
57.1
58.6
50.4
52.6
54--11.8
33.2
47.1
50
Exact Match
Accimay
10
40
29,7
26-
22.0 23.1
26-
2A
25.4
20
HоppoiQA MusiQue 2WikQAA
HolpoiQA MusiQue 2WikQQA
12
NQ
HotpoQA MusiQue 2WSTIQA
Oun Vanilla
Ours w Extended Re-ranking
Figure 11: Performance EXSEARCH when extended with an additional document re-ranking action, where we report the exact match score, Fl score and accuracy for a comprehensive comparison.
After finalizing full trajectory the LLM generates final answer y based on all intermediate reasoning steps via ~ ply x, 2; 0).
Compared to the vanilla EXSEARCH, the only modification in the above variant is the additional doc- ument re-ranking action to the E-step, while the overall EM training framework remains unchanged. Thus, we can apply the same EM-style optimization in Eq. 19 for this variant, with the learning objective formulated as:
arg max =x-p(=)=00)) (w(z) log x;; + w(z) logply x 2,2;
(48)
where w(z) is the importance weight based on how well supports the correct
The log-likelihood decomposes into two components: LR. = Σ logp(æ; =e; 0) + log p(d;
ar R(x:)::)) 0) + p(e; 24 d;; () and L log plu æ z; 0). The former LR corresponds to
learning iterative search, while the latter LA corresponds learning how aggregate information for answer generation.
Experimental results. Figure I1 II] reports the results of incorporating re-ranking into the reasoning trajectory of EXSEARCH. We observe that adding a re-ranking step consistently improves perfor- mance across all evaluated datasets. For example, on MuSiQue and 2WikiQA, we observe average gains of +1.9 and +2.4 points in F1 score, respectively, indicating that selective document filtering can further enhance the model's ability to reason over relevant information within EXSEARCH. These results highlight the extensibility step, we can effectively adapt the search-reasoning loop to more complex settings. This enables move beyond fixed action templates and incorporate specific requirements of downstream tasks. We believe this opens avenues for extending EXSEARCH specialized tasks, such retrieval-augee -uggenetd fact verification, open- multi-hop reasoning. or tool-integrated planning, by incorporating additional task -seciifiii actions.
framework: by augmenting the action space with additional
reasoning operations based on the
F Prompt and Case Study
F1 System prompt in EXSEARCH
To enable step-wise reasoning and evidence-aware retrieval, we design a structured system prompt for EXSEARCH that guides the model to simulate an intelligent search agent. The detailed content shown below. This prompt instructs the model decompose complex query into sub-queries (thinking). retrieve relevant documents from Wikipedia based on each sub-query (search), and extract factual answers from the retrieved content (recording). This system prompt aims to encourage the LLM to perform multi-hop reasoning via an interleaved search-and-reas loop, rather than attempting to answer the question diagnose the model's 's behavior at each stage of the reasoning process. Finally, the generation ends with INAL> token and the final answer, enabling seamless integration into downstream pipelines and evaluation.
single step. also enforces an interpretable action trace, allowing us to
43

---

You intelligent search agent capable of simulating question- answering process by actively seeking information from Wikipedia to answer given question.
Specifically, given an open-domain query, please iteratively: (1) Formulate sub-query search on Wikipedia; (2) Select useful documents from the search results and (3) Extract supporting facts from the selected documents.
Your output should include three types of special actions corresponding the above steps:
(1) <THINK>: Formulate sub sub-query.
(2) <SEARCH>: Retrieve and carefully read the documents using the Formulated sub sub-query.
(3) <RECORD>: Extract the answer to the sub-query from the documents.
Since this a multi-hop question, your output should interleave <THINK >, <SEARCH> and <RECORD> actions until reaching the final answer. Conclude your output with the special token <FINIAL> followed by the final answer
Below is the task for you to complete:
<USER QUERY> (THE_INPUT_TASK,
Your Output:
F.2
Human evaluation
In order to assess the end-to-end performance of the proposed model, we conducted human evaluations verify whether the model's outputs align with the ground-trute answers. The evaluation procedure involves presenting human evaluators with series of tasks, where they are asked to judge whether the model's ' prediction consistent with correct answer. The task provided to the human evaluators is as follows, where placeholders (question), (model output), and [answer) are replaced with the actual question, model's ' response, and the ground-truth answer, respectively. The evaluators are asked to determine whether the model's prediction implies the correct ground-truth answer.
Guidance Human to Evaluate the Correctness of Model Output
In the following task, you are given a question, a model Prediction for the question, and ground-truth Answer to the question. You should decide whether the model's 's Prediction implies the Ground-truth Answer.
Question {question] }
Prediction
(model output)
Ground-truth Answer (answer)
Does the Prediction imply the Ground-truth Answer? Output Yes No:
To guarantee annotation quality, we ask least annotators evaluate the same questions repeatedly. If there is a discrepancy between the two annotators (i.e., when two annotators give a different correctness score), third annotator is asked to review and resolve the inconsistency. To quantify the reliability of the annotations, we compute Cohen's Kappa score to measure inter- annotator agreement. A Kappa value close indicates strong agreement, while value near suggests random agreement. In our human evaluation, the overall Kappa value is 0.771. demonstrating substantial agreement among the annotators. This indicates that our evaluation process reliable and that the results can be confidently used to assess the model's performance.
44

---

F.3
Case studies
Good case: Correct runtime case of EXSEARCH. This example demonstrates a successful reasoning trajectory executed by EXSEARCH for a compositional multi-hop question: Navarone Garibaldi is the half-broon half-brother of a singer who has been married how many times? The task requires (i) identifying the half-sibling relationship: and (ii) retrieving the number of marriages of the referenced singer. The model first identifies Lisa Marie Presley as Navarone Garibaldi's s half-sister, then retrieves the number of times she has been married. It correctly terminates the search after the evidence is obtained and outputs the gold answer, i.e., four.
Below, in each block, the content behind <THINK> and <RECORD> represents mooel--generate outputs, while <SEARCH> denotes retrieved document snippets from the external corpus.
INPUT QUERY: Navarone Garibaldi is the half-brother a singer who has been married how many times? (Golden answer four times)
ssssssss.es Turn anssssessss:
<THINK> Who is Navarone Garibaldi 's half
<SEARCH> [0] Title: Navarone Garibaldi. Content: Navarone Garibaldi Navarone Anthony Garibaldi (born March 1, 1987) is an American musician. He is the frontman of the band Them Guns. Garibaldi is the son of Marco Garibaldi and Priscilla Presley, and also half-brother of Lisa Marie Presley. Navarone Anthony Garibaldi and Marco Garibaldi on March 1, 1987
born actress Priscilla Presley Santa Monica, California. With his mother Priscilla being the former wife of the late Elvis Presley, media scrutiny was often focused on the family. Garibaldi 's histerister is Lisa Marie Presley, Elvis only child. He is the unele of Riley and Benjamin Keough and Harper... [2]... [3]... <RECORD> Lisa Marie Presley
censEss***s Turn =*ss****ann
<THINK> How many times has Lisa Marie Presley been married? <SEARCH> [0] Title: Lisa Marie Presley. Content: Lisa Marie Presley Lisa
Marie Presley (born February 1, 1968) is an American singer-songwriter, She is the only child of singer and actor Elvis Presley and actress
Priscilla Presley, as well as the sole heir to her father's estate. Presley has developed career in the music business and has issued three albums. She has been married four times, including Jackson and actor Nicolas Cage, before marrying music producer Michael Lockwood, father of her twin girls. Lisa Marie was born on February 1, 1968, to Elvis and Priscilla Presley at Baptist Memorial Hospital in Memphis, Tennessee, nine <RECORD> four
singer Michael
sessssss*.s Turn (answer ageregation)
<FINAL> <SEARCH> end search.
The final answer the input question "Navarone Garibaldi is the half- brother of singer who has been married how many times?"
Output: four
Bad case: Over-searching due to missed reasoning. We present a concrete example below, where the LLM gives a correct answer but overly relies on the retrieval module. In Turn and Turn 2, the model successfully retrieves the occupations of both individuals via two well-formed sub-queries. However, instead of reasoning over the retrieved facts to answer the question, the model repeatedly issues similar external queries (Turn 3-5), such as Are there any shared occupations between Maher Sabry and Til Schweiger? These repeated queries yield no new information and reflect an over- dependence on retrieval, rather than using previously acquired knowledge for local inference. This highlights limitation current retrieval-aun retrieell-augmentedd models: they often fail to consolidate retrieved evidence before issuing additional queries, resulting in unnecessary search steps.
45

---

USER QUERY: What occupations shared by Maher Sabry and Til Schweiger? (Golden answer director, and producer)
=na.sss**** Turn
<THINK> What is Maher Sabry occupation?
<SEARCH> [1] Title: Maher Sabry. Content: Maher Sabry is an Egyptian theater director, playwright, film director, producer and screenwriter, poet, writer and cartoonist. A gay activist.... [2]...; [3]...
<RECORD> Maher Sabry occupations include theater director, playwright, film director, producer, screenwriter, poet, writer, and cartoonist.
Turn
<THINK> What is Til Schweiger's 's occupation?
<SEARCH> [1] Title: Til Schweiger. Content: Til Schweiger Tilman Valentin Til Schweiger (born 19 December 1963) is German actor, voice actor film director, film producer, and film editor. He runs own production
company, ...; [2] [3]
<RECORD> Til Schweiger's 's occupations include actor, voice actor, film director, film producer, and film editor.
**E.ss"**.*: Turn =usssss*ss.:
<THINK> Are there any shared occupations between Maher Sabry and Til Schweiger?
<SEARCH> [1] Title: Maher Sabry. Content: Maher Sabry is an Egyptian
theater director, playwright, film director, producer and screenwriter, poet, writer and cartoonist. A gay activist,.. [2] [3]
<RECORD> Maher Sabry's occupations include theater director, playwright, film director, producer, screenwriter, poet, writer, and cartoonist.
<THINK> Are there any shared occupations between Maher Sabry and Til Schweiger?
<SEARCH> [1] Title: Maher Sabry. Content: Maher Sabry is Egyptian
theater director, playwright, film director, producer and screenwriter, poet, writer and cartoonist. gay activist, [2] [3]
<RECORD> Maher Sabry occupations include theater director, playwright, film director, producer, screenwriter, poet, writer, and cartoonist.
c*mpr..**** Turn =amcen.ecae.
<THINK> Are there any shared occupations between Maher Sabry and Til Schweiger?
<SEARCH> [1] Title: Maher Sabry. Content: Maher Sabry is an Egyptian
theater director, playwright, film director, producer and screenwriter, poet, writer and cartoonist. A gay activist [2]... [3]...
<RECORD> Maher Sabry's occupations include theater director, playwright, film director, producer, screenwriter, poet, writer, and cartoonist.
Turn cussss.*.*.
<THINK> Are there any shared occupations between Maher Sabry and Til Schweiger?
<SEARCH> [1] Title: Maher Sabry. Content: Maher Sabry is an Egyptian
theater director, playwright, film director, producer and screenwriter, poet, writer and cartoonist. gay activist,... [2] ... [3]
<RECORD> Maher Sabry occupations include theater director, playwright, film director, producer, screenwriter, poet, writer, and cartoonist.
sesssssaaas Turn
<THINK> Are there any shared occupations between Maher Sabry and Til Schweiger?
46

---

<SEARCH> [1] Title: Maher Sabry. Content: Maher Sabry is an Egyptian
theater director, playwright, film director, producer and screenwriter, poet, writer and cartoonist. gay activist, [2] [3]
<RECORD> Maher Sabry occupations include theater director, playwright, film director, producer, screenwriter, poet, writer, and cartoonist.
Turn (Up to the maximum search times) <FINAL> theater director, playwright, film director, producer, screenwriter
please give the final answer to the input question: What occupations are shared by Maher Sabry and Til Schweiger? Playwright, film director, producer, screenwriter
Bad case: Under-searching due to premature entity binding. In this example, the user asks seemingly straightforward compositional question: Into Dust song by the alternative rock band formed in which city? The correct answer is Santa Monica, California, as the song *Into Dust performed by the alternative rock band Mazzy Star. However, the model prematurely assumes an entity match based on partial overlap with the query phrase "Into Dust," mistakenly linking a different band named *Cities in Dust Canadian punk group. shown the first retrieval result, this band has no association with the queried song but is erroncously selected due to superficial string-level similarity and genre proximity. The model then terminates the search after this single retrieval step and outputs Hamilton, Ontario, as the final answer, based solely on the retrieved band's s origin. This reflects a classic under-searee searchin failure mode, where the model fails to verify whether the retrieved entity is semantically compatible with the query. In this case, simple clarification query, such as "Who performed *Into Dust (song)?" would have revealed the correct band and corresponding city. This example highlights the need for disambiguation-awarr reasoning mechanisms that delay answer generation until entity identity sufficiently grounded.
USER QUERY: Into Dust is song by the alternative rock band formed in which city? (Golden answer: Santa Monica, California)
<THINK> Which city the alternative rock band that performed "Into Dust" formed in?
<SEARCH> [1] Title: Cities in Dust (band). Content: Week Toronto. Independent record label Paper Bag Records signed Cities in Dust in April 2006, in part due to the onstage antics of Frank. Soon after, their debut album Night Creatures World Electronic Music Festival in June 2006. The band was nominated for three awards at the 2006 Hamilton Music Awards: Record of the Year, New Artist/Group of the Year, Alternative Recording of the Year. Night Creatures
released. Cities in Dust appeared at the
of the Top 10 Punk albums of 2006 in Canada. and it received heavy rotation on CBC [2] [3]...
named by Exclaim! magazine
<RECORD> Hamilton, Ontario
USER QUERY: Into Dust is song by the alternative rock band formed in which city?
please give the final answer to the input question: Into Dust is a song by the alternative rock band formed in which city? Hamilton, Ontario
47

---

Table 10: Case study of seaee-gg caused by entity confusion and early termination.
Aspect
Observation
User Query
Into Dust is song by the alternative rock band formed in which city?
Gold Answer
Santa Monica, California (Mazzy Star)
Retrieved Fact
Hamilton, Ontario (hased on Cities in Dust band)
Search Count
Entity Linking Error
Mistook "Into Dust" as song by "Cities in Dust"
Failure Mode
Under-searching: Incorrect assumption based on first retrieved entity, no disam- biguation step.
Suggested Fix
Add entity verification query: "Who performed Into Dust?" or "Into Dust band name" before inferring location.
Table 11: Case study illustrating over-searching due to lack of intermediate reasoning.
Aspect
Observation
User Query
What occupations are shared by Maher Sabry and Til Schweiger?
Query Type
Compositional: Requires comparing two entities' attributes.
Sub-Queries Issued
Turn 1: What is Maher Sabry's 's occupation?
Turn 2: What is Til Schweiger's 's occupation?
Knowledge Retrieved
All relevant occupations were retrieved correctly for both individuals by Turn 2.
Expected Behavior
The model should compare two sets of occupations and output the shared ones.
Observed Behavior
Model redundantly issues semantically similar queries in Turn 3-5, such as: "Are there any shared occupations between Maher Sabry and Til Schweiger?"
Fallure Mode
Over-searching: The model repeatedly queries despite having sufficient informa- tion to answer.
Impact
Increased search cost, unnecessary API calls, and longer latency with no gain in answer quality.
Insight
Reasoning over retrieved evidence should be prioritized once coverage is suffi- cient; redundant querying should be suppressed.
Table 12: Comparison between a successful multi-hop reasoning trajectory and failure case due to over-searching. The successful case stops searching after gathering sufficient evidence, while the failure case redundantly queries instead of reasoning.
Step
Successful reasoning
Over-searching failure
Input Query
Navarone Garibaldi is the half-brother of singer who has been married how many timex?
What occupations are shared by Maher Sabry and Til Schweiger?
Turn
<THINK> Who is Navarone Garihaldi's 's half-sister?
<THINK> What Maher Sabry's occupation?
<SEARCH> Lisa Maric Presley
<SEARCH> + Maher Sabry: [occupations]
<RECORD> Lisa Marie Presley
<RECORD> theater director, screenwziter, etc.
Turn 2
<THINK> How many times has she been married?
<THINK> What is Til Schweiger's 's occupation?
<SEARCH> - [four marriages]
<SEARCH> - Til Schweiger: [occupations]
<RECORD> four
<RECODD> actor, producer, etc.
Turn
<FINAL> The answer four. Search terminated successfully.
<THINK> Are there shared occupations?
<SEARCH> Muher Sabry (aguin)
<RECORD> (repetition)
Fallure Mode
N/A correct answer produced with minimal hops.
Over-scarching: fails reason over retrieved evi- dence, keeps querying.
Insight
Success relies on using retrieved facts to trigger unswer generation.
The model needs an early a reasoning trigger mechanism.
48


Reviews:
[{"review_id": "jaoWEddPAX", "reviewer": "Reviewer_zB4g", "summary": "This paper focuses on the design of an LLM-based information retrieval method enhanced with the Expectation-Maximization (EM) algorithm. To accommodate more complex query scenarios, it models multi-turn, autoregressive retrieval reasoning paths—including generated sub-queries, retrieved documents, and accumulated evidence—as latent variables. Using the EM algorithm, it optimizes the expected log-likelihood of the joint distribution $\\mathbb{E}_{\\text{reasoning path}} \\left[  P(\\text{reasoning path}, \\text{answer} \\mid \\text{query}) \\right]$. Compared to directly optimizing the log-likelihood of the answer given a query, this approach enables more effective exploration of reasoning paths, resulting in more accurate and interpretable answers for challenging information retrieval tasks. This paper also conduct extensive experiments across multiple datasets and foundation LLMs, benchmarking against a diverse set of strong baselines. Detailed ablation studies, in-depth analysis, and case studies further demonstrate the effectiveness of the proposed approach.", "questions": "**Question 1:**\n I’m curious about how EXSEARCH compares to other baselines in terms of training and inference efficiency. Some quantitative insights or comparisons in this regard would be helpful for understanding its practical applicability.\n\n**Question 2:**\n Given the discussion in Weakness 2 about the EM algorithm's local convergence properties, it would be valuable to understand how EXSEARCH behaves under different initialization strategies—such as varying prompt templates or parameter initializations. Has the robustness of convergence under these variations been explored?\n\n**Question 3:**\n As noted in Weakness 1, the term *“Iterative Self-Incentivization”* seems to reflect the optimization dynamics introduced by the EM algorithm. At the same time, EXSEARCH could reasonably be seen as a form of “agentic search” within the information retrieval domain. This raises an open question about naming: should the title highlight the underlying method (e.g., EM-enhanced LLM-based IR) or the emergent behavior (e.g., agentic characteristics)? I’d be interested to hear the authors’ perspective on this balance.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n* **Strength 1:**  The core idea of this paper is to treat the retrieval reasoning path as an unobserved latent variable in an autoregressive manner and optimize it through iterative self-incentivization using the Expectation-Maximization (EM) algorithm. The integration of EM-based modeling is both conceptually sound and practically elegant—resulting in a streamlined algorithm that is not only effective but also insightful. This framework offers a flexible foundation: by tailoring the latent variable assumptions to specific application scenarios, corresponding EM algorithms can be readily designed and adapted.\n* **Strength 2:**  The paper is clearly written and well-organized, with clean and informative figures and tables. The presentation is easy to follow, making the content highly accessible to readers. Furthermore, the appendix provides a thorough discussion of the paper’s limitations, ethical considerations, and societal impacts, along with detailed analyses and commentary on both successful and failure cases.\n* **Strength 3:**  The paper conducts extensive experiments across multiple datasets and foundation LLMs, benchmarking against a wide range of strong baselines. Comprehensive ablation studies, in-depth analyses, and carefully examined case studies further validate the effectiveness and robustness of the proposed approach.\n\n\n**Weaknesses:**\n\n* **Weakness 1:**  The core contribution of this paper lies in integrating the EM algorithm with LLM-based information retrieval. However, the current title—*“Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers”*—does not reflect either the focus on information retrieval or the role of the EM algorithm. In fact, the phrase *“Agentic Searchers”* initially led me to believe the paper was about autonomous agent architecture design, similar to works like *“Automated Design of Agentic Systems”* [1] and *“AgentSquare: Automatic LLM Agent Search in Modular Design Space”* [2]. Given that the experimental setup primarily benchmarks against information retrieval datasets and RAG baselines—and does not compare with other agent search methods (e.g., [1], [2]) or broader agent scenarios (e.g., embodied agents, game environments, tool use)—emphasizing \"information retrieval\" in the title would help clarify the paper’s true focus at first glance and better align with the experimental scope.\n* **Weakness 2:** The theoretical contribution emphasized in the main text—namely, the convergence of the proposed algorithm—does not appear to leverage any domain-specific properties of information retrieval. To my knowledge, it largely reiterates the standard convergence behavior of the EM algorithm with respect to the value of the loss function, which has already been thoroughly established in classical texts such as PRML [3]. This discussion may be more appropriately placed in the preliminaries section. Moreover, the Pinsker’s inequality derivations in the appendix could benefit from clearer attribution—specifically, which parts are novel contributions of this paper. It is also worth noting that the convergence of the EM algorithm in terms of loss value does not imply convergence to a global optimum; it merely guarantees convergence to a stationary point. Given that loss functions in information retrieval are often ReLU-based and may lack strong convexity, the EM algorithm is unlikely to exhibit global convergence in this context. Exploring the impact of different initialization points might provide a more informative and realistic characterization of the model’s convergence behavior.\n\n\n**Minor grammatical issues:**\n\n- [1] Line 38: “an LLM” → consider changing to “a LLM”.\n- [2] Line 112: “$t$th” → might be written as “$t$-th”.\n\n[1] Hu S, Lu C, Clune J. Automated design of agentic systems[J]. ICLR, 2025.\n\n[2] Shang Y, Li Y, Zhao K, et al. Agentsquare: Automatic llm agent search in modular design space[J]. ICLR, 2025.\n\n[3] Bishop C M, Nasrabadi N M. Pattern recognition and machine learning[M]. New York: springer, 2006.", "was_revised": true, "final_justification": "The rebuttal has addressed the majority of my concerns, and based on this, I would like to recommend that this paper be accepted.", "rebuttal": "Dear Reviewer,\n\nThank you for your recognition of this work. We truly appreciate your thoughtful suggestions and guidance, which help us improve our work. In response to your review, we have made the following revisions:\n1. Clarified key terms in the title, specifically \"agentic\", and considered more intuitive titles (addressing Weakness 1 and Question 3).\n2. Reorganized the analysis of convergence and shared more practical implementation insights (addressing Weakness 2 and Question 2).\n3. Compared the training and inference costs of the proposed EXSEARCH with those of the baselines.\n\nMore details are provided below.\n\n---\n\n$  $\n\n# Response to Weakness 1 and Question 3: Explanation about Our Title\n\nOur use of the phrase \"agentic searchers\" aligns with the LLM-based agent literature [1, 2]. In this context, **\"agentic\"** emphasizes an LLM that actively plans, executes, and reflects on a multi-step retrieval process (e.g., think → search → record in this work), rather than a fully autonomous embodied agent.\n\nTo avoid ambiguity and explicitly highlight the Information Retrieval (IR) focus, we are willing to adopt a more straightforward title for the camera-ready version, such as *Enabling Large Language Models to Retrieve via Iterative Self-Incentivization*. This title explicitly positions retrieval as the primary application domain and underscores \"Self-Incentivization\" as the key algorithmic contribution that underpins our Generalized Expectation-Maximization algorithm. We believe this new title will more precisely convey the paper's scope and contributions, making it easier for the research community to understand how our work advances LLM capabilities within information retrieval through a novel learning paradigm.\n\n$   $\n\n# Response to Weakness 2 and Question 2: Explanation about the ExSearch Convergence.\n\nWe deeply appreciate your insightful guidance on our theoretical analysis. Below are our detailed responses.\n\n**For the content re-organization**: Our core contribution lies in providing ExSearch, a flexible agentic search framework, complemented by a rigorous theoretical foundation based on the Expectation-Maximization (EM) algorithm (Section 2.2, Appendix D.1 and D.3), which demonstrates its progressive improvement. We agree that relocating corresponding detailed derivations to the Preliminaries section can enhance overall clarity, and we will revise accordingly.\n\n**For global optimality**: Achieving global optimality for neural-based models remains a crucial yet complex challenge. Many advanced methods, including various variational and gradient-based algorithms, often struggle to guarantee global optima. In agreement with your suggestion, the impact of factors such as different parameter initialization should indeed be explored. In this work, we have taken it as a guiding principle to conduct multiple trials with various initialization strategies, including:\n1. **Exploring different model architectures**: For example, we implement our method with Qwen-2.5-7B-instruct and Llama-3.1-8B, which possess distinct layers, hidden state sizes, and other architectural differences (See Table 1).\n2. **Utilizing models with diverse initial parameters**: For example, Qwen-2.5-7B-instruct and Llama-3.1-8B are pre-trained on different corpora, leading to varied initial neural parameters (See Table 1).\n3. **Varying model parameter scales**: We evaluate our method with models ranging in size, such as Qwen-2.5-3B-instruct and Mistral-2501-24B-instruct (See Table 1).\n4. **Adjusting the quantity of warm-up training data**: We evaluate models that are initially trained on varying amounts of warm-up data, from 100 to 1000 examples (See Section 5.5).\n\nDue to the limited space in our previous submission, we also explored various other settings beyond the experiments reported above. These included: (i) different random seeds, (ii) various prompt templates for warm-up, and (iii) different sampling temperatures for trajectory generation. \n\nFor our experiments, the random seeds and temperatures were collected from well-known baselines, such as Search-R1, and a commonly used toolkit, namely Verl [3].\n\nThis table shows how the model performs with varying random seeds:\n\n| Random Seed |   F1  |   EM  |  ACC |\n|--|--|--|--|\n| 42  | 62.59 | 50.35 | 54.32|\n| 24 | 61.47 | 50.15 | 54.78|\n| 1 | 62.07 | 51.06 | 54.08|\n| 12345 | 62.57 | 50.94 | 53.75|\n\n**Caption**: Model performance on HotpotQA with different random seeds, using Qwen-2.5-7B-instruct as the backbone LLM (with a temperature of 1.0).\n\nThis table shows how the model performs with varying sampling temperatures:\n\n| Sampling Temperature |   F1  |   EM  |  ACC |\n|--|--|--|--|\n| 0.7 | 61.98 | 51.05 | 54.04|\n| 0.8  | 62.59 | 50.35 | 54.32|\n| 0.9  | 62.94 | 50.94 | 54.63|\n| 1.0  | 62.03 | 51.56 | 54.81|\n\n**Caption**: Model performance on HotpotQA with different sampling temperatures, using Qwen-2.5-7B-instruct as the backbone LLM (with a random seed of 42).\n\nWe found that these initial explorations did not reveal significant differences in final evaluation performance (two-tailed t-test with $\\alpha = 0.05$). We will make more room in our paper and discuss these findings, further enriching the practical insight in understanding ExSearch's optimization landscape.\n\n$  $\n\n# Response to Question 1: Added Training Cost Comparison Between ExSearch and Baselines\n\nThank you for your suggestion. We have included a quantitative analysis of both training and inference costs below.\n\nFor the training cost comparison, we implemented our proposed method, ExSearch, and the baselines (Search-R1) using the same backbone LLM, Qwen-2.5-7B-instruct. All methods utilize ColBERTv2.0 as the retriever, with the number of retrieved documents (top-k) consistently set to 5. The following details are based on experiments conducted by training on the HotpotQA training split and evaluating on the HotpotQA validation split.\n\n1. **Training Cost:** As reported below, we find that ExSearch achieves better performance with fewer training hours.\n\n| Method | Training Hours | Performance (Accuracy) |\n|:--|:--:|:--:|\n| ExSearch           |    8.2       | 54.32 |\n| Search-R1 (PPO)    |  19.8      | 46.51 |\n| Search-R1 (GRPO)   |  16.4    | 46.12 |\n\n**Caption**: The count of training hours of different methods on the HotpotQA dataset. \n\n2. **Inference Cost:** We also report average token consumption and the number of retriever calls per example (e.g., per input query).\n\n|       Method        | Avg. Input Token Consumption | Avg. Output Token Consumption | Retriever Calls | Performance (Accuracy) |\n|:-------------------|:---------------------------:|:----------------------------:|:---------------:|:-----------:|\n| ExSearch            | 1859.98                    | 325.30                      | 2.45            | 54.32       |\n| Search‑R1 (PPO)     | 2059.98                    | 304.21                      | 3.14            | 46.51       |\n| Search‑R1 (GRPO)    | 2351.73                    | 332.12                      | 3.56            | 46.12       |\n\n**Caption**: The statistics of different methods during the inference on the HotpotQA dataset.\n\nWe also observe higher accuracy of ExSearch while consuming fewer tokens and making fewer retriever calls, indicating greater efficiency.\n\n---\n\nOnce again, thank you for your review! We will update the final version of our paper with all the revisions discussed. We will also revise the typo issues you mentioned. We hope our responses address your concerns, and we are happy to answer any further questions you might have.\n\n\n\n$ $\n\n\n**Reference**\n\n[1] Li X, Dong G, Jin J, et al. Search-o1: Agentic search-enhanced large reasoning models[J]. arXiv preprint arXiv:2501.05366, 2025.\n\n[2] Jin B, Zeng H, Yue Z, et al. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning[J]. arXiv preprint arXiv:2503.09516, 2025.\n\n[3] Sheng G, Zhang C, Ye Z, et al. Hybridflow: A flexible and efficient RLHF framework[C]//Proceedings of the Twentieth European Conference on Computer Systems. 2025: 1279-1297."}, {"review_id": "cCZZhRtUK2", "reviewer": "Reviewer_RVTA", "summary": "This paper introduces a novel framework, EXSEARCH, designed to enhance the capabilities of large language models (LLMs) as information retrieval agents through an iterative self-motivated process. The core of this framework enables LLMs to dynamically perform document retrieval, evidence extraction, and answer aggregation, thereby improving their knowledge acquisition in complex tasks via a self-motivated mechanism.\n\nMain Contributions:\n\nProposal of the EXSEARCH Framework: EXSEARCH defines three core actions—“thinking” (query generation), “searching” (triggering the retriever), and “recording” (extracting fine-grained evidence)—allowing LLMs to iteratively explore search trajectories and generate final answers based on the entire trajectory.\nTheoretical Analysis and Convergence Guarantee: The authors provide a theoretical analysis of the training process, demonstrating that the self-motivated framework achieves stable convergence.\nExtensive Experimental Validation: Experiments conducted on multiple knowledge-intensive benchmark datasets show that EXSEARCH significantly outperforms existing retrieval-augmented language models.\nIntroduction of EXSEARCH-Zoo: This extended resource generalizes EXSEARCH to a broader range of scenarios, including diverse backbone LLMs from different model families and scales, as well as an expanded action space (e.g., document re-ranking).\nMethodological Details:\n\nSelf-motivated Training Process: EXSEARCH employs a Generalized Expectation-Maximization (GEM) algorithm, alternating between an E-step (trajectory exploration) and an M-step (reweighted trajectory learning). This enables the LLM to learn from its self-generated data, progressively improving its search and reasoning abilities. In the E-step, the LLM generates multiple search trajectories for each input task and assigns importance weights to them; in the M-step, the LLM updates its parameters based on these trajectories and their associated weights.\n\nExperimental Results: EXSEARCH demonstrates substantial improvements over baseline models across several benchmarks. For example, on the HotpotQA dataset, using the 7B Qwen2.5 model, the F1 score increases from 55.40 (RankRAG-70B) and 53.31 (Search-o1-32B) to 62.59. Additionally, EXSEARCH exhibits superior retrieval performance, with recall rates significantly higher than those achieved by using only retrievers or traditional re-ranking methods.", "questions": "1. Currently, EXSEARCH performs retrieval at every reasoning step, regardless of necessity. It may be beneficial to develop a more adaptive strategy that enables the model to determine, based on contextual information, when retrieval is required.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Self-motivated Learning Mechanism: EXSEARCH adopts a novel self-motivated process, enabling LLMs to iteratively learn to retrieve useful information during reasoning. This internal, adaptive feedback loop guides the learning process without requiring explicit, manually designed intermediate rewards for each agent interaction step.\n\n2. Unique Use of the Generalized EM Algorithm: During the E-step, the LLM generates multiple search trajectories and assigns importance weights to each; in the M-step, it trains using a reweighted loss function based on these trajectories. This iterative process establishes a robust self-motivated loop, allowing the LLM to learn directly from its self-generated and implicitly evaluated data, thereby progressively enhancing its search and reasoning capabilities.\n\n3. Theoretical Analysis and Convergence Guarantee: A significant contribution of the framework is the inclusion of theoretical analysis, establishing convergence guarantees for the training process. This provides a solid mathematical foundation for the framework’s stability, which is often challenging to prove in complex RL-based systems.\n\nWeaknesses:\n\n1. The description of EXSEARCH lacks details regarding its initial training phase and how it addresses the “cold start” problem.\n2. Although theoretically sound, implementing the Generalized EM algorithm in large language model training is more complex than methods such as policy gradient (e.g., GRPO). While the self-motivated training process is theoretically guaranteed to converge, in practice, training stability and efficiency may be affected by data quality and the underlying model.\n3. EXSEARCH addresses irrelevant information by weighting useful information through the EM process; however, more explicit technical solutions are needed to prevent retrieved content from biasing the training strategy. It remains challenging to ensure that the EM process can fully and implicitly handle this issue.", "was_revised": true, "final_justification": "Overall, the approach proposed by EXSEARCH is quite effective. I have no further concerns and rate it a 4.", "rebuttal": "Dear Reviewer,\n\nThank you for your recognition of this work. We truly appreciate your thoughtful reviews and suggestions, which help us improve our work. In response to your review, we have made the following revisions:\n1. Clarified details regarding the cold start process (addressing Weakness 1).\n2. Explained the efficiency and superiority of ExSearch (addressing Weakness 2).\n3. Discussed strategies to prevent noise during the retrieval process (addressing Weakness 3).\n4. Addressed enabling adaptive retrieval (addressing Question 1).\n\n---\n\n$  $\n\n# Response to Weakness 1: Explanation about the Initial Training (Cold Start)\n\nWe appreciate your valuable feedback. We've added the following details to our paper for better clarification.\n\nAs described in Line 186 (implementation details), to address the cold start problem, we initially fine-tune the model with 1000 pseudo-examples annotated by GPT-4o. This aims to initialize the model with the basic skills required to follow ExSearch's iterative think-search-record pattern.\n\nSpecifically, we use GPT-4o to label data that adheres to our agentic framework, ensuring that the generated examples align with the goals of the iterative reasoning process. This process develops the model’s basic capacity to engage in the think-search-record cycle proposed in the ExSearch workflow. These data synthesis details, along with further explanations, have been provided in the ***Data Collection for Warm-Up*** section (Line 1014).\n\n$  $\n\n# Response to Weakness 2: Clarification about Why ExSearch is Efficient \n\nThank you for your thoughtful comment. We fully agree that implementing the Generalized EM (GEM) algorithm for training LLMs presents complexities, particularly regarding training stability and efficiency. However, we would like to emphasize that `we have simplified ExSearch's training into an elegant self-training manner (which has been recognized via reviewers ymBc and zB4g) and have conducted extensive experiments to demonstrate its practical stability and efficiency.`\n\nWe provide the following detailed explanations:\n1. **Simple Iterative Training Process**: We have simplified the training approach through an iterative process, which helps mitigate sampling instability during training. This process, as discussed in the paper, allows for a more stable and controlled learning environment, ensuring consistent progress without the typical challenges associated with sampling-based methods.\n2. **Empirical Validation of Stability and Efficiency**: To validate the stability and efficiency of ExSearch, we conduct experiments on multiple datasets. Across these datasets, we observed consistent, stable improvements in model performance. This supports our claim that the proposed training method is stable across various tasks and data settings.\n3. **Model Variety**: As illustrated in Figure 1 and Table 1 in our submitted paper, we evaluated ExSearch across a variety of LLMs, including smaller and larger models, and observed stable improvements in performance in all cases. This further reinforces that ExSearch scales well and is effective across different architectures.\n4. **Efficiency Comparison**: In terms of training efficiency, we conducted a direct comparison between ExSearch and other leading methods such as Search-R1 (trained via PPO / GRPO), as reported below.\n> | Method | Training Hours | Performance (Accuracy) |\n> |:--|:--:|:--:|\n> | ExSearch           |    8.2       | 54.32 |\n> | Search-R1 (PPO)    |  19.8      | 46.51 |\n> | Search-R1 (GRPO)   |  16.4    | 46.12 |\n>\n>**Caption**: The count of training hours of different methods on the HotpotQA dataset. \n\n$  $ $  $ $  $ $  $ $  $ We find that ExSearch achieves better performance with fewer training hours.\n\n$  $ $  $ $  $ $  $ $  $ We also report average token consumption and the number of retriever calls per example (e.g., per input query).\n\n>|       Method        | Avg. Input Token Consumption | Avg. Output Token Consumption | Retriever Calls | Performance (Accuracy) |\n>|:-------------------|:---------------------------:|:----------------------------:|:---------------:|:-----------:|\n>| ExSearch            | 1859.98                    | 325.30                      | 2.45            | 54.32       |\n>| Search‑R1 (PPO)     | 2059.98                    | 304.21                      | 3.14            | 46.51       |\n>| Search‑R1 (GRPO)    | 2351.73                    | 332.12                      | 3.56            | 46.12       |\n>\n>**Caption**: The statistics of different methods during the inference on the HotpotQA dataset.\n\n$  $ $  $ $  $ $  $ $  $ We observe that ExSearch achieves higher accuracy while consuming fewer tokens and making fewer retriever calls, indicating greater efficiency in both training and inference.\n\nWe believe that these explanations address the concerns regarding the practical implementation of GEM in LLM training, confirming that ExSearch offers a robust and efficient solution for dynamic retrieval and reasoning tasks.\n\n$  $\n\n# Response to Weakness 3: Preventing the Noise in ExSearch\n\nWe appreciate this insightful suggestion. Addressing irrelevant information is a long-standing research challenge in Information Retrieval (IR), involving efforts from many research directions, e.g., document re-ranking or content extraction. In our work, we have considered this by extending the document selection action in ExSearch. As described in Section 7 of our submitted paper, we incorporate a re-ranking step into the reasoning process to address this concern.\n\nSpecifically, in Section 7.1, we extend the original ExSearch workflow from the pattern `think -> search -> record` to `think -> search -> re-rank -> record`. The added re-rank step allows us to re-rank the retrieved documents based on their relevance to the current sub-query. Only the top-k documents, as determined by this ranking process, are then passed into the reasoning trajectory. This approach ensures that only the most relevant and supportive information contributes to the learning process, effectively mitigating the risk of irrelevant content biasing the training strategy. This modification remains fully compatible with the original ExSearch training framework, as it only changes the sampling method. By adding this ranking step, we provide an elegant solution that reduces the bias from irrelevant retrieved information, leading to about a 2.8% to 5.0% point improvement in performance. This change enhances the stability of the training process while preserving the core dynamics of ExSearch.\n\nDebiasing irrelevant information presents a non-trivial challenge in IR. We are actively exploring more advanced technical solutions. Future iterations of ExSearch will include more useful solutions, such as dynamic filtering mechanisms, to better control and reduce the impact of irrelevant content on the model's training.\n\n$  $\n\n# Response to Question 1: Enabling Adaptive Retrieval in ExSearch\n\nThank you for your thoughtful suggestion. Handling **when to retrieve** is a long-standing challenge in IR and RAG (e.g., adaptive retrieval, knowledge-boundary reasoning). A key principle there is letting the model explicitly know what it knows and what it still needs. The current version of ExSearch already follows this principle:\n1. **Think step**: The LLM reasons over the current context and formulates a new query to specify missing information.\n2. **Record step**: It reflects on the usefulness of retrieved documents by extracting key evidence, effectively filtering noise.\n\nBeyond the current version, we agree that integrating additional mechanisms is beneficial. Inspired by adaptive retrieval/reasoning work, we provide the following technical solutions:\n1. Pre-determining the query complexity and then selecting the most fitting retrieval strategies [1, 2]\n2. More explicit need-to-know reasoning before each retrieval call [3, 4].\n3. Reward/penalty on unnecessary retrieval to discourage overuse of the search engine [3].\n4.  Adaptive stopping criteria and budgets for retrieval depth.\n\nThese solutions, particularly the *1. query complexity assessment* and *4. Adaptive stopping criteria and budgets for retrieval depth* can be seamlessly integrated into ExSearch, while the other solutions can be plugged into ExSearch's action loop.\n\nWe will add these discussions to the revised paper to address your concern more concretely. \n\n---\n\n\nOnce again, we really appreciate your review! We will update the final version of our paper with all the revisions discussed. We hope our responses address your concerns, and we would be happy to provide further details on any specific aspect if needed.\n\n$     $\n\n**Reference**\n\n[1] Jeong S, Baek J, Cho S, et al. Adaptive-rag: Learning to adapt retrieval-augmented large language models through question complexity. arXiv 2024[J]. arXiv preprint arXiv:2403.14403.\n\n[2] Mallen A, Asai A, Zhong V, et al. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories[J]. arXiv preprint arXiv:2212.10511, 2022.\n\n[3] Ren B, Qiao S, Yu W, et al. KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality[J]. arXiv preprint arXiv:2506.19807, 2025.\n\n[4] Qiao S, Qiu Z, Ren B, et al. Agentic knowledgeable self-awareness[J]. arXiv preprint arXiv:2504.03553, 2025."}, {"review_id": "f98Nn9MNju", "reviewer": "Reviewer_ymBc", "summary": "This paper presents a theoretical framework for training iterative retrieval-and-generation agents through the lens of the Expectation-Maximization (EM) algorithm. The proposed ExSearch algorithm casts the E-step as the agent generating multiple search queries and receiving rewards based on the quality of retrieved information. The M-step then applies importance sampling over the search trajectories to compute a loss used to update the parameters of the language model. This EM-inspired formulation provides a principled approach to learning from the agent’s interaction with the retrieval environment.", "questions": "1. In the EM-style training algorithm illustrated in Figure 2, the key novelty over previous approaches such as Search-R1 appears to lie in the E-step, where multiple trajectories are sampled, and reward-based weights are used to reweight the updates during training. However, this seems conceptually similar to Group Relative Policy Optimization (GRPO), where a batch of trajectories is used to compute group-level advantages for policy updates. Could the authors clarify the conceptual and algorithmic distinctions between their method and GRPO?\n\n2. As raised in the earlier weaknesses, the term $p(z|x;\\theta)$ represents the model’s distribution over search trajectories $z$. However, it remains unclear how the reward associated with each $z$ is quantified during training. Is there any form of ground-truth reasoning path used to evaluate the quality of $z$, or is the reward computed via another proxy, such as perplexity or task performance? More details on how this reward signal is defined and grounded would be helpful.\n\n3. Lines 270–271 refer to \"1,00\" samples, which seems to be a typographical error. Presumably this is meant to be 1,000 samples used for warm-up. Based on Figure 4, the performance gain from using 1,000 versus 800 samples appears marginal. Could the authors clarify why 1,000 was selected as the final choice, and whether there is a principled or empirical reason behind this threshold?\n\n4. Regarding Table 1, the visual design raises several concerns. Both purple and green are used to highlight \"Ours\" methods, yet the distinction between them is unclear. Moreover, in the green-shaded region, some models outperform the purple-highlighted baseline but are not bolded, which is confusing. Could the authors clarify the meaning behind these color codings and whether the comparison settings (e.g., model size, training data) are strictly comparable across the variants in the table?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n* The proposed framework is elegant and intuitive, offering a principled formulation for iterative retrieval and generation. Its effectiveness is supported by thorough comparisons with a wide range of baselines, including standard LLMs, retrieval-augmented generation (RAG), and existing agentic search methods.\n\n# Weaknesses:\n\n* While the EM-based formulation provides a theoretical foundation, the paper introduces considerable mathematical complexity, e.g., ELBO derivations and convergence analysis, for what is essentially a straightforward multi-step retrieval and generation process. This added formalism may reduce the readability and accessibility of the method.\n\n* Despite the mathematical rigor, the method section lacks sufficient clarity on practical implementation details. For example, it is unclear how key components in the loss functions $L_R$ and $L_A$ are instantiated: such as how $w(z)$ is defined, or how $p(y|x, z; \\theta)$ and $p(z|x; \\theta)$ are computed to optimize the LLM parameters. These ambiguities hinder reproducibility and may leave readers uncertain about the concrete mechanics of the proposed approach.", "was_revised": true, "final_justification": "I have read the authors’ response and appreciate the clarifications provided. The rebuttal addresses some of my concerns and offers additional detail, particularly regarding the technical formulation. I'll raise the score.", "rebuttal": "Dear Reviewer,\n\nThank you for your recognition of this work. We truly appreciate your thoughtful reviews and suggestions, which have helped us improve our work. In response to your feedback, we have made the following revisions:\n1. Simplified mathematical derivations (addressing Weakness 1).\n2. Provided a clearer explanation of the key components in our loss function (addressing Weakness 2 and Question 2).\n3. Compared our method with GRPO (addressing Question 1).\n4. Clarified the scale of warm-up training data (addressing Question 3).\n5. Explained the color coding in Table 1 (addressing Question 4).\n\n---\n\n$     $\n\n# Response to Weakness 1: Theoretical Analysis\n\nWe fully understand the concern that the mathematical derivation and analysis may introduce complexity, and **we plan to strike a better balance**.\n\n**Why do we provide a theoretical analysis?** The introduction of the Generalized EM (GEM) framework is essential for ensuring the theoretical soundness and convergence guarantees of ExSearch (as also recognized by reviewer RVTA). We aim to provide an explainable foundation for our method (e.g., how & why it works) for readers and distinguish it from simpler multi-step retrieval processes (particularly those based on prompt engineering). We hope this ensures the long-term convergence and performance across different retrieval scenarios of our method.\n\n**We will further simplify the mathematical derivation.** We agree with your suggestion. In future iterations of the paper, we plan to simplify some of the mathematical details and present the key ideas in a more intuitive and accessible way. This will include providing clearer preliminary explanations and potentially reformatting the formalism to focus on the core insights without losing rigor. We also plan to streamline the ELBO derivations and convergence analysis to make them more digestible while maintaining theoretical guarantees.\n\nWe hope these changes will address your concerns.\n\n$       $\n\n# Response to Weakness 2 and Question 2: Further Illustrating the Loss Function and Trajectory Weight\n\nWe appreciate the comment and clarify below how each quantity is instantiated in practice (see Eq. (6) in the paper).\n\n1. **Probability of a search trajectory $p(z \\mid x)$**: A trajectory $z=$ {$(x_i, d_i, e_i) \\mid i\\in [|z|]$} consists of the *think* (query $x_i$), *search* (retrieved document $d_i$) and *record* (evidence $e_i$) actions.\n\n$      $ $      $ $      $ $      $ $      $ $      $ Thus, $p(z \\mid x; \\theta) = \\prod_{i=1}^{|z|} p(x_i \\mid x, z_{<i}; \\theta)\\; p(e_i \\mid x_i, d_i; \\theta)$, where both factors are standard token-level LM likelihoods (sum of log-probs over generated tokens) computed by the current model $\\theta$.\n\n2. **Answer likelihood conditioned on the trajectory $p(y \\mid x; z)$**: \nWe compute the probability of the model in generating the ground-truth answer y: \n$p(y \\mid x, z; \\theta) = \\prod_{t=1}^{|y|} p(y_t \\mid x, z, y_{<t}; \\theta)$,\nwhich is the LLM token log-likelihood under $\\theta$ with the entire trajectory $z$ in context.\n\n3. **Trajectory importance weight $w(z)$**: Using importance sampling (Eq. (5)), the unnormalized weight is $\\tilde{w}(z) \\propto p(y \\mid x, z; \\theta_t)$, and we normalize over the sampled trajectories $w(z) = \\frac{\\\\tilde{w}(z)}{\\\\sum_{j=1}^{M} \\\\tilde{w}(z^{j})}$.\n\n4. **Loss instantiation**: Given sampled trajectories $z$, weights $w(z)$, and the LLM parameter $\\theta$, we have:\n- Reasoning loss $L_{\\mathcal{R}} = - \\sum_{m} w(z) \\Big( \\sum_i \\log p(x_i \\mid x, z_{<i};\\theta ) + \\log p(e_i \\mid x_i, d_i;\\theta) \\Big)$, as introduced in Line 88 and Line 129 of the submitted paper;\n- Answer loss $L_{\\mathcal{A}} = - \\sum_{m} w(z) \\log p(y \\mid x, z; \\theta)$, as introduced in Line 131 of the submitted paper;\n\n$      $ $      $ $      $ $      $ $      $ $      $ The final objective is $L = L_\\mathcal{R} + L_\\mathcal{A}$. We backpropagate through the LLM as usual to update the model parameters.\n\n> Mini example (conceptual)\n> - Input $x$: “Who wrote *The Old Man and the Sea* and in which year did it win the Pulitzer Prize?”  \n> - Two sampled trajectories $z^{1}$, $z^{2}$ differ in sub-queries and evidence snippets.  \n> - Compute token-level log-likelihoods for each $x_i, e_i$, and for the final answer $y$.  \n> - Normalize $\\tilde{w}(z^{1}), \\tilde{w}(z^{2})$ to get $w(z^{1}), w(z^{2})$.  \n> - We sum the weighted losses $L_\\mathcal{R}$ and $L_\\mathcal{A}$, and update $\\theta$.\n\n\n$      $\n\n# Response to Question 1: Discussing Latent Variable Modeling with GRPO\n\nThe key differences between our method and previous work, such as Search-R1, are as follows, with more details provided in Appendix C.1:\n\n1. **Reward Calculation**: Our approach calculates the reward from the log-likelihood of the generated answer, rather than relying on an exact match score between the predicted answer and the ground truth.\n2. **Trajectory-level Reward Supervision vs. Outcome Reward Supervision**: As shown in Eq. 10 and Appendix Line 743, our weight w(z) evaluates the entire search trajectory’s contribution to generating the answer, rather than focusing solely on the final outcome. The results in the main experiment (Table 1) and the analysis experiments in D.3.4 (Line 919) demonstrate that this more global supervision signal yields better performance.\n3. **Latent Variable Modeling vs. Reinforcement Learning**: Latent variable optimization and reinforcement learning optimization are two distinct paradigms. Our ExSearch models search behavior as latent variables and use an iterative process with maximum likelihood estimation to infer the distribution of latent variables and optimal parameters. In contrast, GRPO focuses on online policy optimization, updating model parameters via policy gradients.\n\nWe will incorporate these clarifications into the revised version of the paper.\n\n\n\n$   $\n\n# Response to Question 3: Explanation about Warmup Training Data Scale\n\nWe appreciate this thoughtful review. Below, we respectfully clarify any misunderstanding about our warm-up training data scale and analysis in Section 5.5.\n\n**First, the choice of 1,000 samples in this experiment was primarily driven by cost considerations.** Training with too many samples increases costs and can reduce the scalability of the method. `Therefore, we initially selected 1,000 samples for training (at a cost of $20) and conducted extensive analysis to compare model performance across different initial training data sizes.`\n\n**Second, it is indeed 100 instead of a typographical error.** In Section 5.5 (the paragraph containing lines 270–271), we aim to investigate the impact of initial warm-up on final training performance and empirically demonstrate the minimal data scale to enable the proposed self-training. Thus, we gradually reduced the number of warm-up training samples and compared the final training performance. We found that the model initially trained on 1,000 samples achieved an exact match of 50.35, while no warm-up only led to around 45. However, `the model trained on only 100 samples achieved around 48, outperforming strong baselines and striking an empirical balance between data creation cost and performance.` Thus, we highlight 100 examples in Lines 270–271.\n\nOverall, the focus of this paper is on constructing a foundational agentic search framework rather than optimizing the warm-up training scale. Thus, we initially set the data scale to 1,000 and then gradually reduced the scale for fine-grained validation and comparison. In the revised version of our paper, we will make room to add this content for explanation.\n\n\n$   $\n\n# Response to Question 4: Why Use Two Color Codings\n\nWe appreciate this feedback regarding the visual design and clarity of Table 1. The distinctions between the purple and green color codings are intended to highlight different types of experiments, and we will clarify these in the revised version of the paper.\n\nSpecifically, due to space constraints in the submitted paper, we combined both the main experimental results and the analysis experiments into a single table (Table 1). To clearly differentiate between the two types of experiments, we use different color codings:\n\n1. **Upper Section with Green (Main Experiment)**: The upper half of the table presents the performance comparison between our method and previous baselines, which we refer to as the main experiments. In this section, we highlight the results for \"Ours\" in purple to indicate the primary comparison.\n\n2. **Lower Section with Green (Analysis Experiment)**: The lower half of the table showcases the analysis experiments to demonstrate the generalizability of our method when applied to different backbone LLMs. These experiments are intended to show that our approach works effectively across multiple model families with varying parameters. To differentiate this section from the main experiment, we highlight \"Ours\" in green.\n\nIn the revised version, we will make room to separate Table 1 into two distinct tables, providing a clearer presentation of our experimental results.\n\n\n\n$   $\n\n---\n\nOnce again, we really appreciate your review! We will update the final version of our paper with all the revisions discussed. We hope our responses address your concerns, and we would be happy to provide further details on any specific aspect if needed.\n\n$   $\n\n**Reference**\n\n[1] Jin B, Zeng H, Yue Z, et al. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning[J]. arXiv preprint arXiv:2503.09516, 2025."}, {"review_id": "vHGmCiiZB8", "reviewer": "Reviewer_XR4h", "summary": "This paper explores the emerging and promising area of agentic search, where language agents interact with search environments through multiple rounds of decision-making. The proposed method, ExSearch, formulates the search process as an iterative EM-based optimization, leveraging trajectory sampling and self-incentivized refinement.", "questions": "see above", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The topic of agentic search is timely and important. The paper attempts to move beyond static or single-pass search paradigms by allowing agents to explore search trajectories iteratively.\n- The use of EM formulation provides a principled perspective on optimizing the latent trajectories, and the effort to connect it to self-incentivized learning is appreciated.\n\nWeakness:\n\n- The core idea of ExSearch as a multi-round interaction is essentially a workflow definition for iterative search. However, the approach lacks the adaptiveness and flexibility typically expected in agentic behavior. Compared with methods like Search-R1, which perform action planning in a test-time reasoning loop, ExSearch appears relatively rigid and less autonomous in decision-making.\n- While the EM-based optimization is well-written, it essentially reduces to importance sampling over trajectories, followed by distribution fitting. This is conceptually similar to ideas in actor-critic RL and related methods, where EM-like steps are implicitly embedded. Thus, although the derivation is mathematically sound, it does not offer a compelling methodological innovation.\n- A critical flaw in the experimental design lies in the lack of consistency across backbone LLMs. Many baselines in Table 1 use different language models, making direct comparison problematic. For a fair and scientific evaluation, it is crucial to control such variables, including the retriever, top-k settings etc. The current setup risks conflating performance gains with model differences.\n- The datasets used (e.g., HotpotQA) are relatively old and may have already been seen during LLM pretraining, weakening the evidence for generalization. Given the agentic search setting, it would be more convincing to include challenging and realistic benchmarks like GAIA or WebWalker, which require true multi-step reasoning and adaptive behavior.\n\nMinor typos:\n\nline 112: $t$th -> $t$-th\n\nline 160: a blank square in the right side", "was_revised": true, "final_justification": "After reading the author’s response, some of my concerns have been addressed. However, I still remain uncertain about the central focus of the paper, as a substantial portion is devoted to explaining the EM optimization steps, despite the authors’ claim that “the novelty lies more in the agentic search process and its self-improving loop.” Given these considerations, I believe my previous assessment remains a fair evaluation.", "rebuttal": "Dear Reviewer,\n\nThank you for your recognition of this work. We truly appreciate your thoughtful reviews and suggestions, which have helped us improve our work. In response to your feedback, we have made the following revisions:\n1. Explained the flexibility and extendability of ExSearch (addressing Weakness 1).\n2. Clarified the role of the Importance sample and the core contribution (addressing Weakness 2).\n3. Clarified the experiment setup (addressing Weakness 3).\n4. Added new experiments on GAIA and WebWalker datasets (addressing Weakness 4).\n\n\n---\n\n# Response to Weakness 1: ExSearch is Flexible and Extensible\n\nWe respectfully point out that ExSearch provides a foundational and flexible framework for the model to learn to retrieve useful information through self-incentivization. The current version of our workflow includes, but is not limited to, the Search-R1 [1] workflow, and is designed to be extensible.\n\nSpecifically, Search-R1 iteratively formulates new queries based on previously retrieved documents. This capability is already integrated into our method, where the LLMs continuously reflect on what additional information needs to be retrieved and generate a query accordingly. Furthermore, our actions also include reflecting on the quality of documents and ranking the retrieved documents. Both are key primitive actions fully compatible with the ExSearch action planning loop. Our ablation experiments in Section 5.2 further demonstrate the necessity of these actions.\n\nAdditionally, we would like to note that Search-R1 was officially released on March 12, making it a concurrent work with ours according to NeurIPS' review guidelines [2]. In summary, we believe our workflow is compatible with Search-R1, and we present a novel and extensible framework.\n\n$    $\n\n# Response to Weakness 2: Clarify the Role of Importance Sample\n\nThank you for the feedback. Importance sampling [3,4] is a basic and essential technique used in various optimization frameworks, like reinforcement learning or probabilistic statistical frameworks. We would like to clarify that the primary role of importance sampling in our work is to simplify the computation of the search trajectory likelihood, rather than serving as the core contribution.\n\nThe key contributions of this work lie in the **ExSearch exploratory search framework**, which includes:\n1. An exploratory search framework that empowers LLMs to engage in agentic retrieval through a self-incentivization process. This allows the model to iteratively refine its search strategy, selecting and reasoning over the most relevant information.\n2. A theoretical foundation for this framework, which provides guarantees for convergence and demonstrates how our approach can be extended to more general scenarios, enhancing the flexibility and scalability of the method.\n\n> Besides, **the training algorithm in ExSearch diverges from actor-critic by**:\n> 1. Latent Variable Modeling: Latent variable optimization and reinforcement learning optimization are two distinct paradigms. Our ExSearch models search behavior as latent variables and uses an iterative process with maximum likelihood estimation to infer the distribution of latent variables and optimal parameters. Actor-Critic, in contrast, focuses on policy optimization in reinforcement learning, where the goal is to maximize cumulative rewards.\n> 2. Reward Calculation: Our approach calculates the reward from the log-likelihood of the generated answer, which emphasizes probabilistic inference instead of rewards provided by a critic model or the environment.\n\n\nWe believe the novelty lies more in the agentic search process and its self-improving loop (which has been recognized by reviewers zB4g, RVTA, and ymBc), rather than in the specific application of importance sampling. We will clarify these points in the revised version of the paper to better highlight the methodological innovations in our approach.\n\n$    $\n\n# Response to Weakness 3: Our Experiments Include Both Open-Source and Closed-Source Models Under the Same Retriever\n\nThanks for this valuable suggestion. We agree that keeping experimental settings consistent across baselines is crucial for fair comparison, and this principle guided our design. We believe there might have been a misunderstanding.\n\n### **1. Consistent Experimental Setup**\nAs described in Section 4 (Line 185), all methods in Table 1 use the same retriever, ColBERTv2.0. Furthermore, Line 178 specifies that the number of retrieved documents is fixed across baselines: we follow common practice by using 10 documents for advanced RAG baselines and 5 for iterative retrieval RAG baselines (the same counts used for ExSearch).\n\n### **2. LLM Backbones**\nRAG research includes many baselines; some rely on closed-source models or don't release code, as explained in Line 181 of our paper. Therefore, our primary focus is on using smaller, open-source backbones and comparing against baselines with equivalent or even stronger backbones. Here are two examples.\n\n1. Both Search-R1 and ExSearch use Qwen-2.5-7B-instruct, yet ExSearch consistently performs better, as shown below.\n\n>| Method    | NQ (F1 / EM / ACC) | HotpotQA (F1 / EM / ACC) | MuSiQue (F1 / EM / ACC) | 2WikiQA (F1 / EM / ACC) |\n>|:-----------|:--------------------:|:--------------------------:|:-------------------------:|:-------------------------:|\n>| Search-R1 | 54.26 / 42.21 / 51.35 | 58.04 / 46.51 / 52.46 | 30.03 / 21.21 / 23.37 | 52.62 / 49.64 / 50.43 |\n>| ExSearch  | 56.37 / 47.07 / 51.75 | 62.59 / 50.35 / 54.32 | 29.68 / 22.03 / 24.34 | 57.14 / 52.62 / 54.37 |\n>\n> **Caption:** The results from Table 1 in our submitted paper.  All these methods use `Qwen-2.5-7B-instruct` as the backbone. EM refers to the exactly match;  ACC refers to the accuracy.\n>\n>| Method        | NQ (F1 / EM / ACC) | HotpotQA (F1 / EM / ACC) | MuSiQue (F1 / EM / ACC) | 2WikiQA (F1 / EM / ACC) |\n>|:---------------|:--------------------:|:--------------------------:|:-------------------------:|:-------------------------:|\n>| InstructRAG   | 39.21 / 37.82 / 37.58 | 37.31 / 36.77 / 35.31 | 25.88 / 14.94 / 20.45 | 40.01 / 44.57 / 38.91 |\n>| RAG-DDR       | 40.74 / 28.76 / 30.51 | 31.71 / 40.04 / 42.41 | 13.54 / 10.57 / 14.21 | 38.40 / 35.44 / 37.41 |\n>| ExSearch      | 55.21 / 43.71 / 50.76 | 60.72 / 47.59 / 53.59 | 30.83 / 20.98 / 24.65 | – / – / –              |\n>\n>**Caption:** The results from Table 1 in our submitted paper.  All these methods use `Llama-3.1-8B-instruct` as the backbone. EM refers to the exactly match;  ACC refers to the accuracy.\n\n2. While DSPy is officially implemented only with the powerful GPT-3.5, ExSearch still surpasses its performance using the smaller, open-source Qwen-2.5-7B-instruct. \n\nBesides the above two experimental comparisons, Table 1 includes more comparisons between ExSearch and previous baselines using the same backbone LLMs under the same retrieval setting. The experiment as well as the experimental results of this work have also been acknowledged by reviewers zB4g, RVTA, and ymBc, showing the effectiveness of our method.\n\nWe will further clarify these experimental settings in the revised paper to ensure no ambiguity.\n\n\n$     $ \n\n# Response to Weakness 4: Why Existing Datasets and Newly Added Experiment\n\nThank you for the suggestion. Below, we explain why we use existing datasets and report results from additional experiments following your suggestion.\n\n### **1. Reason for Our Experimental Benchmarks:**  \n\nWe chose datasets such as HotpotQA to maintain consistency with prior baselines. The **base models, experimental setup, and datasets** strictly follow well-known baselines and previous work. This consistency enables direct, fair comparison and shows that ExSearch outperforms existing methods under the same conditions.\n\n### **2. New Experiments on GAIA and WebWalker:**  \n\nWe agree that more challenging benchmarks strengthen validation. Following your suggestion, we additionally evaluate on WebWalker and GAIA, two more realistic, multi-step benchmarks. The results are reported as follows.\n\n\n| Method  | GAIA (Accuracy %) | WebWalker (Accuracy %) |\n|:-----------------------|:---------:|:-----:|\n| Search-R1 + GRPO   |  23.30  | 22.06    |\n| Search-R1 + PPO     | 25.24 | 23.24 | \n| ExSearch | 27.18 | 25.59 |\n| ExSearch (w/ re-ranking in Section 6) | 28.16  | 26.03 |\n\n**Caption:** We use the official metric, i.e., accuracy, from GAIA and WebWalker, respectively. We use the Qwen-2.5-7B-instruct as the backbone LLMs for all methods and use the Google search as the retriever.\n\nOur results show that ExSearch achieves an accuracy of 25.59  in the WebWalker dataset, while the baseline, e.g., Search-R1, only reaches 23.24  with a statistically significant difference (two-tailed t-test under $\\alpha=0.05$). This demonstrates the advantage of our method in a more realistic and complex benchmark that requires multi-step reasoning and adaptive behavior.\n\nWe will include these results in the revised version of the paper, along with additional analysis on WebWalker, to further validate the advantage of our method on more challenging datasets. We hope this additional experimental evidence addresses your concern and strengthens the case for the effectiveness of ExSearch.\n\n\n---\n\nOnce again, thank you for your review! We will update the final version of our paper with all the revisions discussed. We hope our responses address your concerns, and we are happy to answer any further questions.\n\n\n$     $ \n\n$     $ \n\n**Reference**\n\n[1]  Jin B, Zeng H, Yue Z, et al. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning[J]. arXiv preprint arXiv:2503.09516, 2025.\n\n[2] https://neurips.cc/Conferences/2025/CallForPapers\n \n[3] Ishop C M, Nasrabadi N M. Pattern recognition and machine learning[M]. New York: Springer, 2006.\n\n[4] Surya T Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinary\nReviews: Computational Statistics, 2, 2010."}]

Author rebuttal:

Meta-review: This paper proposes ExSearch, a framework that formulates iterative retrieval for LLMs as an EM-based optimization problem. The search process is cast as latent trajectories (think, search, record), optimized via iterative self-incentivization. The method combines trajectory sampling, importance weighting, and reweighted loss updates, offering both a practical agentic retrieval system and a theoretical convergence guarantee.
Extensive experiments across multiple datasets and model families show consistent improvements over strong RAG baselines. Additional results on GAIA and WebWalker further support generalization to more realistic, multi-step reasoning environments. Ablations, case studies, and efficiency comparisons provide further evidence of robustness.

Overall, the paper presents a principled, well-executed, and empirically validated framework for iterative retrieval in LLMs. The EM-based latent trajectory formulation offers both elegance and practical gains, with efficiency advantages over reinforcement learning–based approaches.

While there are valid critiques regarding novelty (XR4h) and overemphasis on mathematical formalism (zB4g), the majority of reviewers and AC agreed that the contribution is significant and relevant to the NeurIPS audience. The rebuttal effectively addressed concerns by clarifying baselines, adding GAIA and WebWalker results, and explaining theoretical vs. practical aspects more clearly. However, AC agrees the concern that the experiments all used on older datasets (e.g., HotpotQA), limiting claims of generalization. So I encourage authors to include a complete experimental table during camera-ready on GAIA and WebWalker results.
Decision: Accept (poster)