Title: Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers
Venue: NeurIPS 2025
Accepted: yes
Keywords: information retrieval

large language model

retrieval-augmented generation
Abstract: Large language models (LLMs) have been widely integrated into information retrieval to advance traditional techniques. However, effectively enabling LLMs to seek accurate knowledge in complex tasks remains a challenge due to the complexity of multi-hop queries as well as the irrelevant retrieved content. To address these limitations, we propose ExSearch, an agentic search framework, where the LLM learns to retrieve useful information as the reasoning unfolds through a self-incentivized process. At each step, the LLM decides what to retrieve (thinking), triggers an external retriever (search), and extracts fine-grained evidence (recording) to support next-step reasoning.  To enable LLM with this capability, we adopts a Generalized Expectation-Maximization algorithm. In the E-step, the LLM generates multiple search trajectories and assigns an importance weight to each; the M-step trains the LLM on them with a re-weighted loss function. This creates a self-incentivized loop, where the LLM iteratively learns from its own generated data, progressively improving itself for search. We further theoretically analyze this training process, establishing convergence guarantees. Extensive experiments on four knowledge-intensive benchmarks show that ExSearchS substantially outperforms baselines, e.g., +7.8% improvement on exact match score. Motivated by these promising results, we introduce ExSearch-Zoo, an extension that extends our method to broader scenarios, to facilitate future work.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the emerging and promising area of agentic search, where language agents interact with search environments through multiple rounds of decision-making. The proposed method, ExSearch, formulates the search process as an iterative EM-based optimization, leveraging trajectory sampling and self-incentivized refinement.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The topic of agentic search is timely and important. The paper attempts to move beyond static or single-pass search paradigms by allowing agents to explore search trajectories iteratively.\n- The use of EM formulation provides a principled perspective on optimizing the latent trajectories, and the effort to connect it to self-incentivized learning is appreciated.\n\nWeakness:\n\n- The core idea of ExSearch as a multi-round interaction is essentially a workflow definition for iterative search. However, the approach lacks the adaptiveness and flexibility typically expected in agentic behavior. Compared with methods like Search-R1, which perform action planning in a test-time reasoning loop, ExSearch appears relatively rigid and less autonomous in decision-making.\n- While the EM-based optimization is well-written, it essentially reduces to importance sampling over trajectories, followed by distribution fitting. This is conceptually similar to ideas in actor-critic RL and related methods, where EM-like steps are implicitly embedded. Thus, although the derivation is mathematically sound, it does not offer a compelling methodological innovation.\n- A critical flaw in the experimental design lies in the lack of consistency across backbone LLMs. Many baselines in Table 1 use different language models, making direct comparison problematic. For a fair and scientific evaluation, it is crucial to control such variables, including the retriever, top-k settings etc. The current setup risks conflating performance gains with model differences.\n- The datasets used (e.g., HotpotQA) are relatively old and may have already been seen during LLM pretraining, weakening the evidence for generalization. Given the agentic search setting, it would be more convincing to include challenging and realistic benchmarks like GAIA or WebWalker, which require true multi-step reasoning and adaptive behavior.\n\nMinor typos:\n\nline 112: $t$th -> $t$-th\n\nline 160: a blank square in the right side\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical framework for training iterative retrieval-and-generation agents through the lens of the Expectation-Maximization (EM) algorithm. The proposed ExSearch algorithm casts the E-step as the agent generating multiple search queries and receiving rewards based on the quality of retrieved information. The M-step then applies importance sampling over the search trajectories to compute a loss used to update the parameters of the language model. This EM-inspired formulation provides a principled approach to learning from the agent’s interaction with the retrieval environment.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n* The proposed framework is elegant and intuitive, offering a principled formulation for iterative retrieval and generation. Its effectiveness is supported by thorough comparisons with a wide range of baselines, including standard LLMs, retrieval-augmented generation (RAG), and existing agentic search methods.\n\n# Weaknesses:\n\n* While the EM-based formulation provides a theoretical foundation, the paper introduces considerable mathematical complexity, e.g., ELBO derivations and convergence analysis, for what is essentially a straightforward multi-step retrieval and generation process. This added formalism may reduce the readability and accessibility of the method.\n\n* Despite the mathematical rigor, the method section lacks sufficient clarity on practical implementation details. For example, it is unclear how key components in the loss functions $L_R$ and $L_A$ are instantiated: such as how $w(z)$ is defined, or how $p(y|x, z; \\theta)$ and $p(z|x; \\theta)$ are computed to optimize the LLM parameters. These ambiguities hinder reproducibility and may leave readers uncertain about the concrete mechanics of the proposed approach.\n\nQUESTIONS:\n1. In the EM-style training algorithm illustrated in Figure 2, the key novelty over previous approaches such as Search-R1 appears to lie in the E-step, where multiple trajectories are sampled, and reward-based weights are used to reweight the updates during training. However, this seems conceptually similar to Group Relative Policy Optimization (GRPO), where a batch of trajectories is used to compute group-level advantages for policy updates. Could the authors clarify the conceptual and algorithmic distinctions between their method and GRPO?\n\n2. As raised in the earlier weaknesses, the term $p(z|x;\\theta)$ represents the model’s distribution over search trajectories $z$. However, it remains unclear how the reward associated with each $z$ is quantified during training. Is there any form of ground-truth reasoning path used to evaluate the quality of $z$, or is the reward computed via another proxy, such as perplexity or task performance? More details on how this reward signal is defined and grounded would be helpful.\n\n3. Lines 270–271 refer to \"1,00\" samples, which seems to be a typographical error. Presumably this is meant to be 1,000 samples used for warm-up. Based on Figure 4, the performance gain from using 1,000 versus 800 samples appears marginal. Could the authors clarify why 1,000 was selected as the final choice, and whether there is a principled or empirical reason behind this threshold?\n\n4. Regarding Table 1, the visual design raises several concerns. Both purple and green are used to highlight \"Ours\" methods, yet the distinction between them is unclear. Moreover, in the green-shaded region, some models outperform the purple-highlighted baseline but are not bolded, which is confusing. Could the authors clarify the meaning behind these color codings and whether the comparison settings (e.g., model size, training data) are strictly comparable across the variants in the table?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework, EXSEARCH, designed to enhance the capabilities of large language models (LLMs) as information retrieval agents through an iterative self-motivated process. The core of this framework enables LLMs to dynamically perform document retrieval, evidence extraction, and answer aggregation, thereby improving their knowledge acquisition in complex tasks via a self-motivated mechanism.\n\nMain Contributions:\n\nProposal of the EXSEARCH Framework: EXSEARCH defines three core actions—“thinking” (query generation), “searching” (triggering the retriever), and “recording” (extracting fine-grained evidence)—allowing LLMs to iteratively explore search trajectories and generate final answers based on the entire trajectory.\nTheoretical Analysis and Convergence Guarantee: The authors provide a theoretical analysis of the training process, demonstrating that the self-motivated framework achieves stable convergence.\nExtensive Experimental Validation: Experiments conducted on multiple knowledge-intensive benchmark datasets show that EXSEARCH significantly outperforms existing retrieval-augmented language models.\nIntroduction of EXSEARCH-Zoo: This extended resource generalizes EXSEARCH to a broader range of scenarios, including diverse backbone LLMs from different model families and scales, as well as an expanded action space (e.g., document re-ranking).\nMethodological Details:\n\nSelf-motivated Training Process: EXSEARCH employs a Generalized Expectation-Maximization (GEM) algorithm, alternating between an E-step (trajectory exploration) and an M-step (reweighted trajectory learning). This enables the LLM to learn from its self-generated data, progressively improving its search and reasoning abilities. In the E-step, the LLM generates multiple search trajectories for each input task and assigns importance weights to them; in the M-step, the LLM updates its parameters based on these trajectories and their associated weights.\n\nExperimental Results: EXSEARCH demonstrates substantial improvements over baseline models across several benchmarks. For example, on the HotpotQA dataset, using the 7B Qwen2.5 model, the F1 score increases from 55.40 (RankRAG-70B) and 53.31 (Search-o1-32B) to 62.59. Additionally, EXSEARCH exhibits superior retrieval performance, with recall rates significantly higher than those achieved by using only retrievers or traditional re-ranking methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Self-motivated Learning Mechanism: EXSEARCH adopts a novel self-motivated process, enabling LLMs to iteratively learn to retrieve useful information during reasoning. This internal, adaptive feedback loop guides the learning process without requiring explicit, manually designed intermediate rewards for each agent interaction step.\n\n2. Unique Use of the Generalized EM Algorithm: During the E-step, the LLM generates multiple search trajectories and assigns importance weights to each; in the M-step, it trains using a reweighted loss function based on these trajectories. This iterative process establishes a robust self-motivated loop, allowing the LLM to learn directly from its self-generated and implicitly evaluated data, thereby progressively enhancing its search and reasoning capabilities.\n\n3. Theoretical Analysis and Convergence Guarantee: A significant contribution of the framework is the inclusion of theoretical analysis, establishing convergence guarantees for the training process. This provides a solid mathematical foundation for the framework’s stability, which is often challenging to prove in complex RL-based systems.\n\nWeaknesses:\n\n1. The description of EXSEARCH lacks details regarding its initial training phase and how it addresses the “cold start” problem.\n2. Although theoretically sound, implementing the Generalized EM algorithm in large language model training is more complex than methods such as policy gradient (e.g., GRPO). While the self-motivated training process is theoretically guaranteed to converge, in practice, training stability and efficiency may be affected by data quality and the underlying model.\n3. EXSEARCH addresses irrelevant information by weighting useful information through the EM process; however, more explicit technical solutions are needed to prevent retrieved content from biasing the training strategy. It remains challenging to ensure that the EM process can fully and implicitly handle this issue.\n\nQUESTIONS:\n1. Currently, EXSEARCH performs retrieval at every reasoning step, regardless of necessity. It may be beneficial to develop a more adaptive strategy that enables the model to determine, based on contextual information, when retrieval is required.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the design of an LLM-based information retrieval method enhanced with the Expectation-Maximization (EM) algorithm. To accommodate more complex query scenarios, it models multi-turn, autoregressive retrieval reasoning paths—including generated sub-queries, retrieved documents, and accumulated evidence—as latent variables. Using the EM algorithm, it optimizes the expected log-likelihood of the joint distribution $\\mathbb{E}_{\\text{reasoning path}} \\left[  P(\\text{reasoning path}, \\text{answer} \\mid \\text{query}) \\right]$. Compared to directly optimizing the log-likelihood of the answer given a query, this approach enables more effective exploration of reasoning paths, resulting in more accurate and interpretable answers for challenging information retrieval tasks. This paper also conduct extensive experiments across multiple datasets and foundation LLMs, benchmarking against a diverse set of strong baselines. Detailed ablation studies, in-depth analysis, and case studies further demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* **Strength 1:**  The core idea of this paper is to treat the retrieval reasoning path as an unobserved latent variable in an autoregressive manner and optimize it through iterative self-incentivization using the Expectation-Maximization (EM) algorithm. The integration of EM-based modeling is both conceptually sound and practically elegant—resulting in a streamlined algorithm that is not only effective but also insightful. This framework offers a flexible foundation: by tailoring the latent variable assumptions to specific application scenarios, corresponding EM algorithms can be readily designed and adapted.\n* **Strength 2:**  The paper is clearly written and well-organized, with clean and informative figures and tables. The presentation is easy to follow, making the content highly accessible to readers. Furthermore, the appendix provides a thorough discussion of the paper’s limitations, ethical considerations, and societal impacts, along with detailed analyses and commentary on both successful and failure cases.\n* **Strength 3:**  The paper conducts extensive experiments across multiple datasets and foundation LLMs, benchmarking against a wide range of strong baselines. Comprehensive ablation studies, in-depth analyses, and carefully examined case studies further validate the effectiveness and robustness of the proposed approach.\n\n\n**Weaknesses:**\n\n* **Weakness 1:**  The core contribution of this paper lies in integrating the EM algorithm with LLM-based information retrieval. However, the current title—*“Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers”*—does not reflect either the focus on information retrieval or the role of the EM algorithm. In fact, the phrase *“Agentic Searchers”* initially led me to believe the paper was about autonomous agent architecture design, similar to works like *“Automated Design of Agentic Systems”* [1] and *“AgentSquare: Automatic LLM Agent Search in Modular Design Space”* [2]. Given that the experimental setup primarily benchmarks against information retrieval datasets and RAG baselines—and does not compare with other agent search methods (e.g., [1], [2]) or broader agent scenarios (e.g., embodied agents, game environments, tool use)—emphasizing \"information retrieval\" in the title would help clarify the paper’s true focus at first glance and better align with the experimental scope.\n* **Weakness 2:** The theoretical contribution emphasized in the main text—namely, the convergence of the proposed algorithm—does not appear to leverage any domain-specific properties of information retrieval. To my knowledge, it largely reiterates the standard convergence behavior of the EM algorithm with respect to the value of the loss function, which has already been thoroughly established in classical texts such as PRML [3]. This discussion may be more appropriately placed in the preliminaries section. Moreover, the Pinsker’s inequality derivations in the appendix could benefit from clearer attribution—specifically, which parts are novel contributions of this paper. It is also worth noting that the convergence of the EM algorithm in terms of loss value does not imply convergence to a global optimum; it merely guarantees convergence to a stationary point. Given that loss functions in information retrieval are often ReLU-based and may lack strong convexity, the EM algorithm is unlikely to exhibit global convergence in this context. Exploring the impact of different initialization points might provide a more informative and realistic characterization of the model’s convergence behavior.\n\n\n**Minor grammatical issues:**\n\n- [1] Line 38: “an LLM” → consider changing to “a LLM”.\n- [2] Line 112: “$t$th” → might be written as “$t$-th”.\n\n[1] Hu S, Lu C, Clune J. Automated design of agentic systems[J]. ICLR, 2025.\n\n[2] Shang Y, Li Y, Zhao K, et al. Agentsquare: Automatic llm agent search in modular design space[J]. ICLR, 2025.\n\n[3] Bishop C M, Nasrabadi N M. Pattern recognition and machine learning[M]. New York: springer, 2006.\n\nQUESTIONS:\n**Question 1:**\n I’m curious about how EXSEARCH compares to other baselines in terms of training and inference efficiency. Some quantitative insights or comparisons in this regard would be helpful for understanding its practical applicability.\n\n**Question 2:**\n Given the discussion in Weakness 2 about the EM algorithm's local convergence properties, it would be valuable to understand how EXSEARCH behaves under different initialization strategies—such as varying prompt templates or parameter initializations. Has the robustness of convergence under these variations been explored?\n\n**Question 3:**\n As noted in Weakness 1, the term *“Iterative Self-Incentivization”* seems to reflect the optimization dynamics introduced by the EM algorithm. At the same time, EXSEARCH could reasonably be seen as a form of “agentic search” within the information retrieval domain. This raises an open question about naming: should the title highlight the underlying method (e.g., EM-enhanced LLM-based IR) or the emergent behavior (e.g., agentic characteristics)? I’d be interested to hear the authors’ perspective on this balance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}