Title: AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

retrieval augmented generation

self-play
Abstract: Search-augmented LLMs often struggle with complex reasoning tasks due to ineffective multi-hop retrieval and limited reasoning ability. We propose AceSearcher, a cooperative self-play framework that trains a single large language model (LLM) to alternate between two roles: a decomposer that breaks down complex queries and a solver that integrates retrieved contexts for answer generation. AceSearcher couples supervised fine-tuning on a diverse mixture of search, reasoning, and decomposition tasks with reinforcement fine-tuning optimized for final answer accuracy, eliminating the need for intermediate annotations. Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level finance reasoning tasks, AceSearcher-32B matches the performance of the giant DeepSeek-V3 model using less than 5% of iits parameters. Even at smaller scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented LLMs with up to 9× more parameters, highlighting its exceptional efficiency and effectiveness in tackling complex reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose AceRAG, a cooperative self-play framework. AceRAG trains a single LLM to assume two roles: a decomposer, which breaks down complex questions into simpler subquestions to guide information retrieval, and a solver, which integrates retrieved contexts and intermediate answers to produce final responses.\n\nThe training approach includes a two-stage fine-tuning method:\n1. Supervised Fine-Tuning (SFT) using an extended mix of retrieval, reasoning, and decomposition datasets to build general reasoning and retrieval capabilities.\n2. Reinforcement Fine-Tuning optimizing performance through iterative preference optimization, where the solver’s accuracy provides reinforcement signals, guiding the decomposer toward effective decompositions without explicit intermediate annotations. \n\nThe authors extensively evaluate AceRAG across three tasks and ten datasets and demonstrate significant performance improvements (average gain of 7.6%) compared to strong baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n- A key strength of this work lies in how it formulates reasoning-intensive RAG as a cooperative interaction between two roles: a decomposer and a solver. Also, the paper grounds this formulation in a principled way using the DPO framework, the mathematical proof is solid. \n- The authors follow a two-stage training strategy where RFT follows SFT. In the RL stage, they utilize a design of the preference-based reinforcement learning setup, particularly the way preference pairs are constructed separately for the decomposer and solver components. This formulation enables joint optimization of both roles using outcome-based feedback. \n-  The authors conduct extensive experiments using models of various sizes (1.5B to 32B), demonstrating that the proposed method scales well.  Moreover, they compare against a diverse set of baselines, including standard RAG systems, reasoning-enhanced QA models, and recent retrieval+RL-based approaches. This provides a well-rounded empirical picture and shows that the gains of AceRAG are not limited to a narrow benchmark. In addition, the authors show strong awareness of the importance of data diversity. Their SFT stage incorporates a rich mixture of datasets covering QA with context, question decomposition, and chain-of-thought reasoning, which strengthens the model’s robustness across a wide range of question types. \n\n## Weaknesses \n- The design of the reward function may be too rigid, which could limit its generalization or learning efficiency. The reward is defined as the product of two binary indicators: an exact match (EM) between the generated and ground-truth final answers, and a format-based check ensuring the presence of intermediate reasoning steps. This sparse signal could make policy optimization challenging. An ablation or justification of this reward choice would help strengthen the argument.\n\nQUESTIONS:\n- Given that both roles are trained jointly via preference optimization, did you observe any instability where improvements in one degrade the other? Did you try strategies other than randomly shuffle the preference pairs for different components, e.g. first train the solver given the correct context, and then the decomposer? \n\n- Have you tried other reward functions other than the proposed one?  \n\n- How diverse are the decompositions sampled by the model during reinforcement fine-tuning, and how sensitive is performance to decomposition redundancy? Do you apply any techniques to encourage diversity in z during sampling?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper introduces AceRAG, a cooperative self-play framework for reasoning-intensive retrieval-augmented generation (RAG). The approach trains a single large language model to perform both question decomposition and answer generation through supervised fine-tuning and reinforcement learning optimized on final answer correctness. The authors demonstrate performance gains across ten benchmarks and highlight improved parameter efficiency compared to recent baselines.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. This paper is well-organized and clearly written.
2. The idea of unifying decomposition and solving roles in one model is conceptually appealing.
3. The evaluation covers a diverse set of benchmarks, with thorough ablation studies and parameter analyses.
4. AceRAG achieves strong results while remaining lightweight, which is important for the open-source community and low-resource settings.



**Weaknesses**
1. The training pipeline (SFT + RFT) is somewhat intricate, potentially limiting reproducibility despite good documentation.
2. The assumption that reward signals from final answers sufficiently guide decomposition quality could be risky in more ambiguous domains.
3. The decomposition and multi-step retrieval introduce additional inference overhead compared to the standard RAG pipeline, which could limit applicability in real-time scenarios.

QUESTIONS:
Please see Strength and Weaknesses.

LIMITATIONS:
Please see Strength and Weaknesses.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAceRAG is a method for multi-step, reasoning-heavy retrieval augmented generation (RAG) using two roles: a decomposer and a solver.\nThe decomposer role breaks down the original question into subquestions and also takes into account answers to previous subquestions.\nThe solver generates intermediate answers, and the final answer based on previous intermediate answers and retrieved passages.\nThis methodology is not just prompting based but involves a two stage finetuning procedure. The first supervised fine tuning stage finetunes the LLM on a mix of retrieval-oriented QA, decomposition, and chain-of-thought tasks. The second stage uses reinforcement learning to optimize the multi-step retrieval and reasoning policy directly on downstream task success. Across ten multi-hop QA, fact-verification, and document-reasoning datasets, AceRAG-32B improves exact-match by 7.6% on average over strong open-source and RL-enhanced baselines, matching DeepSeek-V3 while using <5% of its parameters. The smaller 1.5B, 8B, 14B variants beat larger models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The RL formulation of directly optimizing the policy of two roles based on final task success is novel in the domain of multi-step RAG.\n2. Very strong experimental results, validating the methodology.\n3. Very thorough experiments: 10 datasets, cross-scale models, ablations on SFT data mix, RL variants, and hyper-parameters. Human studies back quantitative gains. Training/inference compute is reported.\n\nWeaknesses:\n1. Some unclear experimental choices (see Questions).\n\nQUESTIONS:\n1. What is the justification only exploring fairly outdated retriever models such as E5, Dragon and Contriever (Appendix H)?\n2. What is the justification for comparing to Qwen3-32B Reasoning when it does not have access to retrieved contexts.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes AceRAG, an LLM-driven RAG framework where a single LLM is trained to perform two roles: a decomposer that breaks down complex queries and a solver for answer generation. Following recent training receipt of thinking models, AceRAG learned from a two-stage training pipeline: Supervised Fine-Tuning (SFT) followed by Preference-based Reinforcement Fine-Tuning (RFT). The training process requires only final answer supervision, with no intermediate annotations needed. The technical design is well-justified in writing, and AceRAG demonstrates strong performance on standard multi-hop and complex reasoning RAG benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The overall design is reasonable. A unified framework with two roles fits well with the nature of multi-hop complex QA.\n2. The rationale behind the technical choices is clearly justified in Sections 3, 4, and the appendix.\n3. The experimental evaluation is comprehensive, and AceRAG demonstrates strong performance across benchmarks.\n\nWeakness:\n\nThe contribution is more on the engineering side than in theoretical novelty. Most of the key components in the proposed framework can be directly traced to existing literature, including self-play [1, 2], RL-based RAG optimization [3, 4], \"decomposer-solver\" strategy [4], and two-stage training pipelines [5, etc.]. While AceRAG introduces its own implementation refinements, it shares a similar underlying idea with prior work. This limits its novelty from a conceptual standpoint. That being said, I do not consider that this detracts from the fundamental contribution of the work, but view it as a missing citation ([1], [3], and [4])\n\n*[1] Self-playing Adversarial Language Game Enhances LLM Reasoning. NeurIPS 2024*\n\n*[2] Self-play fine-tuning converts weak language models to strong language models. ICML 2024*\n\n*[3] Adaptive Information Seeking for Open-Domain Question Answering. EMNLP 2021*\n\n*[4] IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues. SIGIR 2024*\n\n*[5] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*\n\nQUESTIONS:\n1. Since the \"decomposition-solve\" strategy is well calibrated for complex problems, I would like to confirm the scope of this work. Specifically, is AceRAG designed for multi-hop problems only, or is it a general RAG solution? If the latter, I would like to see the performance of AceRAG on a simple single-hop RAG benchmark to understand the impact of this training strategy on the model behavior.\n\n2. According to Line 142-144, different retriever is used for different tasks. Does this apply to inference time only or both training and inference time?\n\n3. In Figure 5b, the reranking baseline (blue line) is based on RankLLM. Given that this is an LLM-based reranking method, I am curious why it appears as a flat line, with no scaling in recall at all as model size increases. Could the authors provide more details about the experiment setup for Figure 5b, including model configurations?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a retrieval-augmented generation technique for solving complex multi-hop question-answering problems using LLMs. In particular, the authors propose a self-play framework that purposes the same LLM as both a decomposer (to break down the question into subqueries) and a generator (to generate the answers to each subqueries).\n\nThe high-level idea of the framework is quite intuitive, and in fact borrows from a long line of work in the RAG space (i.e., the idea of decomposition of subqueries followed by generation of answers to those). The true contributions and strengths of the work, to me, lie in the following: (1) the proposed DPO-based algorithm, the way they generate the contrastive triplets,  and the fact that they are able to neatly form a single dataset comprising of both the decomposed subqueries and the generated answers; (2) impressive results on standard multi-hop benchmarks, as well as in settings like document-level reasoning, fact verification, etc. that are generally not demonstrated in existing multi-hop RAG works to my knowledge.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. Quality: The formulation and the solution strategy is well-motivated, with theoretical backing.\n\n2. Clarity: There are some details missing, but in general the paper is well-organized, ideas are conveyed clearly.\n\n3. Significance: The paper addresses an important problem, the space is quite crowded, but the manuscript manages to make a meaningful contribution.\n\n4. Originality: As I called out in the summary, even though the high-level formulation of decomposition followed by generation is not original, the work has good and original insights in the way they generate the contrastive triplets and formulate the DPO problem.\n\nI've a few minor concerns listed next, but overall, I find the contributions and the results in the paper to be significant.\n\nWeaknesses: \n\n1. Related Work is sparse and perhaps undersell the work in the paper: For instance, many of existing work on fine-tuning LLMs for RAG focus primarily on decomposition, and use a standard/off-the-shelf generator, unlike AceRAG that solves them jointly by repurposing the same LLM. \n\n2. I didn't find LeReT [26] compared in the experiments. LeReT has a similar strategy for training the decomposer, but uses a standard SLM for generator, but I recall it did show strong results in some of the RAG benchmarks. It would be good to contrast the results of AceRAG with LeReT.\n\n3. I was somewhat puzzled by the need for the first SFT phase (Sec 4.1) when I read it. My thinking was that if you are starting with models like Qwen instruct versions, SFT on standard datasets with next-token prediction might not be as valuable as the RL-based training in the next phase. But the ablations in Table 3 shows that SFT is so crucial! Could you comment on the overlap of datasets between SFT and RFT phases?\n\nQUESTIONS:\nQ1. What's the exact configuration you use for the two CORAG baselines (Greedy & Inference Scaling).\n\nSee strengths and weaknesses for other questions/concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}