AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
Authors: Ran Xu
Yuchen Zhuang
Zihan Dong
Ruiyu Wang
Yue Yu
Joyce C. Ho
Linjun Zhang
Haoyu Wang
Wenqi Shi
Carl Yang
Venue: NeurIPS 2025 spotlight ()
Keywords: large language model
retrieval augmented generation
self-play
Primary area: applications

Abstract: Search-augmented LLMs often struggle with complex reasoning tasks due to ineffective multi-hop retrieval and limited reasoning ability. We propose AceSearcher, a cooperative self-play framework that trains a single large language model (LLM) to alternate between two roles: a decomposer that breaks down complex queries and a solver that integrates retrieved contexts for answer generation. AceSearcher couples supervised fine-tuning on a diverse mixture of search, reasoning, and decomposition tasks with reinforcement fine-tuning optimized for final answer accuracy, eliminating the need for intermediate annotations. Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outperforms state-of-the-art baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level finance reasoning tasks, AceSearcher-32B matches the performance of the giant DeepSeek-V3 model using less than 5% of iits parameters. Even at smaller scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented LLMs with up to 9× more parameters, highlighting its exceptional efficiency and effectiveness in tackling complex reasoning tasks.

AceSearcher:
Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
Ran Xu
Yuchen Zhuang?
Zihan Dong²
Jonathan Wang'
Yue Yu²
Joyce Ho
Linjun Zhang
Haoyu Wang
Wenqi Shi
Carl Yang!
Emory University Georgia Institute of Technology Rutgers University
SUNY Albany SUT Southwestern Medical Center
Dataset/Model: https: https://hagginngfcce.o/// http::/iaaaa......ceeeeeeeeeeeee che
Code: https: htpp:://githute hatp.///ciiaaaaa//eeeeeereeeee
Abstract
Search-augmented LLMs often struggle with complex reasoning tasks due to inef- fective multi-hop retrieval and limited reasoning ability. We propose AceSearcher, a cooperative self-play framework that trains a single large language model (LLM) alternate between two roles: a decomposer that breaks down complex queries and a solver that integrates retrieved contexts for answer generation. AceSearcher couples supervised fine-tuning on a diverse mixture of search, reasoning, and decomposition tasks with reinforcement fine-tuning optimized for final answer accuracy, eliminating the need for intermediate annotations. Extensive experiments on three reasoning-intensive tasks across 10 datasets show that AceSearcher outper- forms state-of-the- baselines, achieving an average exact match improvement of 7.6%. Remarkably, on document-level finance reasoning tasks, AceSearcher-322B matches the performance of the giant DeepSeek V3 model using less than 5% of its parameters. Even at smaller scales (1.5B and 8B), AceSearcher often surpasses existing search-augmented LLMs with up to 9x more parameters, highlighting its exceptional efficiency and effectiveness in tackling complex reasoning tasks.
Introduction
Large language models (LLMs) have demonstrated remarkable performance in areas such as natural language generation [65, 80, 17] and complex reasoning [26, 20]. However, they often fall short when handling long-tailed dynamically evolving knowledge [48]. To address these limitations, growing body of work has explored augmenting L.Mss with external search tools that retrieve relevant information at inference time. Search-augnen augheeted LLMMs not only improve factual accuracy [2, 58), but also facilitate efficient adaptation to new tasks and domains without costly parameter updates [78].
Despite notable advances in retrieval-augmentedd generation (RAG) [2, 39, 45, 86, 50], most existing approaches are restricted to relatively simple questions [33, 48] solvable through single-turn retrieval. However, real-world applications often demand more complex reasoning, requiring (i) multi-hop retrieval to gather relevant evidence from large corpora due to the low recall of direct single-step retrieval [83], and (ii) reasoning capability integrate multiple pieces of information beyond span extraction for response generation [9]. To address these challenges, prior works propose multi-step search via iterative prompting [68, 31, 36, 42, 88), often relying on powerful, closed-source L.LMs with strong reasoning abilities. Alternatively, tree-search algorithms have been explored to improve
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

retrieval and reasoning at inference time [27, 72, 77], but at the expense of increased latency. Recent efforts employing reinforcement learning (RL) frameworks allow LLMs to interact with search engines [30, 92, 62, 4, 63]. While promising. these methods are often memory-intensivy and thus less practical for deployment in resource-constrained environments. Additionally, their exclusive reliance on QA datasets for supervision limits the broader potential of LLMs to integrate search with complex, multi-step reasoning across a wider range of tasks.
Motivated by these challenges, we aim to develop an efficient, data-centric training recipe to enhance the capabilities of LLMs for reasoning- -intensiee search scenarios. Inspired by human problem- solving strategies where complex tasks are decomposed into simpler subproblems [93, 39J, we propose AceSearcher that trains LLMs to act as two roles: decomposer and solver. The decomposer breaks down the original question into subquestions to guide retrieval, while the solver generates intermediate and final answers by integrating subquestions, their answers, and context.
We then introduce two-stage fine-tuning framework to train both the decomposer and solver modules. In the first stage, we perform supervised fine-tuning (SFT) by extending existing open-domain QA datasets with open-source reasoning data. This covers task decomposition and problem-solving in both text and code. This simultaneously boosts the model's ability to extract relevant information from context as well as strengthens its general reasoning capabilities. In the second stage, we apply reinforcement fine-tume tinettunig on targeted reasoning and QA tasks, using rewards derived solely from final outputs. To overcome the lack of intermediate annotations, we hypothesize that better decompositions lead to more accurate answers. The solver is reinforced to produce correct answers based on decompositions and context, while the decomposer is optimized to maximize the solver's accuracy. This framework promotes joint structured reasoning across both roles with one unified model, while eliminating dependence on supervision from proprietary frontier models. Notably, AceSearcher achieves strong performance using iterative preference optimization, without relying on memory-intensive online RL training costly inference-time scaling.
Our contributions can be summarized as follows:
We introduce AceSearcher, a cooperative self-play framework designed to jointly enhance LLM's capabilities in both search and reasoning. By introducing two roles, namely the decomposer and solver, AceSearcher equips single LLM with joint skills of task decomposition and task solving, providing an efficient and flexible solution for complex reasoning in search-aue aarch-auggmeted settings.
We propose a two-stage fine-tuning framework that first applies SFT on a mixture of retrieval, reasoning, and decomposition datasets, followed by reinforcement fine-tuning using rewards solely from the final answer to train the decomposer and solver without intermediate supervision. This approach can be readily applied to LLMs with varying sizes (1.5B - 32B as shown in our study) to enhance the multi-step reasoning ability of augrenee LLMs.
We conduct extensive evaluations of AceSearcher covering three tasks across ten public datasets. Compared to strong baselines, including recent reasoning models and RL-enhanced search LLMs, AceSearcher demonstrates strong empirical performance with 7.6% gain on average. Moreover, AceSearcher demonstrates high parameter efficiency: the 1.5B variant matches the performance of models 10x larger QA tasks, highlighting its suitability for low-resource settings.
2 Related Works
Reasoning-intensiv Search/Retrieval. Standard RAG pipelines often consider single-step retrieval only and cannot handle complex questions well [34, 58, 18]. To incorporate reasoning into RAG pipelines, earlier research [68, 70, 71, 36, 88] design multi-turn prompting techniques for complex QA. Besides, several works [2. 75] leverage SFT on qugh-quuaity of-huuhf-thoughts to improve the reasoning skills of LLMs. but without explicit task decomposition. Additionally, [27, 77] design reward-guider search during inference time, [22, 79] trains the query refinement model based on the feedback of generator LLMs, and [6. 37] leverage multi-agent fine-tuning to further enhance reasoning performance, but at the cost of serving multiple LLMSs in deployment.
Self-play Finetuning for LLMs. Self-play [60] is an effective technique that enables LLMs to learn through self-interaction, promoting diverse experience trajectories and prompt coverage. Recent stud- ies have applied self-play to alignment [8, 76, 84], instruction following [13], theorem proving [15], and reasoning [10, 89]. Unlike these works, we consider collaborative self-play for complex problem solving, and tailor LLM self-play frameworks specifically for rentesie
intensive RAG applications.

---

Stage Supervised Finetuming
Stage
Finetuning
Pe (xlg))
- -
2
lol
log (g(nlgra))))))
Pa(alg)
Σ (o
Instruction
f:
AF
Final Answver
Question Decomposition
Sub--Answer
Context- Rich QA Data
Instruction
Fo
Pafelp x0
Question Decomposition Data
Instruction
Choin-of-Thoughts Data
Instruction
Ao
Z:
Figure I: Overview of AceSearcher. AceSearcher contains a twO-Stage training process to teach LLM for joint precise question decomposition and answer generation with cooperative self-play
RL for Search-augmente LLMs. Very recently (concurrent to us), multiple studies [4, 62, 30, 92, 29] attempted to leverage the RL pipeline for RAG by viewing the search as an external tool and using open-domain QA datasets (e.g. NQ [33]. HotpotQA [83]) to create verification rewards. In contrast to these approaches, we propose a data-centric pipeline that enhances LLM retrieval and reasoning capabilities through unified self-play fine-tuning framework. Our method demonstrates strong generalization across a broad range of reasoning-intente RAG tasks beyond multi-hop QA.
3
Overview of AceSearcher
In this section, we first define the problem setup and present an overview of AceSearcher. Then, we introduce the training and inference pipeline for AceSearcher.
3.1 Problem Formulation
In our setting, let Q denote the space of questions and A the space of all possible answers. Given question q E Q and corpus C (e.g. Wikipedia) that provides background knowledge, the retriever (often embedding models) first find small set of relevant passages D ...k),, then the LLM Sa generates the output a' E A conditioned on both q and as a' ~ po g, D). Note that a' can be short- long-form response, depending on the type of task. In reasoning-intensinn scenarios, the question q may require multi-step reasoning beyond simple retrieval to produce accurate answers.
3.2 AceSearcher: A Cooperative Self-Play Framework
Our AceSearcher model, shown in Figure 1, tightly couples reasoning and search by enabling single LLM to act as two roles (controlled by different input and prompt templates):
A decomposer that converts the original question q into sequence of subquestion templates'
z = (31, 22, ), where the number of subquestions is n and 2 may depend on answers to earlier subquestions. These templates sampled from pe g).
A solver that generates intermediate answers w [wi, w2, wn) and final answer a' in stepwise manner: For each subquestion E 2, the solver produces the intermediate answer as ~ po(-1 2,, Wess D.). where Wis denotes the answers to previous subquestions, D; {d,, Id........ d is the set of retrieved passages for zi. After solving subquestions, the solver predicts the final answer a' ~ (- q, z, w. D) based on the original question, intermediate answers and context passages.
Joint Learning Objective. Given the question g, we train 0 to maximize the probability of the LLM for generating the final answer a. In our framework, the learning objective can be written as
(a 9) = Σ pe(2 9) Σ po 9. w) (w 4,22
(3.1)
In practice marginalizing over all possible decompositions and intermediate answers w is in- tractable. To approximate it during training, we sample small set of candidate (z, w) paths and
Wee refer to them as "templates" since some subquestions are determined by answers to previous ones. An example template [19] "Q1: What items did Aristotle use?: Q2: Is laptop in #1?", In practice, the template is text snippet with tixed format and will be parsed to multiple subquestions, detailed in Appendix F.

---

identify the most promising ones to encouraging the decomposer to help the solver generate the correct answer. inference time, given question 9, the decomposer p generates a subquestion sequence and the solver π reasons over the intermediate answer to derive the final answer a'.
Two-Stage Finetuning for AceSearcher
To enable the LLM to perform both roles effectively, we first apply SFT on publicly available datasets to establish its foundational capabilities. Subsequently, perform reinforcement fine-tuning to further improve LLM's capabilities, using only final answers as supervision.
4.1 Stage I: Supervised Finetuning (SFT)
Although recent studies have introduced data mixing strategies for search- aarrchh-ugmentedd LLMs [39, 45, 86, 37], they focus enhancing the LLM's ability to extract answers from provided contexts. In contrast, our setting presents greater challenge - requiring the LLM to automatically decompose and solve complex questions across a diverse range of tasks that requires reasoning. Towards this goal, we extend the SFT data mixture Dan for the following tasks:
Context-rich QA Data. We follow [45, 39, 86, 37] leverage multiple QA datasets enhance the LLM's capability of using context for generation. Specifically, we consider the following datasets: NQ [33], SQuAD [56]. DROP [16), NarrativeQA [32], Quoref [12], ROPES [38], FEVER [66],
TAT-QA [94], which contains a question, context passages, and an answer.
Question Decomposition Data. To improve the LLM's ability to decompose complex questions into simpler subproblems, we incorporate GSM8K [11], ConvFinQA [9], and StrategyQA [19].
These datasets require generating sequence of subquestions for solving the original problem.
Chain-of-thought Data. To enhance multi-step reasoning, we leverage chain-of- of-thought datasets including GSM8K [11], TabMWP [46], and IfQA [85]. Inspired by studies showing that combining Chain-of-Thought (CoT)) [74] and Soggammoff-Thugghe (PoT) [S] rationales can boost reasoning capabilities, we incorporate MathInstruct [87], which contains CoT and PoT style prompts.
Detailed descriptions of datasets, prompt formats, and the number of training examples provided in Appendix F, F. In total, we curate 180K training examples in the SFT stage. The LLM is fine-tuned using the standard next-token prediction objective.
4.2 Stage II: Preference-based Reinforcement Finetuning (RFT)
While SFT equips the LLM with basic capabilities for question decomposition and answer generation, relies on richly annotated prompts with intermediate question decomposition and chain-of-thought annotations resources that limited in practice. To overcome this scarcity, further fine-tune the LLM on prompts D = {q, a}} covering RAG and context-reason reasoxn-re scenarios that contain only the final answer a* given the question q. We frame this setting the LLM learn to actively decompose the question and generate intermediate reasoning steps with external context. This motivates the use of reinforcement learning to optimize the reasoning trajectory in the absence of explicit intermediate supervision.
interactive environment, where
o Environment for RAG. We collect labeled pairs from multi-hop QA and fact verification datasets, including HotpoiQA [83], 2WikiMHQA [21] and HOVER [28], which require the usage of retrieval to generate accurate answers. To formulate the RAG framework decomposer P first generates a sequence of candidate sub-qucstions z = (Z1...., 2n) E Q". For each sub-question gi, & relevant documents intermediate solutions by generating W/ ~ po (- 24, Wei, Dz), conditioned on the current sub- question, previously generated answers, and retrieved context. Finally, the solver predicts the final answer as a' ~ p- U,_1 Zi. UP Wi, U2_ D,).
environment, the query
retrieved, denoted by D, The solver pe then produces
Environment for Context-Rich Reasoning. Beyond RAG-specific tasks, we also focus on improving the LLM's reasoning abilities. To this end, we incorporate three datasets from the SFT stage, including GSM8K [11], TabMWP [46], and ConvFinQA [9], which involve reasoning over contexts C such as tables, passages, or problem conditions. Under this setting, / used to generate subquestions (21, (2 2n) E Q" and the solver pe produces intermediate solutions by generating

---

Wi po (- 2g, wess C), conditioned on the current subquestion, previous answers, and contexts. Finally, the solver predicts the final answer as a' pol- 1 2, Ui=1 w. C.,CC.
o Reward Design. For both scenarios, the complete trajectory (g, 21, wh,..., 2n, Wn a) is evaluated using a reward signal derived from the final answer. Specifically, the reward function is defined as:
r(g, a', a) = EM(d', a) x (('
(4.1)
where EM denotes the exact match between the model-ggeeertedd a' and ground-truth answer a. The function fqq a') represents a format-base based binary reward, verifying whether the model generates sub-questions, intermediate answers, and reasoning steps in the correct structure.
Optimization Te and pe. During the RL phase, we use the reward function defined above as the feedback to update both тh and pg. Denote ua 2, wlg) pe (=(21)) (=)g))e((, alq, z). Following existing works [52], the overall optimization objective is formulated as
max By E (w=pp. [r(g. a', a)] - 9DKL [uo (a', 2, w ( M)l w [
(4.2)
where 3 is the parameter for controlling deviation from the reference policy. We further decompose the KI, divergence between ug and ther as
DRI(ue M = Σ uola,, 2, w/1) po(s) + po d'12, log prr(()) - log Pier [=. a/,,,0
Poo(zlo) log (2 (g)
Σ (=(1)) Σ (w, a' (2, log po (w, a' 9)
pear 214)
Peot (w; a² 2,
Datteelleed
Then, the optimization objective can be rewritten as
max To = Egg [Be-pe (",a")>> [r(g, a', a]] 3 Dxx ( (pollppe)) BEzmpo DKL (Too [To]| (4.3)
The above optimization problem have the closed-form solution (details in Appendix B) [55] as
( a',
p" (z (z) x pret(= (q) [(upe)))ppt((ll)))
exp
p" (w. a' 2) x (rer(ue, d' q, 2))2)exp exp (Frle a',
What does the form of nt and p² imply? The closed-form policies p² (i.e. p" 4)) and n² (i.e. p" (w,, a' q, =)) align with intuitions: an effective decomposition policy P promotes higher overall expected reward by enabling better intermediate reasoning steps, while an improved solver π directly enhances the reward, regardless of the quality of the decomposition.
Practical Implementation for Optimization. In practice, direct optimization under sparse reward signals from trajeetrry rollouts is often ineffective due to high variance and limited feedback. We employ a rollout strategy to address this challenge and enrich the learning signal. For each question q. first generate m candidate decompositions by sampling from the decomposer policy, ie., 2(1) p-(- q) for sample m' candidate solutions by drawing from the solver policy a, Te(- g, (0) for j = -I...,,m! To construct preference datasets for RFT, we first identify the best and worst decompositions for each question g based on the expected reward over their corresponding solutions as F(g, 2(0) = (= ((g, d', a). This results in the following preference pair dataset:
I,..., m. Then, for each decomposition 2(1), we subsequently
[2(++) = =(1), j = arg max F(g, (0)), Daccompose (lg ((1))) E D), where
(4.4)
= j arg mjn F(q, =(1,
Constructing preference pairs to optimize the answer generation policy æ (with fixed subquestions z) is more challenging due to the presence of multiple intermediate answers along the reasoning
trajectory. Denote the trajectory T() (q. 21, w (i) ..... Zn, wh a ())) with a'O being the final
prediction, we create preference pairs for intermediate Dauby and final question answering Dinat as
Dsuby = U {(z;, w w.) wt * w, (g. a) E DD (2,, E TT,, (2,, w,, ) E }
Dinal [[g, 21, w w ...., En, w 1,at'' "aaat,a'' dd') (g.a) ED.
5

---

where the best and worst trajectories are selected as:
T (q, 21, wt ................................ Zns wt a a²+),
where a' arg max ((g, a ((()),,))))))) a),
T = (q, 21; Wi Bny Wn ))
where a' = arg min r(g, a (6) .).
To jointly optimize both the decomposer and the solver πT, we construct unified preference dataset by combining three sources of pairs: Preef = Deccompoce Daahg U Pinaa. For notational consistency, represent each example as (z, 9 9 ). where is the input, and g" are the chosen and rejected responses. Following [55], we optimize the policy with the following preference loss:
(8(108 (gt x)) Pref (g+ xx)
Pe (g x)) pher
LDoO (B .7 )~DDop log
log
(x)
Multi-turn DPO for Online Optimization. Motivated by the benefits of on-policy data sampling in RI., we adopt an iterative DPO framework for improved optimization. Specifically, in the I-th
iteration, use the LLM policy model? p(t) to act as To and pa to sample preference pairs to create the dataset ((1) pref. Then, we use Dpoer to update the policy model for the next iteration f/ ((++1) as
(1 log Po (g+
(x)
-log Po
((+1)
(g x)) ) (4.5)
(g 1 x:)
LDDOO
-F (x o- YuD to log
Po 12) (g+ x))
10 Pb
Remark. To balance effectiveness and efficiency in practice, we adopt the following strategy: the model me directly generates answers for intermediate questions, while producing a full rationale only for the final answer. To prevent overly long input contexts during final answer generation, we set the total number of documents to N (N = 15 in this study), and allocate up to [N/n] top-ranked documents for each of subquestions produced by TT. We discard preference pairs if the reward for the best and the worst response is the same.
Theorem 4.1 (Informal). Under regularity conditions, with high probability, the minimizer of the loss (Eq. (4.5)) at step / is close to the minimizer of the loss (Eq. (4.2)). Furthermore, as increases, the minmmizer converges to the true parameter 0*
The proof for the theorem is deferred to Appendix C due to the space limit. This theorem implies that our optimization algorithm is equivalent to maximizing the reward in Eq. (4.2). Furthermore, guarantees convergence of our algorithm, which we also empirically validate in Section 5.4.
5
Experiments
In this section, we conduct experiments on various tasks to verify the effectiveness of AceSearcher.
5.1
Experiment Setups
Tasks and Dataset Information. We consider the following 3 types of tasks: (i) Multi-hop hop QA, which includes 2WikiMHQA [21], HotpotQA [83], Bamboogle [53] and MusiQue [67]. (ii) Multi-hop Fact Verification, namely HOVER [28] and Exfever [47]. (iii) Document-level Reasoning, where we use the Doc benchmark [91] with several financial reasoning datasets such as TAT-QA [94], FinQA [7], MultiHiert [90], and TAT-HQA [35]. Note that some of datasets have very long contexts that make retrieval necessary. The detailed information for these datasets is in Appendix D.
Baselines. For Multihop QA and Fact Verification tasks, we compare against the following categories LLMS with Single-turn RAG: we consider Llama-3.1 1-i [17].
of baselines: (i) Instruction-
-1-pSeek-R1-Disseee [20], Qwen-3 3 [80]', Slama-4-Maverice [1], GPT-4o [25], and GPT-4.11 [51].
(ii) Prompt- based Multi-step Retrieval: we include IRCOT [68], Plan RAG [70], Search-ol [36].
IterDRAG [88]. (iil) Finetuned LLMs with Search: we compare with InstructRAG [75]. RAG-
Star [27], ReARTeR [64], CORAG [72] and Iter-RetGen [57]. (iv) LLMs with Search Truined via Reinforcement Learning: Recent agentic search-augmenter works such as Search-R1 [30]. R1-
Searcher [62], DeepResearcher [92], MMOA-RAG [6), and ReSearch [4] are also included for comprehensive evaluation. For document-level reasoning, we follow DocMath-Eai -val [91] to compare
We denote the model after the SFT stage described Section 4.1 as (1)
For Qwen-3, we evaluate both thinking and non- tot-hiiniing prompting modes and report the better result.

---

Table 1: Comparison of AceSearcher and baselines on Multi-hop QA and Fact Verification datasets. stands for results that are not publicly available. : This model often does not follow instructions and generates long answers. :: Concurrent works (preprint appears online after 2025/03/01).
Busclines
ZWIIMHHOA
HolpotQA
Bamboogle
MusiQue
Hover
ExFever
Arg. QA
Avg. All
Ace
EM
F1
Acc
EM
Fl
Ase
EM
Fl
Ase
EM
FI
EM
EM
Auc EM
EM
Rose Size: <. purrsmnier
1.mm-- 1-i
(17)
16.0
26.5
24.4
14.8
12.5
66.1
45.0
53.5/133
27.4
R1 Dusill RAG 8B 120)
30.0
42.8
45.2
25.2
39.2
25.6
37.3
21.6
21.4
65.0
48.2
39.2/25.3
34.1
RAG
35.4
42.0
50.4
65.7
68.8
46.7/316
43.5
Plan RAG 5D (20)
36.6
06.0
47.6
35.2
31.0
23.4
32.2
20.4
12.2
21.3
57.9
625
36.77269
38,0
Scurch-R1 [30]
38.2
43.3
R1-Seandfrr TBF
$2.6
52.57
DespResearcher (92)
72.8
71.0
27.1
[25]
43.2
46.0
24.8
65.3
580
42.4/3177
41.7
MMOA RAG 5B
42.8
41.4
05.6
36.2
CORAG HB (Giundyy) [72]
56.5
62.3
50.1
63.2
37.6
51.4
15.6
29.3
7-0.7
CORAG SB (Inference Scaling) [72]
72.5
77.3
56.3
09.8
54.4
68.3
30.9
42.4
53.5
AceSeander 1.5B
69.8
68.5
00.6
50.4
59.8
34.4
33.6
41.7
37.2
26.8
37.0
60.7
64.2
51,57429
41.4
AccSemmhhr 3B
66.0
76.7
68.2
58.8
50.3
60.8
55.2
63.5
3638
47.7
68.3
73.8
B4.T /3.9
39.8
Lorge Nere: 300 porameters
Owen 255 RAG 14B
64.4
17.8
29.1
50.4
35.6
40.8
22.4
35.4
21.8
18.2
65.7
42.9
39,47214
32.4
R1-Disiil 14b (20)
34.8
20.8
5.2
68.1
62.5
33.3/74
26.7
Qwen-3 14R (X0]
[80]
59.2
42.0
09.1
50.4
36.8
46.7
32.4
25.6
67.5
70.5
51-51346
46.1
51.2
53.2
[30]
52.8
/423
InstrccRAA (75)
50.4
55.1
57.6
41.4
25.5
67.5
65.3
45.9/36.1
46.3
AceSearcher
81.2
65.6
76.6
70.8
61.3
60.0
53.6
65.6
18.6
47.4
69.3
75.0
68,2754.2
60.1
Stze > 300 parurecters
Qwen 2-- R.MG 32B
51.4
31.6
40.6
56.11
38.5
50.4
59.2
51.2
65:2
22.2
10.4
19.5
71.3
69.6
43:2/3229
45.2
R1-Dieill AG
57.2
19.4
51.2
63.2
49.0
62:7
56.4
45.4
10.4
30.7
22.1
67.0
51.61384 16.4
48.8
Owen R.G 128F [80]
61.0
39.8
51.5
63.4
49.0
02.4
56.5
40.8
53.66
32.6
19:0
30.7
70.5
65.3
54.0/37.2
47.4
Scath-ol [36]
58.0
71.4
16.6
18:2 2
74.4
53.1
RAG [70]
32.4
49.
50.0
37.2
66.7
66.4
ReSearch 228"
ActSeamer RAG 738
34.6
41.6
56.5
20.9
6013
57.1
47.2733.5
43.4
R1-Dieill RAG 70R
61.0
50.4
90.8
67.6
53.0
67.3
60.8
48.8
61.1
16.6
23.8
67.7
65.3
56.57438
51.3
Liuma Marerick RAG 128
30.6
49.4
64.8
25.8
26.2
74,0
13.9
53.8/412
52.3
Progriesury Sarierut-Aggeosied L.Ms (For reforence)
GPT-4k RAG [25]
57.8
45.8
57.2
64.01
47.2
63.6
35.2
27.2
37.2
29.5
17.4
31..0
61.7
64.8
46.7/34A
44.0
GPT-4:1 RAG [51]
$1.0
42.4
49.5
60.8
46.0
59.3
40.8
35.2
44.3
30.0
15.4
29.5
67.5
66-4
45.7:35.0
45.7
IRCOT tecro shot. w GPT-4o) [65]
61.4
51.4
61.0
64.2
48.0
63.7
60.
45.4
56.9
33.8
22.4
33.5
63.7
64.8
55.1/42.1
49.5
shot, (IPT-4) [68]
73.9
$2.8
66.4
46.3
44.0
70.3
20.5
64.15508
57.3
RetGen GPT
.2.8
62.6
62.4
40.5
42.6
68.3
69.6
59.3746.2
52.4
RAG-Strr
(27)
ReAKTeR (wt GPT 4o [64]
54.4
30.2
BeDDRA (Geemni--) 1.5, 5M stx) (88)
76.9
67.0
75.2
56.4
51.7
64.4
68.6
65.6
75.6
30.5
22.5
35.0
58.2/117
against general instruction-tumed LLMs [51, 25, 65, 40, 41, 171, reasoning LLMs [26, 20, 80], Code L.MMs [96, 24], Math L.MSs [43, 82] and specialized finance reasoning L. [44, 95].
Implementation Details. We consider four different backbones for AceSearcher with varying sizes including Qwen-2.5 -- Stttrruuct11.B/141B.
[81] and Llama- 3. 1..1--BBTBa -Istructt [17]. For AceSearcher-32B. we apply LoRA fine-tuning [23] with r = 8, a = 16, while other models use full fine-tuning, All models are trained with a batch size of 64 and maximum token of 2048 for epoch on both SFT and RFT stages, with RFT run for 2 total iterations. For HotpoiQA, 2WikiMHQA, MusiQue, we the corpora provided by their respective sources. For Bamboogle, Hover, ExFever, we use the Wikipedia from Dec. 2018 as the corpus. During inference, we set the temperature
0.0, the number of retrieved passages to k = 10. For QA and fact verification tasks, we adopt E5 [73] as the retriever, while for document-level reasoning, we follow [91] and use the retriever. Detailed implementation settings for AceSearcher
OpenAl's Embedd i ng-3-Largge and baselines are in Appendix G.
Evaluation. For QA. report Exact Match (EM), Accuracy, and Fl score. For fact verification, we use EM as the metric. For document-level reasoning, we use Accuracy computed via the official evaluation script, and report the better performance between CoT and PoT prompting [54].
5.2 Evaluation on QA and Fact Verification
The main results comparing AceSearcher with baseline methods are presented in Table 1. From the results, we have the following key observations: (i) AceSearcher achieves strong performance over baselines. Notably, AceSearcher-325 achieves the highest overall score (60.7), outperforming both proprietary and sppncsouuce baselines by up to 7.6%. (ii) Compared to reasoning models, AceSearcher better adapt to RAG tasks. Qwen-3 and Deepseek-R R -distill are trained with extensive knowledge distillation, we observe that their gains are limited. This suggests that long thinking does not fully address the inherent challenge of multi-hop retrieval, while AceSearcher tackles this more effectively. (iii) AceSearcher has strong parameter efficiency. AceSearcher- 1.5B matches or exceeds 8B basclines, while AceSearcher-8D outperforms bascline models with 70B parameters.
7

---

Table 3: Ablation results QA and Dathath-Eval using -lama-3.1-88 We report EM for QA and fact verification due to space constraints. For w/o and w/o オ, we replace the respective components with L1ama-3. BB-Instruct. w/o Search excludes CQA, StrategyQA, and IfQA from SFT; w/o Reasoning removes GSM8K, TabMWP, ConvFinQA, and MathInstruct. w/ CQA follows [45] and finetune solely on context-aware QA tasks.
Model Name
aWIJMHHQA
HotpoRQA
Bamboogle
MusiQue
Hover
ExFever
Avg.
DMcs
DMML
DMcL
Avg.
Ablution Study for Different Componentr of AceSeorh
AncSearcher
66.0)
58.8
55.2
35.4
68.3
73.8
99.6
83.0
50.5
48.0
52.3
59.0
AecSearcher wio RFT
6188
53.8
52.0
34.8
64.1
71.4
56.2
71.5
73.0
49.0
26.3
52.3
AucSeuncher wib SET
40.0
37.0
38.4
18.2
74.7
76.8
47.6
71.0
51.5
46.0
31.0
48.0
AudScuncher wil
$7.8
52.4
$3.6
12.4
65.0
70.5
55.3
81.5
78.0
45.0
29
56.6
AccSeurcher wio
41.4
32.0
20.8
12:2
03.7
73.0
40.9
73.3
72.0
45.0
27.3
52.4
Ablution Study for SFT Daso Mizure
AueSeurcher wio Search
52.6
53.0
51.3
56.5
49.3
79.5
42.0
28.7
56.6
AccSearcer Reasoning
55.8
57.7
74.0
38.0
20.7
53.5
ArcSearcher [451
40.0
22.4
36.3
38.0
20.3
38.6
Ablation Study for RL. Algoritioe
RAFT [14]
63.6
55.6
50.4
32.8
66.7
69.8
56.5
73.0
69.5
43.0
27.3
51.2
RES1 1611
05.4
37.6
53.2
32.8
60.5
08:7
57.2
77.3
81.0
48.0
25.0
56.1
Offinne [S3]
64.6
57.8
53.6
35.3
64.6
73.2
73.8
83.5
49.0
50.0
56.6
(Nerstive) SimPO [49]
67.2
57.3
4688
34.6
60.3
70.5
57.6
78.0
44.0
32.0
55.9
5.3 Evaluation on Document-level Reasoning
Table 2: Results DocMath-Eval [91], sorted by average performance. SS, CS, SL. and CL stands for SimpShort, CompShort, SimpLong and CompLong, respectively.
We evaluate AceSearcher on DocMath-Eval (Ta- ble 2) against large-scale LLMs, demonstrating notable improvements over similarly-sizer base- lines, including both reasoning and Nomain--peciife models. For instance, AceSearcher- 32B and AceSearcher-SB outperform size-comparable base- lines by 6.2% and 9.0%, respectively. Further- more, AceSearcher achieves performance compa- rable to significantly larger models: AceSearcher- 32B matches the accuracy of DeepSeek- V using less than 5% of its parameters, while AceSearcher- 14B exceeds baselines up to 72B (5x) in size. These results highlight AceSearcher's 's strong gen- cralization capabilities beyond factual QA, particu- larly in complex reasoning scenarios involving long documents and tables.
Datasets
DMs
DMs
DMat
DMCt.
Avg.
Proprritaory Models
GPT-o3-mini Gemini-1
86.0
87.5
59.0
35.00
63.9
85.5
80.0
58.0
40.3
63.7
GPT-4,1
85.5
62.6
GPT-4a
86.0
Claude-3.5-Somee
78.0
54.0
Open-Suwreed Models
DeepSeck V3 685B
89.5
86.0
53.0
42.3
66.4
Azcheureher 32B
89.5
84.0
53.0
43.0
66.
DeepSeck-VV 2368
87.0
75.5
61.00
43.0
64.4
DeepSeck- R1 685B
89.0
K3.5
53.0
38.7
64.3
DeepSeck-Code Co 2368
85.0
78.0
$6.0
41.0
63.1
Mistral Large 122B
85.0
76.5
$6.0
41.0
62.5
AceSeircher 148
84.0
82.0
49.0
39.3
62.4
AceMath 72B
77.5
77.0
59.0
39,7
60.9
Qwen-2 Madh 72B
78.0
73.0
5K.0
41.0
60.4
Dianlin
59.9
AzeSeurcher
83.0
48.0
Owen-2
81.0
57.0
DeepSeck-R -iiisill 70B
77.5
76.0
$3.0
34.7
58.0
Qwen:2.5 72B
81.5
$1.0
64.0
24.7
57.9
DeepSeck-R2 RisDiisill 32B
74.0
71.0
50.0
10.3
57.6
5.4 Additional Studies
Llama-3.3 708
79.5
74.5
$4.0
31.7
57.1
Qwen3 32B
80.0
76.0
44.01
25.3
54.5
Qwen3 14B
75.0
Ablation Study. Table 3 reports the results of Ace- Searcher. The top rows show that both SFT and RFT contribute to overall performance gains. Be- sides, AceSearcher improves both question decom- position (p) and answer generation (π), replacing each component with the frozen Llama- -ann--8b-it hurts
Dimnlin-R 7B
67.0
68.5
41.0
29.3
50.0
AceMath 7B
63.5
62.0
47.0
26.7
47.8
AceSearcher 1.5B
66.5
77.5
39.0
18.0
47.6
Qwená 88
76.0
76.5
32.0
11.7
46.5
7B/
DeepSeok Coder
67.5
62.0
32,0
37.6
Qwen-2.5. Math
$2.0
36.0
36.0
the performance. This verifies the complementary roles of these two components.
Ablation Study For SFT Data. The middle rows Table show SFT performance under different data compositions. Removing either the Reasoning or Search data leads to performance drops 58-
60
Avg. Performunce
across both knowledge-intensiv tasks (QA and Fact Verification) and document-level reasoning, indicating that both components are jointly beneficial for building capable LLM with search.
86-
54
QA & Fact Verif
Ablation Study For RFT. In the bottom rows of Table 3, we compare our reinforcement finetuning algorithm with other al- ternatives and find AceSearcher achieves the best performance. This highlights the importance of using both positive and negative trajectories, and shows that online methods outperform their of- fline counterparts. Figure 2 shows results across RFT iterations of
Reasoning
52
SFT
RFT-1
RFT-2
RFT-3
Figure 2: Performance of Ace- Searcher over different stages.

---

SPT
RPT
QA Fast
Reasetting
Ressoning
Avg, Performannss
50
60-
Avp. Performanee
55-
Avg. Perlorannn
60
40-
65-
20
JAC320
Qead Remenap
30-
50-
Ouee RAG
2K
10K
50K
2K
10K
50K
5K
10K
20K
Avg.
per Question
(a) SET Data Efficiency
RFT Data Efficiency
(c) Inference Efficiency
Figure 3: Efficiency Studies of AceSearcher with Llama-2 3. 1-88-- Instrnut as the backbone.
Accioardd
Arg. Periemumes
Resaligt
60
Scone
60
Frees
SFT
13
Pact
Rassning
3.5
1.58
3ig
Model Size
SiB
(a) l
(b) m. m
(a) Human Study
(b) Passage Recall @ 10
Figure Parameter Study
Figure 5: Quality Analysis for AceSearcher
AceSearcher-8B. We observe significant gains in the first two iterations, with diminishing returns in the third. We set the number of iterations to 2 to balance between performance and efficiency.
5.5 Efficiency Studies
Data Efficiency. Figure 3(a) and 3(b) show the accuracy of AceSearcher under varying amounts of data. For SFT, we evaluate the performance with varying SFT subset sizes and its improvement after fix the full SFT dataset isolate the effect of RFT. With
subsequent RFT. For RFT experiments, just 2K SFT examples (~ 1)), AceSearcher matches strong baselines like Search-RI I and Search-O1 (with up to 4x more parameters). and surpasses them after RFT. In the RFT stage, the use of only 5K prompts leads to 1% gain in QA and fact verification and 2% gain on document-level reasoning, justifying the data efficiency of AceSearcher with a diverse set of prompts.
Inference Efficiencyy Figure 3(C) shows the inference time of AceSearcher and baseline models on QA and fact verification tasks. Unless noted, all models 8B similar in size. While AceSearcher incurs higher latency than standard RAG due to question decomposition and multi-step reasoning, achieves substantial performance gains even outperforming 32B models with comparable inference time. Besides, AceSearcher outperforms reasoning models and inference-time scaling methods costs 1.5x to 2.8x more time. These results justify AceSearcher balances between efficiency and efficacy.
5.6 Parameter Studies
We study the effect of varying k. m. and m' on AceSearcher. shown in Figure 4(a), performance improves with more retrieved contexts, with gains plateauing at k: 10, which we adopt in our experiments. Figure 4(b) shows that increasing the number of sampled decompositions (m) and final answers (m') generally improves performance it will generate more valid preference pairs, but increases trajectory collection time. The study on the effect of and retrievers is in Appendix H.
5.7 Quality Analysis
of Question Decomposition Module
As question decomposition is a key component of AceSearcher, we analyze the quality of the generated subquestions. Figure 5(a) shows the average human evaluation scores (on 1-5 scale) for 40 randomly sampled subquestions per task. We observe that both SFT and RFT significantly enhance subquestion quality across different model sizes. To quantify the impact of decomposition on end-task performance, we evaluate passage-level answer recall on HotpotQA after applying question decomposition. As shown in Figure 5(b), AceScarcher achieves up a 25% improvement

---

in recall @ 10 over standard retrieval and surpasses strong passage reranking model". The details for human studies as well as more cases studies are given the Appendix I.
Conclusion
We present AceSearcher, a cooperative self-play framework specifically designed for RAG and document-level reasoning tasks. By training single LLM to act both decomposer and solver, AceSearcher addresses complex multi-hop retrieval and reasoning effectively. Our two-stage fine- tuning framework combines SFT on diverse reasoning tasks with preference- -base RFT guided by final answer accuracy, achieving strong performance without relying on expensive intermediate supervision. Evaluated on ten benchmarks, AceSearcher outperforms state-of-the-art models by 7.6% on multi-hop QA and fact verification, and matches Deepseek- V3 on document reasoning with under 5% of its parameters. Even with smaller models (1.5B, 8B). AceSearcher delivers competitive superior performance, offering an efficient and generalizable solution for advanced reasoning under resource constraints.
Acknowledgment
RX and CY were partially supported by the US National Science Foundation under Award Numbers 2314449, 2312502, and 2442172, as well as the US National Institute of Diabetes and Digestive and Kidney Diseases of the US National Institutes of Health under Award Number K25DK135913. JH was partially supported by the US National Science Foundation (NSF) grant partially supported by the Texas Advanced Computing Center (TACC) and the NVIDIA Academic Grant Program. LZ was partially supported by the NSF CAREER DMS-234022 and AI for Math Fund from Renaissance Philanthropy.
WS was
References
[1] AI AltMeta. The llama herd: The beginning new era of natively multimodal innovation, 2025.
[2] A. Asai, Z. Wu, Y. Wang, A. Sil, and Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In ICLR, 2024.
[3] Balakrishnan, M. J. Wainwright, and B. Yu. Statistical guarantees for the em algorithm: From population to sample-based analysis. The Annals of Statistics, 45(1):77, 2017.
[4] M. Chen, T. Li, H. Sun, Y. Zhou, C. Zhu, Yang. Z. Zhou, W. Chen, H. Wang, J. Pan, et al. Learning to reason with search for llms via reinforcement learning. arXiv preprint arXiv:2503. 19470, 2025.
[5] W. Chen, X. Ma, X. Wang, and W. W. Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. TMLRR 2023.
[6] Y. Chen, I.. Yan, W. Sun, X. Ma, Y. Zhang, S. Wang, D. Yin, Y. Yang, and J. Mao. Improving retreeeal-augmeened generation through multi-agent reinforcement learning. arXiv preprint urXiv:2501.15228 2025.
[7] Z. Chen, W. Chen, C. Smiley, S. Shah, Borova, Langdon, R. Moussa, M. Beane, T.-H. Huang, B. R. Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. In EMNLP, pages 3697-3711, 2021.
[8] 7. Chen, Y. Deng, H. Yuan, K. Ji, and Q. Gu. Self-play fine-tuning converts weak language models to strong language models. In ICML, pages 6621-6642, 2024.
[9] 7. Chen, S. Li, C. Smiley, Z. Ma, S. Shah, and W. Y. Wang. Convfinqa: Exploring the chain of numerical reasoning conversational finance question answering. EMNLP, 2022.
Using attps::/huggiinn co/ http:///wwwa..aatttto/////am/aaa passaa---- for reranking.
10

---

[10] P. Cheng, T. Hu, H. Xu, Z. Zhang, Y. Dai, L. Han, N. Du, and X. Li. Self-playing adversarial language game enhances LLM reasoning. NeurIPS, 2024.
[11] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek,
J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1400 14166, 2021.
[12] P. Dasigi, N. F. Liu, A. Marasović, N. A. Smith, and M. Gardner. Quoref: A reading compre- hension dataset with questions requiring coreferential reasoning. In EMNLP, 2019.
[13] G. Dong, K. Lu, C. Li, T. Xia, B. Yu, C. Zhou, and J. Zhou. Self-play with execution feedback: Improving instructionffollowin capabilities of large language models. In ICLR, 2025.
[14] H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, Pan, S. Diao, J. Zhang, K. Shum, and T. Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
[15] K. Dong and T. Ma. Beyond limited data: Self-play IIm theorem provers with iterative conjecturing and proving. arXiv preprint ar xxxv:2502.002/12, 2025.
[16] D. Dua, Y. Wang. P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL, 2019.
[17] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[18] Y. Gao, Y. Xiong, Y. Zhong, Y. Bi, M. Xue, and H. Wang. Synergizing rag and reasoning: A systematic review. arXiv preprint arXiv:2504.1555 15909, 2025.
[19] M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. TACL, 9:346-361, 2021.
[20] D. Guo, D. Yang, H. Zhang, Song, Zhang, Xu, Q. Zhu, S. Ma. Wang, X. Bi, al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[21] X. Ho, A.-K. D. Nguyen, Sugawara, and A. Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. In COLING, 2020.
[22] Hsu, O. Khattab, C. Finn, and A. Sharma. Grounding by trying: LLMs with reinforcement retrieval. In ICLR, 2025.
[23] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[24] B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. 12186, 2024.
[25] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda,
A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[26] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai ol system card, arXiv preprint arXiv:2412.16720, 2024."
[27] J. Jiang, J. Chen, J. Li, R. Ren, S. Wang, W. X. Zhao, Y. Song, and T. Zhang. Rag-star: Enhaneing deliherative reasoning with retrieval augmented verification and refinement. arXiv preprint 2rXiv:2412.1288) 2024.
(28) Y. Jiang, S. Bordia, 7. Zhong, C. Dognin, M. Singh, and M. Bansal. Hover: A dataset for many-hop fact extraction and claim verification. In Findings of EMNLP, pages 3441-3460, 2020.
[29] B. Jin, J. Yoon, P. Kargupta, O. Arik, and J. Han. An empirical study on reinforcement learning for reasching-search interleaved Ilm agents. arXiv preprint 1711v:2505.15117, 2025.
11

---

[30] B. Jin. H. Zeng, Z. Yue. J. Yoon, S. Arik, D. Wang. H. Zamani, and J. Han. Search-rl: Training 11ms to reason and leverage search engines with reinforcement learning. arXiv preprint urXiv:2503.05516 2025.
[31] O. Khattab, K. Santhanam, X. L. Li, D. Hall, P. Liang, C. Potts, and M. Zaharia. Demonstrate- search-predict: Composing retrieval and language models for knowledge-intensive nip. arXiv preprint arXiv:2212.14024. 2022.
[32] T. Kočiský, J. Schwarz, P. Blunsom, C. Dyer, K. M. Hermann, G. Melis, and E. Grefenstette. The narrativeqa reading comprehension challenge. TACL, 2018.
[33] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. Natural questions: benchmark for question answering research. TACL, 2019.
[34] P. Lewis. E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih. T. Rocktäschel, al. -ugmeette generation for knowledge-intensitn nlp tasks. NeurIPS, 33, 2020.
[35] M. Li, F. Feng, H. Zhang, X. He, Zhu, and T.-S. Chua. Learning to imagine: Integrating counterfactual thinking in neural discrete reasoning. In ACL, pages 57-69, 2022.
[36] X. Li, G. Dong, J. Jin, Y. Zhang, Y. Zhou, Y. Zhu, P. Zhang, and Z. Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05336 2025.
[37] X. Li, S. Mei, Z. Liu, Y. Yan, S. Wang, S. Yu, Z. Zeng, H. Chen, G. Yu, Z. Liu, M. Sun, and C. Xiong. RAG-DDR: Optimizing retrieval -ugmenttd generation using differentiable data rewards. In ICLR, 2025.
[38] K. Lin, O. Tafjord, P. Clark, and M. Gardner. Reasoning over paragraph effects in situations. In Workshop on Machine Reading for Question Answering, 2019.
[39] X. Lin, X. Chen, M. Chen, W. Shi, M. Lomeli, R. James, P. Rodriguez, J. Kahn, G. Szilvasy, M. Lewis, L. Zettlemoyer, and W. tau Yih. RA-DIT: Retrieval-augmeened dual instruction tuning. In ICLR, 2024.
[40] A. Liu, B. Feng, Wang. B. Wang, B. Liu, C. Zhao, C. Dengr, C. Ruan, D. Dai, D. Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint 245v:2205..04434 2024.
[41] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-vå v3 technical report. arXiv preprint arXiv:2412./9437 2024.
[42] T. Liu, H. Jiang, T. Wang, R. Xu, Y. Yu, L. Zhang, T. Zhao, and H. Wang. Roserag: Robust eetrieval-augmente generation with small-scale lims via margin-aware preference optimization. arXiv preprint 250. 2025.
[43] Z. Liu, Y. Chen, M. Shoeybi. B. Catanzaro, and W. Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint 1Xiv:2442.11004 2024.
[44] 7. Liu, X. Guo, F. Lou, I. Zeng, J. Niu, Z. Wang, J. Xu, W. Cai, Z. Yang. X. Zhao, et al. Fin-rl: large language model for financial reasoning through reinforcement learning. arXiv preprint 2503v:2503. /6252 2025.
[45] Z. Liu, W. Ping, R. Roy, P. Xu, C. Lee, M. Shoeybi, and Catanzaro. Chatqa: Surpassing gpt-4 on conversational qa and rag. In NewrIPS, pages 15416-15459, 2024.
[46] P. Lu, L. Qiu, K.-W. Chang, Y. N. Wu, S.-C. Zhu, T. Rajpurohit, P. Clark, and A. Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In ICLR, 2023.
[47] H. Ma, W. Xu, Y. Wei, I. Chen, I.. Wang, Q. Liu, and Wu. Ex-fever: A dataset for multi-hop explainable fact verification. In Findings of ACL, pages 9340-9353, 2024.
12

---

[48] A. Mallen, A. Asai, V. Zhong. R. Das, D. Khashabi, and H. Hajishirzi. When not to trust language models: Investigating efffcciivnnss of parametric and non-maraetrric memories. In ACI., 2023.
[49] Y. Meng. M. Xia, and D. Chen. SimPO: Simple preference optimization with reference-free reward. In NeurIPS, 2024.
[50] N. Muennighoff, H. Su, L. Wang, N. Yang. F. Wei, T. Yu, A. Singh, and D. Kiela. Generative representational instruction tuning. In ICLR, 2025.
[51] OpenAI. Introducing gpt-4.1 in the api, 2025.
[52] L. Ouyang, J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. NeurlPS, 35, 2022.
[53] O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of EMNLP, pages 5687-5711, 2023.
[54] L. Qian, W. Zhou, Y. Wang, X. Peng, H. Yi, J. Huang, Q. Xie, and J. Nie. Finol: On the transferability of reasoning enhanced Ilms to finance. arXiv preprint arXiv: 2025.
[55] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, 2023.
[56] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. EMNLP, 2016.
[57] Z. Shao, Y. Gong, Y. Shen, M. Huang. N. Duan, and W. Chen. Enhancing retrieval-augmenned large language models with iterative retrieval-generationn synergy. In Findings of EMNLP, 2023.
[58] W. Shi, S. Min. M. Yasunaga, M. Seo, James, M. Lewis, L. Zettlemoyer, and W.-t. Yih. REPLUG: Retrieval-augmented black-box language models. In NAACL, 2024.
[59] W. Shi, R. Xu, Y. Zhuang, Y. Yu, J. Zhang, H. Wu, Y. Zhu, J. Ho, C. Yang, and M. D. Wang. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In EMNLP, pages 22315 2223999, 2024.
[60] D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre. G. Van Den Driessche, Schrittwiesrr, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 2016
[61] A. Singh, J. D. Co-Reyes, Agarwal, A. Anand, P. Patil, Garcia, P. J. Liu, J. Harrison, J. Lee, K. Xu, A. T. Parisi, A. Kumar, et al. Beyond human data: Scaling self-training for problem-solving with language models. TMLR, 2024.
[62] H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J.-R. Wen. R1- searcher: Incentivizing the search capability in ilms via reinforcement learning. arXiv preprint arXiv:2503.05524 2025.
[63] H. Sun, Z. Qiao, J. Guo, X. Fan, Y. Hou, Y. Jiang, P. Xie, F. Huang, and Y. Zhang. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505..0 045588 2025.
[64] Z. Sun, Q. Wang, W. Yu, X. Zang, K. Zheng, J. Xu, X. Zhang, S. Yang. and H. Li. Rearter: Retrieval- auggeeete reasoning with trustworthy process rewarding. arXiv preprint arXiv:2501.0786) 2025.
[65] G. Team, Georgiev, I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arxiv preprint arXiv:2403.055300 2024.
[66] Thorne, A. Vlachos, C. Christodoulopoulls, and Mittal. Fever: A large-scale dataset for fact extraction and verification. In NAACL, 2018.
13

---

[67] H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal. Musique: Multihop questions via single-hop question composition. TACL, 10:539-5544 554, 2022.
[68] H. Trivedi, N. Balasubramanian, Khot, and Sabharwal. Interleaving retrieval with chain- of-thought reasoning for knowledge-intensive multi-step questions. In ACL, 2023,
[69] A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[70] P. Verma, S. P. Midigeshi, G. Sinha, A. Solin, N. Natarajan, and A. Sharma. Plan-RAG:
Planning-guine guided retrieval augmented generation, 2025.
[71] H. Wang, R. Li, H. Jiang, Tian, Z. Wang, C. Luo, X. Tang, M. Cheng, T. Zhao, and
J. Gao. Blendfilter: Advancing retrieval-augmentt large language models via query generation blending and knowledge filterig. In EMNLP, pages 1009-1025, 2024.
[72] L. Wang, H. Chen, N. Yang, X. Huang, Z. Dou, and Wei. Chain-of-retrieval augmented generation. arXiv preprint arXiv:2501 14342, 2025.
[73] I.. Wang, N. Yang. X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 12x 2022.
[74] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q V. D. Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. NeurIPS, 2022.
[75] Z. Wei, W.-L. Chen, and Y. Meng. InstructRAG: Instructing retrieval-augmented generation via self self-synthesized rationales. In ICLR, 2025.
[76] Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang. and Q. Gu. Self-play preference optimization for language model alignment. In ICLR, 2025.
[77] G. Xiong, Q. Jin, X. Wang. Y. Fang, H. Liu, Y. Yang, F. Chen, Z. Song, D. Wang, M. Zhang, Rag-gym: Optimizing reasoning and search agents with process supervision. arXiv preprint N.Vv:25002.1955) 2025.
[78] R. Xu, H. Liu, S. Nag. Z. Dai, Y. Xie, X. Tang, C. Luo, Y. Li, J. C. Ho, C. Yang, and Q. He. SimRAG: Self-improving retreeeal-augmented generation for adapting large language models to specialized domains. In NAACL, pages 1534-11550, 2025.
[79] R. Xu, W. Shi, Y. Zhuang, Y. Yu, J. C. Ho, H. Wang, and C. Yang. Collab-rag: Boosting augreeel generation for complex question answering via white-box and black-box Ilm collaboration. arXiv preprint arXiv:2504.044 04915. 2025.
[80] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng. B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09350 2025.
[81] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.151111 2024.
[82] A. Yang, B. Zhang, B. Hui, B. Gao, B. Yu, C. Li, D. Liu, J. Tu, Zhou, J. Lin, al. Qwen2. 5-math technical report: Toward mathematical expert model via seff-improvement. arXiv preprint arXiv:2409.
2024.
[83] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
[84] Z. Ye, R. Agarwal, T. Liu, R. Joshi, S. Velury, Q. V. Le, Q Tan, and Y. Liu. Evolving alignment via asymmetric self- self-play. arXiv preprint wxXiv2241.00000 2024.
[85] W. Yu, M. Jiang, P. Clark, and A. Sabharwal. Ifqa: A dataset for open-domain question answering under counterfactual presuppositions. In EMNLP, pages 8276-8288, 2023.
[86] Y. Yu, W. Ping, Z. Liu, B. Wang, J. You, C. Zhang, M. Shoeybi, and B. Catanzaro. Rankrag: Unifying context ranking with retrieval-augmented generation in Ums. In NeurlPS, pages 121156-121184, 2024.
14

---

[87] X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang. H. Sun, Y. Su, and W. Chen. MAmmoTH: Building math generalist models through hybrid instruction tuning. In ICLR, 2024.
[88] Z. Yue, H. Zhuang, A. Bai, K. Hui, R. Jagerman, H. Zeng, Qin, D. Wang, X. Wang, and M. Bendersky. Inference scaling for long-context retrieval augmented generation. In ICLR, 2025.
[89] A. Zhao, Y. Wu, Y. Yue, T. Wu, Q. Xu, M. Lin, S. Wang, Q. Wu, Z. Zheng, and G. Huang. Absolute zero: Reinforced self-play reasoning with data, arXiv preprint arXiv:2505.03335, 2025.
[90] Y. Zhao, Y. Li, C. Li, and R. Zhang. Multihiertt: Numerical reasoning over multi hierarchical tabular and textual data. In ACL, pages 6588-6600, 2022.
[91] Y. Zhao, Long, H. Liu, R. Kamoi, Nan, L. Chen, Y. Liu, X. Tang, R. Zhang, and Cohan. Doemath-eval: Evaluating math reasoning capabilities of Ilms in understanding long and specialized documents. In ACL. pages 16103-16120, 2024.
[92] Y. Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, and P. Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025.
[93] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, C. Cui, O. Bousquet, Q V. Le, and E. H. Chi. Least-to-most prompting enables complex reasoning in large language models. In ICLR, 2023.
[94] Zhu, W. Lei, Y. Huang, C. Wang. S. Zhang. Lv, Feng, and T.-S. Chua. Tat-qa: A question answering benchmark on hybrid of tabular and textual content in finance. In ACL, 2021.
[95] J. Zhu, Q. Chen, H. Dou, J. Li, L. Guo, F. Chen, and C. Zhang. Dianjin-rl: Evaluating and enhancing financial reasoning in large language models. arXiv preprint 17X1v 2025.
[96] Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li. H. Gao, S. Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406. 11931, 2024.
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claim made in the abstract is supported by the experiments in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in Appendix A.
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The proof are shown in Appendix B and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the details of the data collection for training and evaluations in Section 5, Appendix E- G.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibilityy which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes)
Justification: For training data, they are all public available data with open access in https: / //huggingf ace. co Acer datasets. We open-source code for reproducing our
results
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/nnips.. cc/
(ellli//guides//odeeeee wtt.ssuuwiiiiioooooooooooooooeoe for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cc/ ccc/public/geee hwlpc//guie///eieeeeee (ooeee/m/ooSudiiiii cestuummiiiiiiiii issisnnnn sisionooiccy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Experimental details can be found section 5 and Appendix G.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We conduct evaluation zero-shot manner using greedy sampling, all results of AceSearcher are deterministic.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide such information in Appendix G.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ httpp:s////uuuss..../ htte//uulc//Etiiieeeeee ceshhiiicniddnnnnne..cc
Answer: [Yes]
Justification:
The work conducted in this paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss both potential societal impacts and negative impacts in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
20

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide citations to all datasets and models used the paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [Yes]
21

---

Justification:
We provide details in Appendix I.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: Our human study has obtained IRB approval and we mentioned this in Ap- pendix
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We describe our experimental setup Section 5.1.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: / http://c/.c...eeeeee/22000/55 for what should or should be described.
22

---

A Limitations and Impact Statement
Linitations. While AceSearcher demonstrates strong empirical performance across wide range of RAG and document-ievel reasoning benchmarks, several limitations remain. First, our framework evaluated primarily on complex QA, fact verification, and document-level reasoning tasks; its applicability to other tasks such as open-ended generation, dialogue, or use of real-time tools remains to be explored, though our scope is comparable (or even broader) compared concurrent works [62, 30, 92]. Second, AceSearcher relies on a fixed retriever during training and inference. Joint optimization of retrieval and reasoning could offer further gains but left for future work. Third, our decomposition based pipeline introduces inference overhead, which may limit applicability in latency-sensitee settings. Nonetheless, as shown Figure 3(c), AceSearcher achieves favorable tradeoffs, and many strong baselines [70, 36, 72] also adopt multi-turn retrieval. Finally, due to resource constraints, we adopt iterative preference optimization (Online DPO) practical and efficient alternative fully online reinforcement learning. While this approach achieves strong results in our setting, exploring more expressive RL formulations may offer further improvements.
Impact Statement. This work advances the development of search-augmented LLMs capable of complex reasoning. By enabling smaller open-source LLMs to search and reason more effectively, AceSearcher reduces reliance on proprietary or extremely large models, which may have high computational or financial barriers. This can promote democratization of advanced Al capabilities in low-resource or domain-specifie applications, such as finance, scientific discovery, and healthcare.
B Derivation Step for Optimal Policy &' and pr
We aim to maximize the following objective:
To Fgg Pe-pot-la) (o a', a)] 3 DKL (pollPret) B Re-po(-is) [DKL (Foll
Since p and appear in separate terms, we can optimize them independently.
1. Optimal π for each z. For fixed z, consider the Lagrangian
(=(TT))=== Σ Tw, a' a'o,,a'l,,,2, 2)), - B Σ x (w,, a' [1. 2) In T(w, al 1g,
(B.1)
Teet (w, a' 21
w,a"
+ X
(w, a' (11,))=) -1)
(B.2)
Taking the functional derivative with respect to r(w, a' (1, 2) and setting to zero gives
rlg,a', a', a) B [In[(w, a' lllg,, 2) In In a' (q,2) + 1) + d 0.
Rearranging yields
In (w, a' (q, In mor a' alla.'l.,0 z) + ( r(q,a',a)
B
constant in w.a!
Hence, the optimal policy is:
" (w. a q.2)= = Z (q, 2) Free Feee 11 (frla.d..))
where
Σ (we a q, 11,2) exp 1 (3
Z (x(1))) 2)
wa.
Now compute G (T*)
n (w. a 9,2)
log
Teed [w a z) 3r(1, -log 2z
23

---

S0
r(g, a) -llog <a q,2) (a 9,2)
B log Zs (q, 2)
Therefore,
G (T²") E Ewaarr 18log Zx (x, log - (9,2)
Optimal p. Substitute ग back into Jq. Denote
-ba== Ez-p Blog (1, log p(= q) BR2mp log Prot(z [g) (q. z)
Prof = [g)
p(= 1 Q)
together with the constraint P(21)) 1. Introduce multiplier μ and form
Σ plz 9) In p(z (2) Prot(z 9)
(E plz 19) -1
C/p,u] Σ plz 1)8 2)
Taking
ac 0p(z q)
3og Z (q, ((zp(2) q) -nn -n 1g) 1) + 0.
Rearranging:
(1).
1
In pn 9) In Pret(2 ( log (x, 2) +
(3
The optimal policy is:
1
p² Zo(g) Pret Pref(2 (g.2x(9;2)
where
Zo(g) Σ Prer(z (n)
Combining these two results yields exactly the stated closed-form solutions
(z x Pret(z 4) (wse)))mattt exp ( (f)la.d.ao)
(B.3)
p'( d' 2) x Peelww, 9,2) exp () rla.dda))
(B.4)
C
Omitted Theorems and Proofs
C.1 Notion
Let B(r, z)) represent the by-ball of radius centered at x. For two positive sequences {a,,) and {b,, J.
an 2 bn if an Cb The l2 norm of vector x E Rd is defined as ||I||| := (Σ 1/2
A
sequence of random variables X,, is said to be (1(1) if X,, E, 0. that is, X,, converges to 0 in prob-
ability as n - oc. The Kulbbeck-Leibler (KI. divergence from a discrete distribution p to a discrete distribution g (defined
common support X') is given by T0KL. [p q) Exexp(a) log p(x) under the assumption that whenever p(x) 0, also has q(x) 0 for all X.
((x))

---

C.2 Main theorem
Recall the losses (4.2) and (4.5) are defined follows:
E (w)) Ir(a', qq a)] 5I... [uo(a', ,2, 2, w q) uerl w 9)]
(C.1)
Pe
1
(g
2) log
(++1) (g (2)
x)
CmDPO
- (a.gt 997 jouDee log pet
log
(C.2)
Pa
(2)
(at
x))
Pa
(g
xx)
To enable decomposition into a decomposer and a solver, require the following assumption: assumption C.1 (Conditional Probability decomposition), We assume the following decomposition holds:
po(a q) = po(z (z) Φ) Σ po(a g, po q,2))
We present the informal version of our theorem below. Formal statements are given in Theorems C.2 and C.3.
Theorem C.1 (Informal). Under regularity conditions, with high probability, the minimizer of the loss (C.2) at step ć is close to the minimizer of the loss (C.1). Furthermore, as increases, the minimizer converges the true parameter 0*,
be divided into components. The first component establishes the equivalence between loss (C.1) and loss (C.2) are equivalent. The second component shows that, the equivalence is established and the maximizer of loss (C.1) converges, the minimizer of loss (C.2) also converges.
Remark C.I. The main theorem
The proof's is organized as follows: In Appendices C.3 and C.4, we analyze the convergence properties of the maximizer of the population version loss (C.3) and sample version of loss (C.6) which corresponds exactly to loss (C.1). In Appendix C.5, we demonstrate the equivalence of loss (C.1) loss (C.2). Finally, in Appendix C.6, building on these results, prove that the minimizer of loss (C.2) converges as well.
C.3 Population Version
Based on the loss (C.1), define the population version loss as
L(0 0,-1) B(g.a)<pp-(( Exmpot s) ( (r(a', q,a)]]]]
- DxI. (us(a', (a', a, 4) us, (a', 2,', |4)). (C.3)
Define the operator M O,
M(0) arg max L(l') ()). M'E
where θ represents the parameter space. Notice that it is natural to assume that 0* satisfy the self-consistency, i.e. 0" M(")). So the first assumption will be: assumption C.2 (Self-consistency).
= (MM(()
assumption C.3 (A-strong Concavity). There is some À > 0 such that
T.(0] | 0*) - L(O2 0*) (VI 0*), 01 02) s - - [10 -221 for all 01, O2 E B(r, 0*).
(C.4)
Definition C.1 (First-order stability). The functions {L(- 0); 0 θ satisfy the First-order stability condition over B(r, 0*) if
(V) 8*) VL(M(()) 0)]l, - 41 8* ll
for all 0 B(r, 0*).
assumption C.4. Assume the functions (T(- (L(-10). [0), E Q} satisfy the First-order stability condition (C.1).
25

---

Proposition C.1 (Population Version), For some radius r > 0 and pair (μ, A) such that 0 S < A. suppose that the Assumption C.1-C.4 hold, then the population operator M is contractive over B(r, 0*), in particular with
[ - 1- - 11 1 - 12 for all E B(r, 8*).
Proof. By the first order optimality condition, have:
(VL(0" 100))0-00) - 4/
VL(00) (())) 0²) 0
(L(M(()-1) 0)-1), 8 (0(-1) 5 40
(L(M(0)-1) 00-1), ((0-11)) 50
Combine the two terms,
(VL(0* (7) VL(M((--11 (0-1)) - -0) S.
Thus,
(VL(0") 0*) (L(M(0.-1) (0)) 07)
(VL(M 0*) - VL(M -00
For the right-hand side, by Cauchy-Schward inequality.
(V 0*) - VL(M(0--=) 07) <
10) -L000-00000000000000000001-11110
By Assumption C.4,
(VL((((--I) 0²) (L(M(O--1) (0- 100 - -
For the left-hand side, by Assumption C.3,
[M - 11 s L(0") 8*) - L(M(()-1) (r) (VL(0) ( -00)
2
(0,1)) - 0-1 L(M(0--1) 10t) - L(O0 10 (VL(M(()-1) 100),0- - -M
2
Hence,
(1 0-113 (VL(0" 00) 10) -0). (C.5)
Combining all.
( 0-.1 - (1) -
Remark C.2. This theorem follows the idea in [3]. suggests that, under self-play procedure, the algorithm progressively approaches the true underlying distribution. This behavior is characterized by a contraction parameter K. which ensures convergence toward the ground-truth parameter 6* The incorporation of an intermediate reasoning step smooths the local optimization landscape, rendering the loss approximately convex and thereby facilitating convergence to the global optimum.
C.4 Sample Version
We define the below sample version: assume we have the data
D Dy.a = {g:, az 1-
The loss will be:
(8 0.-1) = E(aaa> [Eerpo (11)) (f []]]]
- DDx (uolaa w. q) up.,, (a', 2, (9), (C.6)
26

---

where ý represents the empirical distribution defined as
1 N Σ alq.aa = (qi a;)}.
p(g. a)
We also have the similar convergence property. Similar to the population version, define the sample-based operator My Q + t
Mx(0) = arg max LN (0' 0). er
For a given sample size N and tolerance parameter e E (0, 1), define SM (N, e) as the smallest scalar such that
sup [| ( Suail (N. (E
(C.7)
with probability at least 1 -. Proposition C.2 (Sample Version). Suppose that for all 0 E B(r, 0"), the mapping M satisfies
[IM(0,-)) 1- S - 1100-11 0-112
with probability at least c. Then have
- X 1100- 00 Cunif eN,)) for all E B(r,07
MM (0.-1) 10 S with probability at least
C.
Proof. The result follows directly from the triangle inequality:
122 S (0t - 0" 1 + [[Mx((0-1)) (0--1--M(0--))))
1 s x [10 00112 +C3
curif (N,c).
C,5 On the Equivalence with DPO
In the deterministic setting where the maximum and minimum rewards are selected depending on the data tuple ((amuxx _ wxx)) (aminn mmin wimin) a,, we note that in practice the construction of positive and negative samples vary, some containing or (z, w), and others including full triples such as (a', z, w). For simplicity, we unify the representation and consider the minimal component shared across all formats, namely the tuple (a', 2,),, This process can thus be interpreted as observing a finite dataset:
m' is fixed and the responses with
D (((att=tt
2T w ), as, 9:)
Then the DPO loss will be:
Lm (e-1) = -E -Es(at a((attat,,tt) (-o-===---)a)----
log (mmro log ua(at xt, wi 4) log ua w (011) us (a 2
1
(C.8)
uo u (at, zt, wt [
To demonstrate the closeness between the loss (C.6) and the loss (C.8), we first show that, with high probability, optimizing the loss (C.8) over the dataset D equivalent to maximizing the original reward up scaling factor.
Specifically, we can derive closed-form solution for Equation (C.6) at step t:
uaz (a', z, w. 9) x ug (a', 2, w q) exp ( r(d 9.0)) (.))
(C.9)
27

---

where or denotes the ground truth parameter at step t. Accordingly, the reward function r(a', a) can be written as ,,, (a', q. a) to emphasize its dependence on the true reward parameter 7 Specifically, consider the dataset D, which follows the following deterministic model:
P((a x=)= > (a ,-,,,,,,00
= if Tng (a (. > Tng (a: a),
(C.10)
indicating that we always select a. the positive sample. To approximate this deterministic behavior, we introduce the a-BI model:
eT (at .9.a)
P((a 22t, w+) (a ,2 , w 11))=
(C.11)
exrpp (at. (.00) + am (aT.9.0)
As a - oc, the a-BT model becomes close to the deterministic model (C.10), Then given the above dataset DD we define the following data set
Da [((a0t 20 w.,e), (a..: Wo,s). Oi q:)
where (la a,, w ;, (a 20 ()):)) is generated according to the a-BT model (C.11). To ensure the closeness between the dataset and Da. we have the following lemma:
assumption C.5 (Reward Seperation Condition). Assume that given (g:a), for any
((a ,,,,,=+,,,, wt), (a_,= ;- w))), there exists ê such that [rng (at -q,a) (a, ,baa] 8.
Lemma C.I. Suppose the Assumption C.5 holds, given there exists op 2 1 Nx4
F(D Dao)
(C.12)
2'
Proof. We start by bounding the probability of disagreement between two actions:
2g (a .9.0)
1
P( 20,0 wh + (at 2,,, wt "" (at 9.0) + e (a aaa)
1 gag
The total probability that the datasets D and Da differ is bounded by
N
P(D + Da)
Σ
P(a + +0,)5
N
1 + gand
log N 8
Given ox
conclude that
PPDD = D # 22)21-1
we can take our data generated according to the Qg-BT model. In this case the new reward will be
Foom; (a', q, a) ooring (a', q,a).
(C.13)
Under this model, the minimizer of the loss (C.8) can be obtained via two-step optimization procedure [55]: Step 1: minimize the negative log-like likelihood to obtain the reward:
LNNuL(? (o) = -Eiiat,st st,at,st,wt))) wt) -=- ).,
log a (Foon (g.t.g.)= Fao.(("....a)) (C.14)
Denote the minimizer as Fa. (a', - 9. a). Step 2: maximize the reward F Ax. (a', q, a):
(0 0.-1) = B(a.))rrppp (+) Ex (, (-1a,)) (e (a', a]]] AIDRODKA (wo(a' 2, w. a) a) || uo._ l.w.gg.a) (C.15)
28

---

The solution will be
"g, (a' 2, W/ 2, a) ( w, q,q a) exp
1
Fap 3w, Fx, 9. (dd..a)))
(C.16)
BDDPO
This expression identical to Equation (C.9), except that it uses a different parameterization of the reward. Specifically, the reward function nn (a' 9. a) is parameterized by the ground truth and hyyperparameter og. To ensure uniform consistency of the maximum likelihood estimator, we invoke the following lemma, which is modified from Theorem 5.7 in [69]. This result guarantees that the minimizer in Step converges to the true reward function Fau.t (a' q. a). We need the following assumption:
assumption C.6. Suppose that there exists constant Ca 0, for every 0, such that:
sup sup LNNLL(T (x)) - (N [o) - 0,
(C.17)
a (c)
where I' represents the parameter space and
sup (o) 00) x 2
sup
- (LNLL(?)] ao) - LNL an)) < O.
(C.18)
Lemma C.2 (Uniform MLE Consistency). Let INNNL((7 | oo) be the negative log-likelihood function, and let ( on) denote its expected version. Let Assumption C.6 holds, then for the sequence estimators YN obtained form minimizing the loss (C.14), we have: given c > 0, there exists N1, when N 2 Ni, for any ag [Ca., xc),
IP (I 7i l12 5c 2 - 2
(C.19)
Proof. For given e, according to the Equation (C.18), there exists C,,NL. such that:
sup
sup 7: 2e
-(INLL(V on) (1 on)) -CC,NLL-
aoE(ca,
For CL, according to Equation (C.17), there exists N1, when N N1, for any ag [ca, oo),
F F(LNNNL(FINa (o) (FNI ao)l - Ce,NLL
4'
P PILNNNI an) LN ao)l N.,LL
Since 7N is the minimizer of loss (C.14), for any og [ca. 00), we have:
IN.NLL oo) & LN.NLL(% (0))
Consequently,
P (LNUL ao) LNLL(70 ao)) 2 2C,,NLL
6 2
2 1
Thus for any ag E [Ca, 20).
P(IfiN, - 78ll2 <)) 2 P(-(IaLL(t)) oo) - INLI(70 oo)) 2
20,,NLL
6
2
1
3
2
Having established the necessary groundwork, we are now ready to present Theorem C.2, which establishes the equivalence between the minimizes of the two loss functions:
Theorem C.2. Assume Assumptions C.5 and C.6 hold, given e, there exists N1 and 8mDPO 2 log x B, the minimizer of loss (C.8) ODDDPO will satisfy:
0 mDDD 07 0:|11 2() e,
TP
(C.20)
where 07 is defined in Equation (C.9).
29

---

Proof. First, according to the Lemma C.2, there exists Ni. if we define the event f2 a Vi ll2 I e),, we have:
P( 2 2
Secondly, just choose the sample size of D as NK, define the event f22 = (D =Dag ). by Lemma
C.I, when we take or) 2 log +
V
we have:
P(22) 2 6
2
Since Ca is a constant, we may, without loss of generality, take og 2 analysis to the event S2 O Ω2, which occurs with probability at leastl Conditioned this event, the data can be viewed as being generated from the ao-BT model (C.11). Consequently, the minimizer of the loss (C.8) coincides with that of Equation (C.16):
log 22 Henceforth, we restrict 8
N2
" (a', 2, q) uo,_, (a', 2, w g) exp
SDDPO Fan, (a', q.a)
uo (a', 2, g) exp
Ma (d' (d..).a)
BiDDO
log N1
8, controlling the distance be- tween ôt, mDPO and ff reduces controlling the distance between 9N1, and 7; as established by Lemma C.2. Consequently, we obtain:
Compared to the solution in (C.9), when PMDOO = ao,
IP [ P..DDO - 0:11 2 <e
This concludes the proof.
C.6 Convergence Property of DPO
Finally, combining Proposition C.2, we conclude that the sequence 0r, converges as t increases.
We formally state the following theorem:
Theorem C.3. For given iteraton number T, for some radius r > and pair (u, A) such that 0 < < X, suppose that the Assumption C.1-C.6 hold and assume (e Cunif (N, e)) (1 +r,, then with probability least 1T
(T+ + 1)e, we have:
@T..mDPO 001/= < (5)
1 x
10ter 0-112 +
Cumif (n, E)
Proof. Notice that 07 = Mx (0 (.mD..))))))) apply Proposition C.2, we get:
110, 0 @.1.m 0%
+ (N, c)
with probability at least - Combining Theorem C.2,
00.mDDo 0-
1.-1. 8-| + SMr (N,c),
with probability at least 1 - 2c. Notice that +c+ Cupif (N, c)) s (1 Based on this, we can perform iteration:
f)r, then DDD E B(r, 0*).
@TpDDPo 81112 - / 0 1.mDPO - 00/
+et CM + (N.
0 Ør @T-2, 8-
+ Si (N,
T (4)
1
110rer - 0-11 +
T-1
Σ
11 (e + Saril C (N,
s=0
u T
|10ref - 0-112 +
(e + Cup (N,)))
#
with probability at least (T + 1)c.
30

---

D Information for Test Datasets
The information of the test datasets used in AceSearcher listed in the following table. Note that We conduct evaluations on all questions from StrategyQA and Bamboogle, and the first 500 questions from the development sets of the other datasets following existing studies [68, 57, 36]. For dataset in EoaMathEval, we use the testmini version as the evaluation set to compare the performance of AceSearcher and baselines.
Table 4: Descriptions of datasets used in AceSearcher, For SimpLong and CompLong, we use text-embedding-2 to retrieve top-10 relevant context before generate the answer.
Dataset
Description
2WikiMHQA [21]
2WikiMultiHopQA is multi-hop question answering dataset built from Wikipedia, where each question requires reasoning over two distinct articles. It emphasizes information synthesis across multiple documents for accurate answer retrieval.
HotpotQA [83]
HotporQA is crowd-sourced multi-hop QA datasct where each question demands reasoning over multiple Wikipedia passages. It also includes supporting fact annotations to promote explainability in QA systems.
Bamboogle [53]
Barnboogle is multi-hop QA dataset constructed using Bing search engine snippets. It presents naturally occurring. challenging questions requiring reasoning over diverse web smippets rather than structured sources like Wikipedia.
MusiQue [67]
MusiQue is multi-hop QA dataset featuring real-world questions from community forums like Quora and Yahoo Answers. It targets complex questions requiring synthesis across multiple evidence passages, each carefully annotated.
HOVER [28]
HOVEE is multi-hop QA dataset with annotated supporting facts, built entity- linked Wikipedia documents. It stresses explainable reasoning by providing intermedi- ate evidence chains.
EXFEVER [47]
ExFEVER extends the FEVER dataset by introducing multi-hop claims requiring evidence from m documents. It is designed to support research on fact verification and evidence- reasoning.
DMss (DocMath SimpShort)
dataset reannotated from TAT-QA [94] and FinQA [7], consisting of short financial documents with single table for simple numerical reasoning.
DMcs (DocMath CompShort)
dataset reannotated from TAT-HQA [35], consisting short single-table documents for complex numerical reasoning. including hypotheticals.
DMst. (DocMath SimpLong)
dataset reannotated from MultiHiertt [90], consisting of long multi-tuble financial documents for simple reasoning realistic contexts.
DMCl. (DocMath CompLong)
A dataset of long. structured financial documents requiring multi-step compositional numerical reasoning.
31

---

E Details of Training Data
We provide the data composition for SFT and RFT, including their corresponding tasks, links to access the data, and the number we use in each stage in Table 5. To avoid data contamination, we follow the instructions in the MusiQue repository and remove the training data with overlapping IDs from NQ and Squad avoid data leakage.
Table 5: The data composition for SFT and RFT stages.
Dataset
Task
Link
Count
Data composition for
NarrativeQA [32]
Context-rich QA
httpe///ugginnaac...//aasasstss narrativeqa
20000
SQuAD 1.1 [56]
Context-rich QA
10000
SQuAD 2.0 [56]
Contexi-rich QA
10000
TAT-QA [94]
Context-rich QA
12000
tree/maaster/aataseee_rr
FEVER [66]
Context-rich QA
10000
DROP [16]
Context-rich QA
20000
Quoref [12]
Context-rich QA
20000
quoref
ROPES [38]
Context-rich QA
https///aggiiaacoo.oo/aattttttlt ropes
10000
NQ [33]
Context-rich QA
https://dl.
hwtp://aaaaaiccclssscccoooo/aapp 20000
GSM8K [1]]
Question Decompusition
https://huggingfaen
7000
gamék/rreeerrsoorattii
Con-FinQA [9]
Question Decompusition
1000
StrutegyQA [19]
Question Decomposition
https:
1600
StrategyQA
IIQA [85]
CoT
https://githut: http://thw...cmmmm/////////////a 2000 dataset
TabMWP [46]
CoT
http:///poopttggtittbb...nnnnnn dataset
10000
GSM8K [11]
CoT
https: https://uuggingafcc.co/// co.battasets//ppnasi
7000
MathInstruct-CC [87]
CoT
https://huggingface.com hot../aaaaaatttse/t//ta.aaaa KathInstruct
10000
Mathlnstruct-POTT [87]
CoT
https: htwps///gggggggg..gcoocoo wttp:/aa..cccoccaaeeeeeeee htw.g/ggw.a...aaaaattt///aaaaaaa
10000
NathInstruct
TOTAL
180600
Data composition for RFT
HotpotQA [83]
RAG
10000
2WikiMQA [21]
RAG
10000
2hikiMultihopQA
HOVER [28]
RAG
https://gttaab.cmmmvvvrre/////
10000
GSM8K [1]]
Context-rich Reasoning
https://huggiiggface
7000
gansh//iiewe//sccaaiic
TabMWP [46]
Context-rich Reasoning
https://promptpg. github.io/index.
10000
dataset
ConvFinQA [9]
Context-rich Reasoning
2000
Total
49000
32

---

Prompt Templates
F1 Prompts for Direct RAG
You have the following context passages: (context)
Given the question: "[question]" )" as well as the context above, please answer the above question with one list of entities with the given context as the reference. Your answer needs to be a span with one or list of entities.
Figure 6: Prompt for direct RAG on complex question answering tasks.
Answer the following questions with SUPPORTED or NOT SUPPORTED with the given context as the reference.
Question: [question] Context: [context]
Your answer should only be SUPPORTED or NOT_SUPPORTED.
Figure 7: Prompt for direct RAG on fact verification tasks.
You have the following passages and table:
Passages:
[passage)
Tables:
[table)
For the question "(question)", write Python program to solve the question. Store the final result in the variable
Figure 8: Prompt for direct RAG on document-level reasoning tasks with PoT.
You have the following passages and table:
Passages: [passage]
For the question "[question]" J", reason step by step to calculate the final answer. Please use \boxed(] to wrap your final answer.
Figure 9: Prompt for direct RAG on document-level reasoning tasks with CoT.
33

---

F.2 Prompts for Decomposition
Please break down the question "(question] ]" into multiple specific sub-questions that address individual components the original question.
Mark each sub-question with #### at the beginning. If you need to refer to answers from earlier sub-questions, use #1, #2, etc., to indicate the corresponding answers.
Decomposed question:
Figure 10: Prompt for question decomposition on complex question answering tasks.
Please break down the claim "(claim)" " into multiple smaller sub-claims that each focus on a specific component of the original statement, making it easier for model verify. Begin each sub-claim with ### If needed, refer to answers from earlier sub-claims using #1, #2, etc.
Decomposed claim:
Figure 11: Prompt for question decomposition fact verification tasks.
You have the following passages and table:
Passages:
(passages)
Tables:
(tables)
Please break down the question "(question]" into multiple specific sub-questions that address individual components of the original question, with the table and passages as the reference. Use ### mark the start of each sub-question.
Decomposed question:
Figure 12: Prompt for question decomposition on document-level reasoning tasks.
F33 Prompts for subquestion answering
You have the following context passages: passages)
Please answer the question "(subquestion]" with a short span using the context reference. If no answer found in the context, use your own knowledge. Your answer needs to be as short possible.
Figure 13: Prompt for subquestion answering on complex question answering tasks.
34

---

You have the following context passages: (passages)
Please verify whether the claim "(subquestion"" is correct using the context as reference. If answer is found in the context, your own knowledge. Please only output Yes No and do not give any explanation.
Figure 14: Prompt for subquestion answering on fact verification tasks.
You have the following passages and tables:
Passage:
[passages]
Table:
[tables)
For the question "(subquestion)" )", write a Python program to solve the question. Store the final result in the variable ans.
Figure 15: Prompt for subquestion answering on document-lever reasoning tasks with PoT.
You have the following passages and tables:
Passage:
(passages)
Table:
(tables)
For the question "[subquestion )", reason step by step to calculate the final answer. Please use \boxed[ to wrap your final answer.
Figure 16: Prompt for subquestion answering document-level reasoning tasks with CoT.
F.4 Prompts for final answer generation
You have the following passages: [passages]
You are also given some subquestions and their answers: subquestion #1: (subquestion_]] Answer: [answer_] 1]
# subquestion #2: (subquestion_2) 2 Answer: (answer_2)
Please answer the question "[the original question]" with short span using the documents and subquestions as reference.
Make sure your response is grounded in documents and provides clear reasoning followed by a concise conclusion. no relevant information found, use your own knowledge.
Wrap your answer with <answer> and </answer> tags.
Figure 17: Prompt for final answer generation complex question answering tasks.
35

---

You are given some subquestions and their answers:
# subquestion #1: (subquestion 1) Answer: (answer 1]
subquestion #2: (subquestion 2) Answer: {answer_22 __]
Please answer the question _rt.oriiialuuueitiinnnnoaooo |" with only Yes or No using the subques- tions reference. Provides clear reasoning followed by concise conclusion. If no relevant information is found, use your own knowledge.
Wrap your answer with <answer> and </answer> tags.
Figure 18: Prompt for final answer generation on fact verification tasks.
You have the following passages and table:
Passages: [passage]
For the question "(question)" "", here is referenced breakdown: (decomposition).
Write Python program to solve the question. Store the final result in the variable ans.
Figure 19: Prompt for final answer generation on document-levés reasoning tasks with PoT.
You have the following passages and table:
Passages: (passage)
For the question "[question]" here is referenced breakdown: [decomposition).
Reason step step to calculate the final answer. Please \boxed to wrap your final answer.
Figure 20: Prompt for final answer generation on document-level reasoning tasks with CoT.
36

---

E,5
Prompts for InstructRAG
Read the following documents relevant the given question: (question}
Documents: (documents)
...
Please identify documents that are useful to answer the given question: "[question]". If none of the documents is aligned with the answer, in that case, you have to explain the answer only based on your own knowledge, without referring to the provided information.
Note that the question may be compositional and require intermediate analysis to deduce the final answer. Make sure your response is grounded and provides clear reasoning details followed by concise conclusion. Your answer should be in short span with few keywords. Use <answer> and <lanswer> tag to mark your final answer.
Figure 21: Prompt for InstructRAG on complex question answering tasks.
Read the following documents relevant to the given question: {question |
Documents: [documents]
Please identify documents that are useful to answer the given question: "[question)" If none of the documents is aligned with the answer, that case, you have explain the answer only based on your own knowledge, without referring to the provided information.
Note that the question may be compositional and require intermediate analysis to deduce the final answer. Make sure your response is grounded and provides clear reasoning details followed by a concise conclusion. Your answer should yes or no only. Use <answer> and </answer> tag to mark your final answer.
Figure 22: Prompt for InstructRAG on fact verification tasks.
Additional Implementation Details
G.1
Implementation Details for SFT
For SFT, we set the batch size to 64 for every example, and set the learning rate as Table With maximum number of tokens to 2560.
Table 6: Results for different model sizes for SFT.
Model Size
Learning Rate
Warmup Steps
AceSearcher 1.5B
5e 6
5%
AceSearcher 8B
le 6
5%
AceSearcher 14B
le 6 5%
AceSearcher 32B (LoRA)
le 5 5%
G.2 Implementation Details for RFT
We set the hyperparameters m 3, m' 4, and 1.0 when generating multiple rollouts. Examples with identical maximum and minimum rewards are discarded. For RFT, use B 0.1
37

---

Table 7: Results for different model sizes for RFT.
Model Size
Learning Rate
Warmup Steps
AceSearcher 1.5B
le 6
5%
AceSearcher 8B
5e 7
5%
AceSearcher 14B
5e 7
5%
AceSearcher 32B (LoRA)
le 6
5%
and run for the DPO for 2 iterations by default. All models are optimized using Adam with 31 0.9 and 82 0.98, and experiments are conducted 8 NVIDIA A100 GPUs.
G.3 Implementation Details for Baselines
We implement and evaluate a variety of baselines using standardized decoding and prompting config- urations ensure fair comparison. For Qwen-3, follow the official guidance" adopt distinet sampling strategies depending on the task setting. In thinking mode (enable, thinking-True), we temperature 0.6, top-p 0.95, top-k 20, and min-p generation. Greedy decoding is explicitly avoided to prevent performance degradation and repetitive outputs. In non-thinking mode (enable_the sanolle_thikkin-fallsee we slightly increase the temperature to 0.7 and reduce top-p to 0.8 while keeping top-k and min-p unchanged. In practice, we find that using the thinking mode leads slightly better performance despite being slower. For R1-distill models, we set the maximum generation length to 32,768 tokens and use temperature 0.6, top-p 0.95. In Plan-RAG, we incorporate 3-shot demonstrations the prompt to guide the model toward producing outputs in the correct format. For InstructRAG, we use the same SFT training set as AceSearcher and generate CoT--style demonstrations tailored to context-rich QA datasets. For Llama-4, GPT-4.1, and GPT-40,, we use greedy decoding (temperature = 0) for consistency with their default inference behavior. For IRCOT and RAG-Star, we reproduce results by following the original repositories and hyperparameter settings. For these methods, we tune the number of retrieved passages from {5, 10, 20} and report the best performance. We refer to other baselines" reported numbers in the corresponding paper.
encourage diverse yet coherent
H
Additional Experimental Results
heta-0.05
beta-0.5
60
Ours
Ours
beta=0.1
Dragon
Dragon
Contriever
50-
Contriever
40
30-
56
0
Qwen-3
Plan-RAG AceSearcher
Qwen-3
Plan-RAG Scarhhre
Fact Verif
Reasoning
(a) MusiQue
(b) HotpotQA
Figure 23: Parameter Study B
Figure 24: Effect of different retrievers.
Effect of 3. We study the effect of 3 in preference optimization with Llama-3.1 1-1BB as the backbone. and find that AceSearcher generally robust this parameter, with 3 0.1 leads slightly better performance.
Effect of Different Retrievers. We evaluate AceSearcher and representative baselines (at the 8B scale) using two different retrievers: Dragon and Contriever". Overall, the E5 retriever achieves
38

---

Table 8: Performance comparison across models and prompting methods.
Model
Prompt Method
StmpSSoort
CompShort
DMstupLLoog
Comppoonng
Avg.
AceScarcher-32B
PoT
89.5
84.0
53.0
43.0
66.1
AveSearcher-14ke
PoT
84.0
82.0
49.0
39.3
62.4
AceSearcher-8B
83.0
32.3
59.0
AceSearcher- 1.5B
PoT
66.5
77.5
39.0
18.0
47.6
AceSearcher-321
CoT
73.5
70.0
50.0
33.0
54.5
AceScarcher-1ae 14B
CoT
78.5
75.5
44.0
34.7
57.0
AceSearcher-80
CoT
44.0
31.5
30.0
15.7
28.5
AveSearcher-1.50
CoT
37.5
32.0
23.2
the best performance, supporting our hypothesis that stronger retrieval models yield more relevant passages and thus enhance answer quality. Notably, AceSearcher consistently outperforms baselines across different retrievers, demonstrating its robustness to retrieval choices.
Comparison of CoT and PoT for Document-level Reasoning. Table 8 presents comparison between Program of Thought (POT) and Chain of Thought (COT) prompting methods across four evaluation settings. POT consistently outperforms COT across all tasks, with notable improvements on both simple and complex reasoning benchmarks. For example, across models, POT yields higher average scores than COT on DMCompL ong (e.g., 43.0 vs. 33.0 for AceSearcher-32b, and Simint demonstrating its advantage in guiding structured reasoning.
(e.g., 89.5 vs. 73.5 for AceSearcher-324 These results highlight the effectiveness of POT enhancing model performance on decision-making tasks requiring multi-step reasoning, regardless of model scale.
I
Case Studies
Details of Human Study Our human study has received IRB approval from institute. The below the form used in human evaluation:
You are provided with:
The original complex question
A proposed decomposition into subquestions
Please assess the overall quality of the decomposition using the criteria below.
Relevance: Do the subquestions help solve the original question?
Completeness: Are all important aspects covered?
Coherence: Is the breakdown logically structured and easy to follow?
Usefulness: Does the decomposition make the reasoning process casier more inter- pretable?
Please rate the overall quality of the subquestion decomposition.
I: Very Poor:
Subquestions are irrelevant, incomplete, or incoherent.
2: Poor: Some relevance, but major gaps or unclear logic.
3: Fair: Moderately helpful with minor issues in coverage or clarity.
4: Good: Clear and mostly complete decomposition.
5: Excellent: Subquestions are highly relevant, complete, and well-structured, with proper reference to answers from previous subquestions.
Details of Case Study
We present two case studies from the MusiQue, 2WikiMHQA and DoeMathEval datasets Tables 10 and 11. Despite its larger size, the Qwen3-32B -eresoinB-r model struggles with handling complex
39

---

questions. In contrast, AceSearcher effectively decomposes each question into manageable sub- questions, generates accurate answers for each sub questions, and performs multi-step reasoning to arrive at the correct final answer.
Table 9: A case study from MusiQue dataset.
A sample from MusiQue
Question: In which state is Vera Barbosa's place of birth located?
Ground Truth Answer: Lisbon District
AceSearcher-BB
Sub-question 1: Who is Vera Barbosa?
Answer 1: a Portuguese track and field athlete Sub-question 2: Where was Vera Barbosa born?
Answer 2: Vila Franca de Xira
Sub-question 3: In which state is #2 located?
Answer 3: Lisbon District
Final Answer:
The documents state that Vera Barbosa is a Portuguese track and field athlcte and was born in Vila Franca de Xira. Additionally, Vila Franca de Xira is located in the Lisbon District. Therefore, Vera Barbosa's 's place of birth, Vila Franca de Xira, is in the Lisbon District.
Lanswer>Lisbon District</anser>>
Qwen3-32B Reasoning
<think>
Okay, let's see. The question is asking about the state where Vera Barbosa's 's place of birth is located, First, need to figure out which Vera Barbosa they're 're referring here because there several people named Vera Barbosa in the provided contexts.
Looking at the first entry: Vera Barbosa, born 13 January 1989 Vila Franca de Xira, Portugal. That's Portuguese athlete. Then there's 's another Vera Krasova, a Russian model, and Vera Begié, a Croatian athlete. There's is also mention of place called Barbosa in Colombia and Brazil. But the key here is the name Vera Barbosa specifically.
The asking about Vera Barbosa's birthplace state. The first context says she was born Vila Franca Xira, which in Portugal. However, question is phrased as "In which state Vera Barbosa's place birth located?" So, states are typically administrative divisions in countries like the US, Brazil, etc. Portugal is divided into districts, not states. But maybe the user is using "state" in general sense here.
Wait, but looking other entries, there's Carlos Barbosa in Brazil's Rio Grande do Sul state. and Bárbara Barbosa a Brazilian footballer. But the specific Vera Barbosa mentioned in the first paragraph the Portuguese athlete. Her birthplace is Vila Franca Xira. Portugal. Since Portugal doesn') have states, but the question uses "state," perhaps the answer is Portugal. Alternatiell, if the user is confused and actually refers to different person, but based on the given data, the only Vera Barbosa with birthplace listed is the Portuguese one. Therefore, the answer should be Portugal. </think>
<answe>PPrtugaalk/aswwrr
40

---

Table 10: case study from 2WikiMHQA dataset.
sample from 2WikiMHQA
Question: Which film the director who was later, The Silver Treasure or Taxi To Paradise? Ground Truth Answer: Taxi To Paradise
AceSearcher-SB
Sub-question 1: Who directed The Silver Treasure?
Answer 1: Rowland V. Lee
Sub-question 2: Who directed Taxi To Paradise?
Answer 2: Adrian Brunel
Sub-question 3: When was director of #1 hor??
Answer 3: September 6, 1891
Sub-question 4: When was the director of #2 born?
Answer 4: 4 September 1892
Sub-question 5: Is the year of #3 later than #4?
Answer 5:
Final Answer:
From the documents, "The Silver Treasure" was directed by Rowland V. Lee, born on September 6, 1891 (Q3), and "Taxi To Paradise" was directed by Adrian Brunel, horn on September 1892 (Q4). Comparing these dates, Adrian Brunel born later than Rowland V. Lee (QS confirms this). Thus, the film with the director born later is "Taxi To Paradise."
Tanswer>Taax To Paradise</answer>
Qwen--32B Reasoning
<think>
Okay, let's The user is asking which film has the director born later between The Silver Treasure and Taxi Paradise.
First, need to find the directors of both films. From the context:
The Silver Treasure (1926) is directed by Rowland V. Lee. Taxi to Paradise (1933) is directed by Adrian Brunel.
Now. need to check their birth dates. The context doesn't provide birth dates for these directors. So I'll have to rely on my own knowledge.
Rowland V. Lee was born on April 22, 1894. Adrian Brunel was born on October 2, 1887. Wait, Brunel was born earlier than Lee? Wait, no: if Brunel was born in 1887 and Lee in 1894, then Brunel was born carlier. Therefore, Rowland V. Lee was bom later. So The Silver Treasure's director was born later than Taxi to Paradise's director. Therefore, the answer The Silver Treasure. </think
<answer> The Silver Treasure<answer>
41

---

Table 11: case study from DMcs subset of the Evcl dataset.
A sample from DMcs subset of vath
Table:
Year
Segment
Americas
Europe (Amounts in thousands)
Asia
Total
% of Total
2019
TS
S67.728
$3,285
$646
$71,159
90%
2019
HPP
$5,294
S771
$1,837
$7.902
10%
2019
Total
$72,522
$4,056
$2,483
$79.061
100%
2019
& of Total
92%
5%
3%
100%
2018
552,034
$9,059
$1,344
$62,437
86%
2018
HPP
$8,424
$1,266
$789
$10,479
14%
2018
Total
S60,458
$10,325
$2,133
$72.916
100%
2018
% of Total
83%
14%
3%
100%
Table 12: The following table details the Company's 's sales by operating segment for fiscal years ended September 30, 2019 and 2018. The Company's sales by geographic area based on the location of where the products were shipped or services rendered are follows: Substantially all Americas amounts are United States.
Question: What the percentage change in Asia sales between 2018 and 2019 if the 2019 sales is doubled and increased by another thousand? (in percent)
Ground Truth Answer: 151,5705
AceSearcher-SB
Sub-question 1: What was the value of Asia sales in 2019?
Sub-question 2: What is the value of Asia sales in 2019 after doubling it?
Sub-question 3: What is the value of Asia sales in 2019 after doubling and then adding 400,000?
Sub-question 4: What was the value Asia sales in 2018?
Sub-question What the net change in the value Asia sales from 2018 2019, after adjusting the
2019 value?
Sub-question 6: What is the percentage change in the value of Asia sales from 2018 to 2019, after adjusting
the 2019 value?
Final Answer:
Given data
as asia_sales _019 2483 in thousands (Q1) asia_sales 2018 2018 2133 # in thousands (Q4)
Adjust 2019 sales: double it and add 400 thousand (Q2, Q3)
adjusted_ _0____211___________ sales baaia_sals__. 2019 2) 400
Compute value change (Q5)
change laangeiin_saleee innsalessseaeeeeeeeeeeeeeeeeeeee = adjusted_ 2019_sales
2018
Compute percentage change (Q6)
(change_in, in_eeissseell__eeeeeeeeeeeeeeeee asia, sales _018) 100
151. 57055789967183
Owen3-32B Reasoning
Original 2019 Asia sales in thousands
original_200i (aaalll201___aaa asia 646
Adjusted 2019 sales after doubling and adding 400 thousand
2ajusttd_2019__sst asia original_20n asia * 2 400
Original 2018 Asia sales in thousands
original, (aaigiaal_2i_______11111111 2018_000_______ aaia 1344
Calculate percentage change
percentage_cchage = badjussed_2019_astt
[iginaal2008_asiaa /
triginal_2018 2018 asia) * 100
ans percentage_ change
888888887742887466
42


Reviews:
[{"review_id": "R20MwVLUdl", "reviewer": "Reviewer_qRxF", "summary": "The manuscript proposes a retrieval-augmented generation technique for solving complex multi-hop question-answering problems using LLMs. In particular, the authors propose a self-play framework that purposes the same LLM as both a decomposer (to break down the question into subqueries) and a generator (to generate the answers to each subqueries).\n\nThe high-level idea of the framework is quite intuitive, and in fact borrows from a long line of work in the RAG space (i.e., the idea of decomposition of subqueries followed by generation of answers to those). The true contributions and strengths of the work, to me, lie in the following: (1) the proposed DPO-based algorithm, the way they generate the contrastive triplets,  and the fact that they are able to neatly form a single dataset comprising of both the decomposed subqueries and the generated answers; (2) impressive results on standard multi-hop benchmarks, as well as in settings like document-level reasoning, fact verification, etc. that are generally not demonstrated in existing multi-hop RAG works to my knowledge.", "questions": "Q1. What's the exact configuration you use for the two CORAG baselines (Greedy & Inference Scaling).\n\nSee strengths and weaknesses for other questions/concerns.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n1. Quality: The formulation and the solution strategy is well-motivated, with theoretical backing.\n\n2. Clarity: There are some details missing, but in general the paper is well-organized, ideas are conveyed clearly.\n\n3. Significance: The paper addresses an important problem, the space is quite crowded, but the manuscript manages to make a meaningful contribution.\n\n4. Originality: As I called out in the summary, even though the high-level formulation of decomposition followed by generation is not original, the work has good and original insights in the way they generate the contrastive triplets and formulate the DPO problem.\n\nI've a few minor concerns listed next, but overall, I find the contributions and the results in the paper to be significant.\n\nWeaknesses: \n\n1. Related Work is sparse and perhaps undersell the work in the paper: For instance, many of existing work on fine-tuning LLMs for RAG focus primarily on decomposition, and use a standard/off-the-shelf generator, unlike AceRAG that solves them jointly by repurposing the same LLM. \n\n2. I didn't find LeReT [26] compared in the experiments. LeReT has a similar strategy for training the decomposer, but uses a standard SLM for generator, but I recall it did show strong results in some of the RAG benchmarks. It would be good to contrast the results of AceRAG with LeReT.\n\n3. I was somewhat puzzled by the need for the first SFT phase (Sec 4.1) when I read it. My thinking was that if you are starting with models like Qwen instruct versions, SFT on standard datasets with next-token prediction might not be as valuable as the RL-based training in the next phase. But the ablations in Table 3 shows that SFT is so crucial! Could you comment on the overlap of datasets between SFT and RFT phases?", "was_revised": true, "final_justification": "I think this is a very good paper. However, it would be good to include a comprehensive comparison with LeReT in the final version. Otherwise, I am happy with the paper, and vote for acceptance.", "rebuttal": "Thank you for your thoughtful feedback and for carefully reviewing our work. Please find our responses to your suggestions below.\n\n\n***\n\n> W1: Related Work is sparse and perhaps undersell the work in the paper: For instance, many of existing work on fine-tuning LLMs for RAG focus primarily on decomposition, and use a standard/off-the-shelf generator, unlike AceRAG that solves them jointly by repurposing the same LLM.\n\nA: Thank you for the appreciation and helpful suggestion. While our initial intention was to provide a broader discussion of related work, we recognize that including too many references may distract readers from our main contribution. We agree that a clearer positioning of AceRAG in contrast to prior methods would help clarify the novelty of our joint training framework. We will revise the Related Work section to better highlight these distinctions and more effectively emphasize our contributions.\n\n***\n> W2: I didn't find LeReT [26] compared in the experiments. LeReT has a similar strategy for training the decomposer, but uses a standard SLM for generator, but I recall it did show strong results in some of the RAG benchmarks. It would be good to contrast the results of AceRAG with LeReT.\n\nA: Thank you for pointing this out. Following your suggestion, we provide a comparison between LeReT and AceRAG on HotpotQA and Hover below. We report EM scores and include only the 8B version of AceRAG for fair comparison with similarly sized LeReT models.\n\n| | HotpotQA | Hover |\n|:-:|:-:|:-:|\n| LeReT w/ Llama 8B | 52.5 | 69.8 |\n| LeReT w/ Gemma 9B | 54.3 | 71.5 |\n| AceRAG 8B | 58.8 | 68.3 |\n\nIt's important to highlight that LeReT leverages **direct supervision** by comparing retrieved documents against ground-truth retrievals, and uses a reader (Llama-3.1-70B) with much more parameters (8x). In contrast, AceRAG uses **indirect supervision** by deriving reward signals from final answer correctness. While direct supervision may yield stronger alignment with decomposition quality, it requires access to oracle retrieval labels, which limits scalability. We choose to use indirect supervision in this work to enable broader applicability across real-world settings. It is also worth noting that many existing works such as [1,2] also adopt the setting where only the final ground-truth answer is available.\nThat being said, the results show that AceRAG can outperform LeReT on HotpotQA and achieve comparable performance on Hover under this setting, further verifying the efficacy of our design. We will add the comparison and discussion of LeReT in the next version of the paper.\n\n\n***\n> W3: I was somewhat puzzled by the need for the first SFT phase (Sec 4.1) when I read it. My thinking was that if you are starting with models like Qwen instruct versions, SFT on standard datasets with next-token prediction might not be as valuable as the RL-based training in the next phase. But the ablations in Table 3 shows that SFT is so crucial! Could you comment on the overlap of datasets between SFT and RFT phases?\n\nA: Thanks for the insightful question! There is some overlap between the datasets used in SFT and RFT, especially for context-rich reasoning tasks. As described in Section 4.1, the **SFT phase** uses a curated mixture of context-rich QA data, question decomposition data, and chain-of-thought examples from a wide range of open-source datasets. This provides direct supervision to help the model learn how to answer questions accurately and generate effective decompositions. In contrast, the **RFT phase** fine-tunes the model primarily on QA pairs from HotpotQA, 2WikiMultiHopQA, HOVER and reasoning datasets such as GSM8K, ConvFinQA, StrategyQA (see Line 152 and 160), which teaches the model to refine its own response based on the final answer supervision. \n\nAlthough the base LLM is already instruction-tuned, the SFT phase is critical for explicitly teaching the model the specialized behaviors required by AceRAG – specifically, decomposition and multi-step reasoning grounded in retrieved context. General instruction tuning supports general language capabilities but does not provide sufficient supervision for RAG applications. By incorporating additional context-rich QA and question decomposition datasets in SFT, we ensure the model reliably acquires these targeted behaviors. This targeted supervision in SFT establishes a strong foundation for RFT to further refine model performance. We will incorporate this discussion into Section 5.4 to clarify the motivation behind our two-stage training design.\n\n***\n> Q1: What's the exact configuration you use for the two CORAG baselines (Greedy & Inference Scaling).\n\nA: Since the implementation and training datasets for CORAG are not publicly available, we report the numbers directly from their paper for comparison. CORAG is an iterative approach that generates sub-queries and answers step by step using tree-search algorithms.\n- **CORAG Greedy:** It uses greedy decoding to generate one sub-query and its corresponding sub-answer at each step, proceeding sequentially. This has similar computational complexity to AceRAG.\n- **CORAG Inference Scaling:** At each step, it applies breadth-first search (BFS) with multiple retrieval chains. It generates 10 sub-questions per step and uses best-of-N sampling (N=8) to select the final answer. This setting is significantly slower at inference, as noted in their paper.\n\n***\n# References \n\n[1] Jin et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" arXiv preprint arXiv:2503.09516 (2025).\n\n[2] Song et al. \"R1-searcher: Incentivizing the search capability in llms via reinforcement learning.\" arXiv preprint arXiv:2503.05592 (2025).\n\n***\n\nThank you for taking the time to review our work so carefully. Your feedback is invaluable, and we welcome any additional questions or comments you might have."}, {"review_id": "MEfabtrtKy", "reviewer": "Reviewer_8AMY", "summary": "This work proposes AceRAG, an LLM-driven RAG framework where a single LLM is trained to perform two roles: a decomposer that breaks down complex queries and a solver for answer generation. Following recent training receipt of thinking models, AceRAG learned from a two-stage training pipeline: Supervised Fine-Tuning (SFT) followed by Preference-based Reinforcement Fine-Tuning (RFT). The training process requires only final answer supervision, with no intermediate annotations needed. The technical design is well-justified in writing, and AceRAG demonstrates strong performance on standard multi-hop and complex reasoning RAG benchmarks.", "questions": "1. Since the \"decomposition-solve\" strategy is well calibrated for complex problems, I would like to confirm the scope of this work. Specifically, is AceRAG designed for multi-hop problems only, or is it a general RAG solution? If the latter, I would like to see the performance of AceRAG on a simple single-hop RAG benchmark to understand the impact of this training strategy on the model behavior.\n\n2. According to Line 142-144, different retriever is used for different tasks. Does this apply to inference time only or both training and inference time?\n\n3. In Figure 5b, the reranking baseline (blue line) is based on RankLLM. Given that this is an LLM-based reranking method, I am curious why it appears as a flat line, with no scaling in recall at all as model size increases. Could the authors provide more details about the experiment setup for Figure 5b, including model configurations?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The overall design is reasonable. A unified framework with two roles fits well with the nature of multi-hop complex QA.\n2. The rationale behind the technical choices is clearly justified in Sections 3, 4, and the appendix.\n3. The experimental evaluation is comprehensive, and AceRAG demonstrates strong performance across benchmarks.\n\nWeakness:\n\nThe contribution is more on the engineering side than in theoretical novelty. Most of the key components in the proposed framework can be directly traced to existing literature, including self-play [1, 2], RL-based RAG optimization [3, 4], \"decomposer-solver\" strategy [4], and two-stage training pipelines [5, etc.]. While AceRAG introduces its own implementation refinements, it shares a similar underlying idea with prior work. This limits its novelty from a conceptual standpoint. That being said, I do not consider that this detracts from the fundamental contribution of the work, but view it as a missing citation ([1], [3], and [4])\n\n*[1] Self-playing Adversarial Language Game Enhances LLM Reasoning. NeurIPS 2024*\n\n*[2] Self-play fine-tuning converts weak language models to strong language models. ICML 2024*\n\n*[3] Adaptive Information Seeking for Open-Domain Question Answering. EMNLP 2021*\n\n*[4] IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues. SIGIR 2024*\n\n*[5] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*", "was_revised": true, "final_justification": "The rebuttal confirms my original understanding. I maintain my positive recommendation.", "rebuttal": "We appreciate your valuable insights and the time you dedicated to evaluating our paper. Our responses to your comments are as follows.\n\n\n***\n\n> W1: The contribution is more on the engineering side than in theoretical novelty. Most of the key components in the proposed framework can be directly traced to existing literature, including self-play [1, 2], RL-based RAG optimization [3, 4], \"decomposer-solver\" strategy [4], and two-stage training pipelines [5, etc.]. While AceRAG introduces its own implementation refinements, it shares a similar underlying idea with prior work. This limits its novelty from a conceptual standpoint. That being said, I do not consider that this detracts from the fundamental contribution of the work, but view it as a missing citation ([1], [3], and [4])\n\nA: Thanks for providing those related works! We will add them to the references in the next version of the paper. \n\n***\n> Q1: Since the \"decomposition-solve\" strategy is well calibrated for complex problems, I would like to confirm the scope of this work. Specifically, is AceRAG designed for multi-hop problems only, or is it a general RAG solution? If the latter, I would like to see the performance of AceRAG on a simple single-hop RAG benchmark to understand the impact of this training strategy on the model behavior.\n\nA: Thanks for raising this question. AceRAG is mainly designed for multi-hop problems, as it benefits from the joint learning of the decomposer and question solver. For simple single-hop RAG problems, as they typically do not require such decomposition, standard single-round retrieval can already achieve very strong performance [1, 2].  As a result, we do not expect  AceRAG to offer significant advantages in this case. We will clarify the scope of AceRAG in the next version.\n\n\n\n***\n> Q2: According to Line 142-144, different retriever is used for different tasks. Does this apply to inference time only or both training and inference time?\n\nA: The use of different retrievers applies only at inference time. During training, we use E5 as the sole retriever across all tasks. At inference time, E5 remains the default retriever, with the only exception being the document-level reasoning task, where we follow the original benchmark setup [3] and use OpenAI’s Embedding-3-Large to ensure a fair comparison.\n\n***\n> Q3: In Figure 5b, the reranking baseline (blue line) is based on RankLLM. Given that this is an LLM-based reranking method, I am curious why it appears as a flat line, with no scaling in recall at all as model size increases. Could the authors provide more details about the experiment setup for Figure 5b, including model configurations?\n\nA: Thank you for pointing this out. The RankLLM baseline is based on a **fixed 7B model** with no varying sizes. As such, it yields a single performance point rather than a curve. We represented it as a flat dashed line to facilitate visual comparison with AceRAG’s scaling behavior. To avoid confusion, we will revise the figure in the next version by adding a marker at the 7B position to explicitly indicate the model size.\n\n***\n# References\n[1] Wei et al. \"Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales.\" ICLR 2025.\n\n[2] Yu et al. \"Rankrag: Unifying context ranking with retrieval-augmented generation in llms.\" NeurIPS 2024.\n\n[3] Zhao et al. \"DocMath-eval: Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents.\" ACL 2024.\n\n\n***\n\nThank you again for your helpful review. We truly appreciate your feedback and are glad to address any further questions or engage in additional discussion."}, {"review_id": "xjFNmrpEB7", "reviewer": "Reviewer_JZzw", "summary": "AceRAG is a method for multi-step, reasoning-heavy retrieval augmented generation (RAG) using two roles: a decomposer and a solver.\nThe decomposer role breaks down the original question into subquestions and also takes into account answers to previous subquestions.\nThe solver generates intermediate answers, and the final answer based on previous intermediate answers and retrieved passages.\nThis methodology is not just prompting based but involves a two stage finetuning procedure. The first supervised fine tuning stage finetunes the LLM on a mix of retrieval-oriented QA, decomposition, and chain-of-thought tasks. The second stage uses reinforcement learning to optimize the multi-step retrieval and reasoning policy directly on downstream task success. Across ten multi-hop QA, fact-verification, and document-reasoning datasets, AceRAG-32B improves exact-match by 7.6% on average over strong open-source and RL-enhanced baselines, matching DeepSeek-V3 while using <5% of its parameters. The smaller 1.5B, 8B, 14B variants beat larger models.", "questions": "1. What is the justification only exploring fairly outdated retriever models such as E5, Dragon and Contriever (Appendix H)?\n2. What is the justification for comparing to Qwen3-32B Reasoning when it does not have access to retrieved contexts.", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The RL formulation of directly optimizing the policy of two roles based on final task success is novel in the domain of multi-step RAG.\n2. Very strong experimental results, validating the methodology.\n3. Very thorough experiments: 10 datasets, cross-scale models, ablations on SFT data mix, RL variants, and hyper-parameters. Human studies back quantitative gains. Training/inference compute is reported.\n\nWeaknesses:\n1. Some unclear experimental choices (see Questions).", "was_revised": true, "final_justification": "I appreciate the authors' clarifications. I have raised the clarity score to 4. The overall score of 5 continues to represent the strength of this work.", "rebuttal": "Thank you for your detailed review and constructive suggestions. We address your feedback point by point below.\n\n\n***\n\n> Q1: What is the justification only exploring fairly outdated retriever models such as E5, Dragon and Contriever (Appendix H)?\n\nA: Thank you for raising this point. We selected these retrievers due to their widespread use in prior works, such as Search-R1 [1], InstructRAG [2], CORAG [3], Plan-RAG [4], etc. This choice ensures fair and meaningful comparisons with existing methods. We agree that evaluating with more recent retrievers (e.g., Qwen3-embedding [5]) could further improve performance, and we will consider this in the future work.\n\n***\n> Q2: What is the justification for comparing to Qwen3-32B Reasoning when it does not have access to retrieved contexts.\n\nA: Thanks for the question. We would like to clarify that all the baselines, including Qwen3-32B reasoning model, **have access** to retrieved contexts by default. We also keep the same retriever and retrieval corpora as our method during evaluation to ensure fair comparison.\n\n***\n# References\n\n[1] Jin et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" arXiv preprint arXiv:2503.09516 (2025).\n\n[2] Wei et al. \"Instructrag: Instructing retrieval-augmented generation via self-synthesized rationales.\" ICLR 2025.\n\n[3] Wang et al. Chain-of-Retrieval Augmented Generation. arxiv preprint 2501.14342.\n\n[4] Verma et al. \"Plan-rag: Efficient test-time planning for retrieval augmented generation.\" arXiv preprint arXiv:2410.20753 (2024).\n\n[5] Zhang et al. \"Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models.\" arXiv preprint arXiv:2506.05176 (2025).\n\n***\nThank you once again for your insightful review. We appreciate your feedback on our work. Feel free to let us know if you have any further questions, and we are happy to discuss further."}, {"review_id": "2JcBx8aPWQ", "reviewer": "Reviewer_GkXC", "summary": "This paper introduces AceRAG, a cooperative self-play framework for reasoning-intensive retrieval-augmented generation (RAG). The approach trains a single large language model to perform both question decomposition and answer generation through supervised fine-tuning and reinforcement learning optimized on final answer correctness. The authors demonstrate performance gains across ten benchmarks and highlight improved parameter efficiency compared to recent baselines.", "questions": "Please see Strength and Weaknesses.", "limitations": "Please see Strength and Weaknesses.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. This paper is well-organized and clearly written.\n2. The idea of unifying decomposition and solving roles in one model is conceptually appealing.\n3. The evaluation covers a diverse set of benchmarks, with thorough ablation studies and parameter analyses.\n4. AceRAG achieves strong results while remaining lightweight, which is important for the open-source community and low-resource settings.\n\n\n\n**Weaknesses**\n1. The training pipeline (SFT + RFT) is somewhat intricate, potentially limiting reproducibility despite good documentation.\n2. The assumption that reward signals from final answers sufficiently guide decomposition quality could be risky in more ambiguous domains.\n3. The decomposition and multi-step retrieval introduce additional inference overhead compared to the standard RAG pipeline, which could limit applicability in real-time scenarios.", "was_revised": true, "final_justification": "Thanks for the responses. I tend to keep my original score.", "rebuttal": "We are grateful for your careful assessment and insightful comments on our paper. Our responses to your suggestions are provided below.\n\n***\n> W1: The training pipeline (SFT + RFT) is somewhat intricate, potentially limiting reproducibility despite good documentation.\n\nA: Thank you for the valuable feedback. We would like to mention that it is a de facto standard to do SFT + RFT for post-training LLMs. We also want to clarify that our training pipeline is built on top of the Llama-Factory codebase, which **provides streamlined support for both SFT and RFT**, making it easy to use and reproduce. To further ensure accessibility, we will **open-source our code and data** along with detailed documentation and ready-to-use scripts. Users will be able to reproduce our experiments or fine-tune models on their own datasets with minimal effort.\n\n***\n> W2: The assumption that reward signals from final answers sufficiently guide decomposition quality could be risky in more ambiguous domains.\n\nA: Thank you for the insightful comment. The tasks considered in our work primarily involve questions with short-form, verifiable answers, where reward signals from final answers provide reliable and sufficient supervision. In more ambiguous domains involving long-form or non-verifiable answers, we could adopt an LLM-as-a-judge approach to generate reward signals. However, since such methods are less scalable and unnecessary for our tasks, we do not employ them in this work.\n\n***\n> W3: The decomposition and multi-step retrieval introduce additional inference overhead compared to the standard RAG pipeline, which could limit applicability in real-time scenarios.\n\nA: Thank you for the question. We would like to point out that most strong baselines in our comparison also adopt multi-step retrieval, and AceRAG does not introduce additional inference overhead beyond those methods. While AceRAG does have higher latency than standard single-step RAG due to its decomposition and multi-step reasoning components, this overhead is justified by the performance gains. As shown in **Section 5.5 and Figure3(c)**, AceRAG achieves substantial performance gains – even outperforming 32B models with comparable inference time. We believe this trade-off is a reasonable compromise for the gains in effectiveness.\n\n***\nThank you for your insights. We appreciate your feedback and are happy to address any further questions you may have."}, {"review_id": "trGZ7B7iAT", "reviewer": "Reviewer_Z9aX", "summary": "The authors propose AceRAG, a cooperative self-play framework. AceRAG trains a single LLM to assume two roles: a decomposer, which breaks down complex questions into simpler subquestions to guide information retrieval, and a solver, which integrates retrieved contexts and intermediate answers to produce final responses.\n\nThe training approach includes a two-stage fine-tuning method:\n1. Supervised Fine-Tuning (SFT) using an extended mix of retrieval, reasoning, and decomposition datasets to build general reasoning and retrieval capabilities.\n2. Reinforcement Fine-Tuning optimizing performance through iterative preference optimization, where the solver’s accuracy provides reinforcement signals, guiding the decomposer toward effective decompositions without explicit intermediate annotations. \n\nThe authors extensively evaluate AceRAG across three tasks and ten datasets and demonstrate significant performance improvements (average gain of 7.6%) compared to strong baselines.", "questions": "- Given that both roles are trained jointly via preference optimization, did you observe any instability where improvements in one degrade the other? Did you try strategies other than randomly shuffle the preference pairs for different components, e.g. first train the solver given the correct context, and then the decomposer? \n\n- Have you tried other reward functions other than the proposed one?  \n\n- How diverse are the decompositions sampled by the model during reinforcement fine-tuning, and how sensitive is performance to decomposition redundancy? Do you apply any techniques to encourage diversity in z during sampling?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths \n- A key strength of this work lies in how it formulates reasoning-intensive RAG as a cooperative interaction between two roles: a decomposer and a solver. Also, the paper grounds this formulation in a principled way using the DPO framework, the mathematical proof is solid. \n- The authors follow a two-stage training strategy where RFT follows SFT. In the RL stage, they utilize a design of the preference-based reinforcement learning setup, particularly the way preference pairs are constructed separately for the decomposer and solver components. This formulation enables joint optimization of both roles using outcome-based feedback. \n-  The authors conduct extensive experiments using models of various sizes (1.5B to 32B), demonstrating that the proposed method scales well.  Moreover, they compare against a diverse set of baselines, including standard RAG systems, reasoning-enhanced QA models, and recent retrieval+RL-based approaches. This provides a well-rounded empirical picture and shows that the gains of AceRAG are not limited to a narrow benchmark. In addition, the authors show strong awareness of the importance of data diversity. Their SFT stage incorporates a rich mixture of datasets covering QA with context, question decomposition, and chain-of-thought reasoning, which strengthens the model’s robustness across a wide range of question types. \n\n## Weaknesses \n- The design of the reward function may be too rigid, which could limit its generalization or learning efficiency. The reward is defined as the product of two binary indicators: an exact match (EM) between the generated and ground-truth final answers, and a format-based check ensuring the presence of intermediate reasoning steps. This sparse signal could make policy optimization challenging. An ablation or justification of this reward choice would help strengthen the argument.", "was_revised": true, "final_justification": "I think this is good paper backed up by sufficient experimental results. \nMy questions and concerns are addressed in the authors' rebuttal so I will keep my original score.", "rebuttal": "Thank you for your thorough review and helpful remarks. Below, we address each of your suggestions in detail.\n\n***\n> W1: The design of the reward function may be too rigid, which could limit its generalization or learning efficiency. The reward is defined as the product of two binary indicators: an exact match (EM) between the generated and ground-truth final answers, and a format-based check ensuring the presence of intermediate reasoning steps. This sparse signal could make policy optimization challenging. An ablation or justification of this reward choice would help strengthen the argument.\n>\n> Q2: Have you tried other reward functions other than the proposed one?\n\nA: Thank you for raising this important point. To assess this, we experimented with both EM and F1 score as reward signals across multiple QA benchmarks.\n\n|           | 2WikiMHQA | HotpotQA | Bamboogle | MusiQue | Hover | ExFever |\n|-----------|-----------|----------|-----------|---------|-------|---------|\n| EM Reward | 66.0      | 58.8     | 55.2      | 35.4    | 68.3  | 73.8    |\n| F1 Reward | 65.0      | 57.0     | 53.6      | 36.6    | 66.6  | 71.4    |\n\n|           | DMSimpShort | DMCompShort | DMSimpLong | DMCompLong |\n|-----------|-------------|-------------|------------|------------|\n| EM Reward | 83.0        | 80.5        | 48.0       | 32.3       |\n| F1 Reward | 81.5        | 79.5        | 47.0       | 33.3       |\n\n The result demonstrates that the simple EM reward yields very robust performance, while performance drops a little when using F1 as the reward.\n\n***\n> Q1: Given that both roles are trained jointly via preference optimization, did you observe any instability where improvements in one degrade the other? Did you try strategies other than randomly shuffle the preference pairs for different components, e.g. first train the solver given the correct context, and then the decomposer?\n\nA: Thank you for the question. As shown in Figure 2, with reasonable hyperparameter settings (e.g., lr=5e-7, 5% warmup for Llama-8b), we observed stable average performance across QA and reasoning datasets, without significant fluctuations. We also tried sequentially training the solver and decomposer in separate stages, but encountered forgetting, where previously learned skills were lost in the second stage. In contrast, our joint training strategy using shuffled preference pairs enables both components to learn more effectively and maintain stability.\n\n***\n> Q3: How diverse are the decompositions sampled by the model during reinforcement fine-tuning, and how sensitive is performance to decomposition redundancy? Do you apply any techniques to encourage diversity in z during sampling?\n\nA: Thank you for the thoughtful question. To encourage diverse decompositions during reinforcement fine-tuning, we set the sampling temperature to $t=1.0$, which promotes variability in the generated decompositions. In practice, we observe that most sampled decompositions differ meaningfully from one another. Additionally, to reduce redundancy and ensure training efficiency, we explicitly filter out duplicate or highly similar decompositions before training. This helps maintain a diverse set of training signals and supports more robust learning.\n\n***\nThank you once again for your insightful review. We appreciate your feedback on our work. Feel free to let us know if you have any further questions, and we are happy to discuss further."}]

Author rebuttal:

Meta-review: This paper presents AceRAG, a self-play framework for complex multi-hop question answering. Its core contribution lies in alternating between the roles of decomposer and solver, leveraging both supervised fine-tuning and reinforcement learning. All reviewers acknowledged the work’s substantial novelty and technical depth, with four reviewers awarding it the high score of 5 and one reviewer assigning a score of 4. Moreover, the authors effectively addressed most of the reviewers’ concerns during the rebuttal.
Decision: Accept (spotlight)