Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration
Authors: Ran Xu
Wenqi Shi
Yuchen Zhuang
Yue Yu
Joyce C. Ho
Haoyu Wang
Carl Yang
Venue: COLM 2025 (2025)
Keywords: large language model
retrieval augmented generation
complex question answering
TL;DR: We introduce Collab-RAG, a collaborative training framework that leverages mutual enhancement between a white-box small language model (SLM) and a black-box large language model (LLM) for RAG.

Abstract: Retrieval-Augmented Generation (RAG) systems often struggle to handle multi-hop question-answering tasks accurately due to irrelevant context retrieval and limited complex reasoning capabilities. We introduce Collab-RAG, a collaborative training framework that leverages mutual enhancement between a white-box small language model (SLM) and a black-box large language model (LLM) for RAG. Specifically, the SLM decomposes complex queries into simpler sub-questions, thus enhancing the accuracy of the retrieval and facilitating more effective reasoning by the black-box LLM. Concurrently, the black-box LLM provides feedback signals to improve the SLM's decomposition capability. We observe that Collab-RAG relies solely on supervision from an affordable black-box LLM without additional distillation from frontier LLMs, yet demonstrates strong generalization across multiple black-box LLMs. Experimental evaluations across five multi-hop QA datasets demonstrate that Collab-RAG substantially outperforms existing black-box-only and SLM fine-tuning baselines by 1.8%-14.2% on average. In particular, our fine-tuned 3B SLM surpasses a frozen 32B LLM in question decomposition, highlighting the efficiency of Collab-RAG in improving reasoning and retrieval for complex questions. Our implementation is available at \url{https://github.com/ritaranx/Collab-RAG/}

Published as a conference paper at COLM 2025
Collab-RAG: Boosting Retrieval-Augmenter Generation for Complex Question Answering via White-Box and Black-Box LLM Collaboration
Ran Xu 15 Wenqi Shi?, Yuchen Zhuang', Yue Yu Joyce C. Ho', Haoyu Wang', Carl Yang'
Department of Computer Science, Emory University
2 University of Texas Southwestemn Medical Center
3 College of Computing, Georgia Institute of Technology
4 Departiment of Computer Science, SUNY Albany [ran. xu, j. carl y @emorr. edd
Abstract
Retrieval-Augmeneed Generation (RAG) systems often struggle to han- dle multi-hop question-answert tasks accurately due to irrelevant con- text retrieval and limited complex reasoning capabilities. We introduce Collab-RAG, collaborative training framework that leverages mutual en- hancement between white-box small language model (SLM) and a black- box large language model (LLM) for RAG. Specifically, the SLM decom- poses complex queries into simpler sub-questions, thus enhancing the accu- racy of the retrieval and facilitating more effective reasoning by the black- box LLM. Concurrently, the black-box LLM provides feedback signals to improve the SLM's decomposition capability. We observe that Col lab-RAG relies solely on supervision from an affordable black-box LLM without ad- ditional distillation from frontier LLMs, yet demonstrates strong generaliza- tion across multiple black-box LLMs. Experimental evaluations across five multi-hop QA datasets demonstrate that Collab-RAG substantially outper- forms existing black-box-only and SLM fine-tuning baselines by 1.8%-14.2%% on average. In particular, our fine-tuned 3B SLM surpasses frozen 32B LLM in question decomposition, highlighting the efficiency of Collab-RAG in improving reasoning and retrieval for complex questions. Our imple- mentation is available at https: / //g thub. carr//iaarann//llllaa---aa
1 Introduction
Despite the strong performance of Large Language Models (LLMs) across a wide range of language tasks, they face several limitations such as hallucinations (Shi et al., 2024a), and difficulties adapting to evolving or domain-specific knowledge (Zhang et al., 2024b). edaptin Retrieval-Augmented Generation (RAG) has emerged as a powerful technique to address these challenges by integrating external knowledge sources, enabling LLMs to improve fac- tual accuracy (Lewis al., 2020) and response reliability (Lin et al., 2024) to their responses.
In a standard RAG pipeline, retriever first searches from external corpora for relevant information based on a given query. The retrieved context is then combined with the query and fed into the LLM, allowing it to generate more contextually grounded response. Although stronger black-box LLMs have shown strong capability in language modeling and have been used RAG readers in practice (Shi et al., 2024a; Jeong et al., 2024), the standard RAG pipeline often has low-recall issues, where relevant information cannot be retrieved (Yu et al., 2024a). This issue particularly pronounced in complex question-answering (QA) tasks, where multiple pieces of evidence are required for accurate reasoning.
To enhance the ability of black-box LLMs for handling complex questions in RAG appli- cations, several studies have attempted to improve retrieval quality, such as training an
FFqual Contribution.

---

Published as a conference paper at COLM 2025
auxiliary model for text embedding (Shi et al., 2024a) or query re-ranking (Mao et al., 2024). However, these approaches primarily focus on refining single-step retrieval and fail to address the inherent challenges of complex question-answering scenarios, where iterative evidence gathering is essential for assembling the most relevant information from the corpus. Moreover, Jiang et al. (2023); Li et al. (2025); Liu et al. (2024a); Wang et al. (2024) propose training-free free methods to leverage LLM itself to refine queries or perform multi-turn retrieval. However, without dedicated training, LLMs have limited capabilities in effective query decomposition and refinement, making these methods suboptimal for complex retrieval tasks. Recently, some studies finetune small language models to improve RAG performance (Liu et al., 2024b; 2025; Wei et al., 2025), however, updating parameters for black-box LLMs remains inefficient and pesourc-intensive, limiting the practicality of these methods in real-world applications. To summarize, it is still crucial yet challenging to fully unleash the capability of black-box LLMs for complex question answering tasks.
In this work, we introduce Collab-RAG, an RAG framework that enhances black-box LLMs for complex question answering by incorporating an addi- tional white-box small language model (SLM). As shown in Figure 1, the SLM serves as a decomposer, breaking down complex queries into smaller, atomic subquestions to improve the retrieval of relevant contexts (Patel et al., 2022; Khot et al., 2023). The black-box LLM then acts as a reader, generating in- termediate answers for each subques- tion and synthesizing them into fi- nal response. By structuring retrieval around decomposed subquestions, this framework effectively harnesses the context extraction capabilities of black- box LLMs to answer the complex ques- tions step-by-step.
(a) Standard RAG
Query Retriever
Black-Box
LLM
(b) IRCOT
Query
Black-Box Box
Black-Box
LUM
Sub-Queries
(c) Collab-RAG
Query
White-Box
Black-Box
SLM
Sub-Queries
LLM
Suparvision
Figure 1: Comparison of various LLM-based RAG pipelines. Collab-RAG fosters collaboration be- tween the SLM query decomposer and the LLM reader, allowing them to enhance each other.
Using SLMs directly for question decomposition is often ineffective, as they are not ex- tensively trained on relevant data, and collecting large-scale, high-quality annotations is expensive. тo overcome this challenge, we propose a self-improoing training strategy that depends solely on feedback from black-box LLMss During training, we model the black- box LLM (GPT-40-mini) as an environment that generates responses alongside queries and retrieved contexts. The SLM engages in multi-turn interactions with this environment to iteratively refine its decomposition strategy. Our key insight is that higher-quality question decomposition leads to better final answers. Based on this, we design an iterative preference optimization approach (Pang et al., 2024) that uses feedback from the black-box LLM to improve the SLM's decomposition capabilities. Specifically, inspired by (Guo et al., 2025; Jin et 2025), we adopt a rule-based evaluation to distinguish between effective and ineffective decompositions by assessing both the format of the subquestions and the accuracy of the final answer. This preference optimization framework enables the SLM learn optimal decomposition strategies that enhance retrieval and reasoning ability for the RAG pipeline, without relying on costly human annotations or distillation from frontier-class LLMs.
Our contributions can be summarized as follows: (i) Problem Wise, we propose Collab-RA RAG to enable dynamic collaboration between white-box SLMs and black-box LLMs, which have not yet been widely explored in existing RAG studies for complex question answering. (ii) Methodology Wise, Collab Rallab-RAG employs an iterative preference optimization approach using outcome feedback from black-box LLMs without relying on distillation from frontier LLMss Collab-RAG is trained using feedback from GPT-40-mini only but shows strong generaliza- tion across various LLMs. (iii) Experimental Wise, follab-RAG outperforms both black-box LLM-only approaches and small LLM tine-tuning baselines by 1.8%- 14.2%, demonstrating improved reasoning and retrieval effectiveness in complex question-answering tasks. No-
2

---

Published as a conference paper at COLM 2025
tably, for question decomposition, fine-tuned 3B SLM achieves better results than frozen 32B LLM on average, justifying the efficiency and effectiveness of Collab-RAs
2 Related Work
Ketrieval-Augmented Generation. RAG enhances LLMs by integrating external knowledge retrieval, improving the accuracy and relevance of generated responses. Earlier works study improving refrievers for RAG, as Shi et al. (2024a); Shao et al. (2023) finetune retrievers based on language model feedbacks, Then, with more available open-source LLMs, several works also design effective instruction finetuning pipelines towards RAG applications via collecting diverse training data (Lin et al., 2024; Liu et al., 2024b; Yu et al., 2024b), generating synthetic data (Xu et al., 2024; Zhu et al., 2024; Shi et al., 2024b), or incorporating chain-of- thought reasoning process (Yu et al., 2024a; Wei et al., 2025). More recently, reinforcement learning (RL) techniques have been employed to optimize retrieval relevance (Dong et al., 2024) and enhance the quality of chain-of- oftouhhttthtttttt reasoning (Liu et al., 2025; Zhang et al., 2024a). RAG-Gym (Xiong et al., 2025) and RAG-Star (Jiang et al., 2024) employ reward models to guide LLM generation, but they rely on supervision from GPT-4 series models, which introduces additional supervision costs. Different from these works, we leverage preference-bate fine-tuning method to better decompose the complex questions based on the feedback of final answer, which alleviates the need for intermediate passage relevance supervision and can serve as a generic plug-in for LLMs.
Query Optimization. To improve end-to-end performance of RAG pipelines, several query optimization techniques have been proposed. Several studies have explored query rewriting techniques to enhance response quality (Ma al., 2023; Mao et al., 2024), particularly in conversational question-answering systems where rewriting helps better capture user intent (Mo et al., 2023). More related to our work, some approaches focus on decomposing complex queries to facilitate step-by-step reasoning. Prompt-basee decomposition methods have been proposed for reasoning tasks (Khot et al., 2023; Khattab et al., 2022) and have been extend to RAG scenarios (Verma al., 2024; Liu et al., 2024a), while some other approaches refine queries progressively using retrieved contexts (Yu et al., 2023; Li et al., 2025). Additionally, recent methods leverage knowledge distillation from proprietary models to learn query decomposition strategies (Chan et al., 2024).
3 Preliminaries
Retrieval-Augmentt Complex Question Answering. Standard open-domain QA typically relies on single-step retrieval to locate relevant information. In contrast, complex QA (also referred as multi-hop QA), requires extracting and integrating multiple pieces of information while performing multi-step reasoning (Ho et al., 2020; Yang et al., 2018; Trivedi et al., 2022). In this work, we focus on enhancing RAG with LLMs for complex QA tasks, specifically targeting complex QA datasets. We do not study single-step QA in this work.
Problem Formulation. We consider a complex QA task that requires multi-step reasoning and the retrieval of external knowledge to derive solutions. Specifically, given a ques- tion X; E X' that necessitates reasoning across multiple documents, the objective is to generate comprehensive solution composed of (i) a decomposition of original question to multiple simpler sub-questions Q, = {9:11 f..2..., JII, with intermediate solutions ; Vi2. ,,,,, and (ii) final answer y; E )) where T, E IN+ denotes the number of reasoning steps, Each sub-question qi, is used to iteratively retrieve relevant information from a corpus C via a retriever Ty (-) that progressively leads to the final answer.
4 Methodology
In this section, we introduce Collab-RAG, an advanced RAG framework designed for com- plex QA by leveraging collaboration between a white-box SLM and a black-box LLM. Denote Q* as the set of finite sequences for subquestions and ) the set of possible solutions, our framework Collab-RAG consists of two parts: (i) a white-box small language model fo()), to serve question decomposer. The decomposition function fo X - Q maps the
3

---

Published as a conference paper at COLM 2025
Query Decomposer
Context Reader
Preference Iterative DPO Pairs
Qi
(9 912 " 4tr
X;
White-Box (LM
Q:
9ca Retriever
Dic Black- Box LLM Sg
DPO white-Bon Box
(m))
D (m++1)
riv
SLM 8 ((
t-th Iteration
PLTTN
Next Iteration Model
The iterative training framework of Collab-RAG. The SLM updates its parameters based on the generation quality of the LLM reader. The above process is conducted over multiple iterations to gradually improve SLM's decomposition capability.
Figure 2:
question to a variable-length sequence of retrieval queries. (ii) a black-box large language model 86( (-)::: Q x C - )d to serve as the reader for answering the intermediate questions and deriving the final answer with retrieved contexts. Further details are provided below.
4.1 Overview of Collab-RAG
In our framework, the SLM is employed as query decomposer to transform complex queries into simpler, manageable step-by-step subquestions (Section 4.3). Subsequently, the decomposed subquestions guide the LLM-based RAG system sequentially retrieving and synthesizing relevant information. We view the RAG system as an environment (Section 4.2) and construct a diverse and extensive training dataset through interactions between the white-box SLM decomposer and the black-box LLM reader (Section 4.4). Since updating black-box LLM is costly, we refine the system by updating the parameters of the SLM fa(-). This is achieved through iterative preference optimization, leveraging feedback from the LLM (Section 4.5). The framework of Collab-RAG is illustrated Figure 2.
4.2 RAG Environment
We formulate the LLM-based RAG system as an environment that interacts with the LLM question decomposer. At each step f E {1,2, Ti}, retriever ry takes the current question fi, and searches over a set of documents D;, = To (9))) that are most relevant to qip- Next, generator gø uses the current question 4i,t, the retrieved knowledge Di,, and all previous responses [1:x) 1/-1 to produce response Ui,+ as:
Vi,- = 8o (9i,, D 1.
(1)
After the T; steps, the RAG system outputs the final solution Ú.T, The quality of the question decomposition is evaluated using a reward function u X' x Q" + defined as combination of format and accuracy rewards (Guo al., 2025):
uxx;, fx())) eval(x,, (x, if format(x;, (1,1) 1;
(2)
otherwise.
Here, the format reward (format(x;, Pin;; mainly check whether the model can out- put the decomposed question with correct reference of answers from previous round of answers and the accuracy reward (eval(.)) evaluates whether the response is cor- rect, where y; is the ground-truth answer. In our study, Match (EM) and Accuracy (Acc)² as the final accuracy reward, defined eval(2, Vi,,, 0.5 (EM(Íi, = y:) + Acc yi)). A higher reward indicates a more effective decom- position (x) in guiding the RAG system to the correct answer.
combination of Exact
We enforce the model to output correct reference to previous round answers such as 'Whaf is the the magazine mentioned in #1?', where #1 is the answer for subquestion 1. However, sometimes model will still output 'What the name of the magazine mentioned Question I', which yields suboptimal retrieval results.
2 AAcuuracy only requires the ground truth answer exist the response, which relaxed from EMM.
4

---

Published as a conference paper at COLM 2025
4.3 White-Box SLM as Query Decomposer
Complex questions frequently include multiple interdependent sub-questions, which stan- dard retrieval systems often struggle to resolve effectively. To address this, we use lightweight, white-box SLM specifically as a query decomposer. Given complex query X the decomposer generates structured sequence of simpler sub-questions:
fo (x;) = {91,1 (12...4
(3)
where each sub-question qi, clearly specifies the retrieval target, enhancing retrieval rele- vance and efficiency.
4.4 Black-Box LLM 8 as Context Reader
Despite their effectiveness in text generation tasks, LLMs still struggle to systematically decompose and interpret complex questions in RAG systems due to: (1) limited multi-step reasoning for effective decomposition in smaller models, (2) suboptimal decompositions from zero-shot scenarios, and (3) difficulties integrating evidence dynamically across rea- soning steps. To address these limitations, we formulate the RAG system as an interactive environment, utilizing feedback from the black-box LLM context reader as the supervision signal. Specifically, we regard decomposition as positive if the black-box LLM can accu- rately interpret retrieved information and produce correct answers based on the decomposed queries, thereby improving the overall capability of the RAG system. For each input X;, we
initially sample sub-questions using the LLM-based decomposer: Qi = |- ~ fx().) We then select the best-of-N decomposition based on the highest reward u(x;, fo (x;)) as the positive example, and the worst-of-N decomposition as the negative example?:
Q++ = Qij. j = arg max u(x; Qin), 1SHSN
(4)
P Qip, j= =- arg min u(x;, Qin)-
1SHSN
To prevent overfitting and encourage generalization, we follow reinforcement learning with human feedback (RLHF) principles (Ouyang et al., 2022) and ensure each positive and negative pair shares the same input prompt, thus constructing a balanced training dataset DiDro
(x Qi }
4.5 Iterative Preference Optimization via LLM Feedback
Directly employing SLMs for question decomposition often falls short due their limited reasoning capabilities, while collecting extensive, high-quality annotations is resource- intensive. To overcome these limitations, we leverage feedback from black-box LLMs to curate training data for SLMs (Choi et al., 2024). It starts with supervised fine-tuning (SFT) based on rejection sampling and subsequently improves through iterative preference optimization guided by black-box LLM feedback.
Warmup:
Supervised Fine-tuning with Rejection Sampling. Smaller language models often struggle with reinforcement learning due to their limited capabilities, leading to unstable training and poor convergence (Zheng et al., 2023). To address this challenge, first fine-tune the model on self-generated high-quality data: For each question x;, employ rejection sampling (Zelikman et al., 2022) to generate candidate decompositions: {Qr1, Qi...... } fx;)x Then, only prompts with decomposition having the highes! reward will be included in the SFT dataset D = {(x, Qi) u (x, Qi) 2 0.5). The SLM is
then fine-tuned using next-token prediction loss, conditioned on the input question:
LSFT - -E (x,,0))
log fo 11( (0
(5)
9The prompt will be discarded if all decompositions lead to the same reward (e.g. all of them result in correct/ incorrect answers).

---

Published as a conference paper at COLM 2025
where Qill] is the I-th token in the generated decomposition, Through SFT, the white-box SLM acquires the essential query decomposition skills needed for subsequent optimization.
Iterative DPO. Directly using decompositions generated by SLMs can lead to overfit- ting due to imbalances between positive and negative examples, resulting in simplistic patterns and limited model improvements. To mitigate this, we propose iterative opti- mization framework that interleaves data collection and model training. Initially, we set the model parameters to g(o) = f after the warm-up phase and collect an initial dataset D(O), At iteration m, we optimize model parameters gim)) and generate fresh samples (0%0) (m) (m) ... 2 faim (x). Following Section 4.4, we then construct preference pairs, forming the training dataset D ((m)
iDPO = (x, (m) .. To update the model for the next iteration plm+1), we employ the direct preference optimization (DPO) for parameter optimization, using the model from the previous iteration as the reference. Consequently,
the training objective for iterative DPO of the SLM at m-th iteration is formulated as:
T LIDO (0(m-1))) = -I (=.t+--))D(aa loggo (Blog
(0-l))
fm+1)) (Q-1x)
] (O [x)
BBlog
" (Q (x) 28
(6)
Experiments
5.1 Experiment Setups
Evaluation Datasets. We use five representative multi-hop QA datasets: (1) Hot- potQA (Yang et al., 2018), (2) 2WikiMQA (Ho et al., 2020), (3) MusiQue (Trivedi et al., 2022), (4) StrategyQA (Geva et al., 2021), and (5) Bamboogle (Press et al., 2023). We conduct evaluations on all questions from StrategyQA and Bamboogle, and the first 500 questions from the development sets of the other datasets following existing studies (Trivedi et al., 2023; Shao et al., 2023; Wang et al., 2024). For Bamboogle, we used the Wikipedia dump from December 2018 as the corpus, while for the other datasets, we use the corpora provided by their respective original sources. Detailed descriptions are in Appendix A.
Training Datasets. We sample 10000 (question, answer) pairs from the training set of HotpotQA, MusiQue, and 2WikiMQA as the training data".
Evaluation Metrics. Different studies often use various metrics evaluate the performance of RAG models. To ensure comprehensive evaluation, we consider Exact Match (EM), Accuracy and F1 Score jointly as the metric, while EM is used as the main metric.
Baselines. We consider the following baselines for comparison: (1) White-box LLMs with RAG: where we consider most recent RAG models based on open-source LLMs including DRAGIN (Su et al., 2024), GenGround (Shi et 2024b), ChatQA
et al, 2024b), RankRAG (Yu al., 2024b), and Retrieval-augmented Finetuning (RAFT) (Zhang et al., 2024b; Lin et al., 2024). (2) Black-box LLMs with RAG: where we consider IRCOT (Trivedi et al,, 2023), FLARE (Jiang et al., 2023), RA-ISF (Liu et al., 2024a), BlendFilter (Wang et al., 2024), Search-ol (Li et al., 2025), and IterDRAG (Yue et al., 2025). (3) Additional Baselines: we also consider baselines including Chain-of-Thougee RAG (Lewis et al., 2020), RAG with question decomposition (Khot et al., 2023), RAG with reranking, RAFE (Mao et al., 2024), Iter-RetGen (Shao et al., 2023), RQ-RAG (Chan et al., 2024), RAG-Star (Jiang et al., 2024), and one recent baseline RAG-Gym (Xiong al., 2025). The details of baselines are in Appendix B.
(COT, Wei et al. (2022)), vanilla
5.2 Implementation Details
Backbones. For both white-box SLMs and black-box LLMs, we consider different variants to test the generalization of Collab-RAs Specifically, we consider Qwen-2, 2.-......BBBB-- 1-3B-Instruee (Yang
It is worth noting that we do not use the intermediate reasoning chain, or gold passages provided by the original datasets during our training and evaluation to ensure fair comparison.

---

Published as a conference paper at COLM 2025
Baselines
StrategyQA EM
HotpotQA
MusiQue
2W1KiMQA
Bambougle
Acc
EM
F1
Ace
RM
F1
Acc
EM
F1
Acc
EM
F1
RAG with White-box LLMs (For Reference Only)
DRAGIN (Su al. Best)
68.9
31.4
42.3
30.4
39.3
GenGround (Shi et 719)
77.1
47.2
52.2
20.2
27.4
45.6
50.2
ChatQA (Liu et al. 708)
12.2
51.4
34.9
37.4
RankRAG (1 at. 208)
42.7
55.4
38.2
43.9
RAFT (Qwen 2.5-nntt 3B)
47.0
38.2
48.1
15.8
11.0
20.7
27.4
36.4
42.1
23.2
16.8
25.5
(Llamm-3.1-nnsiruee
13.8
30.
Only)
(jing CPT-3.5
RA-ISK (Liu et Best)
75.9
46.5
36.1
BlendFilter (Wang et GPT-3.5)
74.4
50.8
62.4
40.4
47.0
Search- ol et al., 328))
45.2
57.3
16.6
28.
58.0
71,4
56.
67.8
HteDRAG Yoo et Cemimi-1.5,328
44.4
38.4
49.9
30.6
22.6
35.0
76.8
67.0
75.2
56.8
44.3
54.6
GPT-lo-mini as LLM Reader
CoT (Wei et al.)
78.2
37.2
25.6
37.7
18:4
9.6
21.6
33.8
27.6
34.5
39.7
25.8
42.4
RAG at.l)
78.6
58.4
41.8
57.1
22.6
11.4
23.4
45.0
37.2
45.9
28.0
23.2
32.5
RAG GPT-to-mini
(Khot al.)
76.8
61.6
46.6
60.9
41.4
24.2
39.3
74.0
59.0
71.5
57.6
45.6
60.6
RAG GPT-lo-mini Rerank
76.8
56.0
40.2
56.5
23.2
12.4
25.3
44.2
36.8
44.7
34.4
28.0
37.0
RAFF (Mao
55.4
20.2
11.8
23.8
43.4
36.2
39.3
30.4
24.0
31.5
IRCoT shot (Trived)
77.7
44.8
57.6
31.4
18.8
31.2
54.8
43.6
36.6
37.6
28.8
41.6
IRCo]-10 shot (Irivedii et
RetGen
25.8
40.4
50.2
65.3
50.4
40.8
51.0
RO-RAG et at
73.5
60.2
45.4
59.5
31,0
19.2
31.0
59.6
50.2
58.8
25.2
14.8
21.5
RAG-Star (jiang et al.)
69.0
49.0
46.0
60.0
27.0
22.0
30.7
43.0
38.0
46.8
RAG-Cym (Xiong al,y
59.4
Collab (Owen-2.5-38 backbone)
39.6
63.0
74.5
55.4
Collab-RAC (Llama..311 backbone)
$1.6
53.0
65.6
45.8
26.4
42.4
79.0
63.2
74.6
CoT (Wei al.)
79.5
$6.2
29.4
48.9
26.8
17.0
28.9
58.4
11.8
53.6
60.8
12.8
59.5
RAG (Lewis al.)
81.2
61.0
17.2
63.6
29.8
17.4
30.1
57.8
45.8
57.1
35.2
27 2
37.2
RAG w/ GPT-4o Decompose (Khot el.)
83.2
67.2
52.2
65.6
44.8
27.8
42.3
78.8
62.2
73.3
69.6
62.4
74.9
RAG w GPI-4 Heeank
80.8
60.6
46.4
59.5
27.3
15.6
27.1
49.4
42.2
30.2
43.2
79
42.6
RATE (Mao et
78.6
61.0
41.
58.8
22.8
13.8
23.2
50.6
43.2
44.1
35.2
26.4
36.9
IRCoT-0 shot (Trivedi et
78.6
64.2
48.0
63.7
33.8
22.4
33.5
61.4
51.4
61.0
60.8
46.4
56.9
IRCoT:10 shot (Trivedi et al.)
81.2
66.4
528
66.0
44.2
29.4
43.9
78.0
62.2
70.0
66.4
57.6
70.0
Bee-RetGen
714
52.8
62.4
67.7
RQ-RAG (Chan al
77.3
62.0
46.2
60.3
31.6
20.0
32.4
602
50.8
59.5
32.8
23.6
33.9
RAG-Star (liang al.""
81.0
57.0
48.0
68.6
40.0
29.0
43.5
63.0
48.0
61.7
collab-RAG 38l (Qwmm-2.5538 as backhone)
82.5
68.3
400
67.0
63.6
60.0
Collab (Llame-3.1-88 backbone)
29.0
77.0
Table 1: Comparison of various baselines multiple datasets. Baselines with specific sizes have been annotated in parentheses. *: Baselines require gold passage annotation labels. t: Baselines require distillation from frontier LLMs (e.g. GPT-4 or GPT-4o). 4: The original paper studied over reading comprehension setting where gold passages given, which is different from the setting of this study.
et al., 2024) and Llama-3. 1--1--8IIsst Instruct (Dubey al., 2024) white-box SLMs, and con- sider GPT-4o-mini and GPT-4o (Hurst et al., 2024) as black-box LLMs during evaluation. In the model training stage, we only use GPT-4o-mini the LLM reader.
Hyperparameters. Training Collab-RAG is conducted on eight NVIDIA A100 GPUs. We employ the AdamW optimizer with a learning rate of 2e-6 for both the Qwen-2.5-3B and Llama-3.1-81 models during the warmup stage, and 1e-6/5e-7 for Qwen-2.5-3B and Llama- 3.1-8B, respectively, in the preference optimization stage. The batch size set to 64. We set 0.5, N = 5 in iterative preference optimization by default. For retrieval setup, use the Dragon-Plus (Lin et al., 2023) as the retriever and set the number of retrieved passage k to 10 To ensure fair comparison, we use the same retriever for baselines and test the performance of k E (5, 10, 15, 20} and report the best performance for baselines. For generation, we use greedy sampling and set the max number generated tokens to 64.
5.3 Main Experiment Results
Table compares Collab-RAG and baselines. We have the following observations:
Collab-RAG exhibits strong empirical performance: With lightwighted GPT-40-mini as the backbone, it outperforms existing black-box and white-box retrieval-augmeetted language models on complex QA tasks by 14. 44.2% and 6.6% on average.
chttps://huggingg coleee..c./oo ho/faaebook/draaa drap//ddgggopppeoo cisionnplu-cccee context-encoder We study the effect of different and / in Appendix C.1.
7

---

Published as a conference paper at COLM 2025
Collab SAG
Collab RAG AB
Self Decompose
Collab RAG JB
Distillation Qwen
Llame 3,1 1.1-BB es Reads
Owan 25-11B
ELM Reader
GPT-4o LLM Raader
GPT-4o as ELM Reader
Exact Match
60
Toace Match
60-
60
40-
40-
40-
20
HoparQA
HorporQA
HorporQA
MaeiQue
HeeporQ 2W14MQQ
Figure 3: Different LLM Readers
Figure Collab v.s. Distillation
Retrive
Collab RAG-3B
Coliab-RAG-18
Collab-RAG--gg
X Collab-RAG
Qwes-2.5
RAG-8B
2WILMQA
0.8
Exaut Match
50
0.7
HospotQA
50-
0.6
R5
0.5
MusiQue
0.4
16'
R
R6910
Ri020
Frozen
SFT
DPO41 DPO-2 DPO-3
Paramss (B)
(a) Recall Analysis
(b) Different Iterations
(c) Different Decomposition LMs
Figure 5: Additional Studies. GPT-4o-mini as the default LLM reader.
Collab-RAG is efficient: Using an 8B model for question decomposition, it outperforms LLM-based decomposition baselines on most datasets (5/5 for GPT-4o-mini, 4/5 for GPT-4o) in EM, without requiring GPT-4o distillation. With 3B model, it surpasses the GPT-44-based decomposer with an average gain of 0.7%.
Collab-RAG achieves competitive performance against recent baselines: It surpasses RAG-Gym on 2/3 datasets and RAG-Star on 3/4 datasets. Note that these baselines are based on process reward models and inference-tine search algorithms. Since our contributions are orthogonal to these methods, they have the potential to be combined for further improvements.
Ablation Studies
Effect of Different Designs in Finetuning SLMs. We analyze the impact different components in Table with GPT-4o-mini LLM reader. Notably, incorporating relaxed reward (e.g., accuracy-based reward) and a format-based reward contributes to improved EM performance across target datasets. Additionally, relying solely on single-step SFT and DPO results in significant performance degradation, particularly when using Llama-3-8B as the backbone, highlighting the necessity of iterative preference optimization.
Different LLM and Retriever Backbones. Figure 3 presents the performance of Collab-RAG with various LLM readers (Llama-3.1-8B-4 (Dubey et al., 2024) and Qwen-2.5-14B-Itt (Yang et al., 2024)) and Table 3 shows the performance with different retrievers (COCO-DR (Yu et al., 2022), GTE (Li et al., 2023) and E5 (Wang et al., 2022)) when GPT-4o-mini is used as the LLM reader. For LLM backbones, we observe that Collab-RAG provides significant gains, particularly when using a less powerful L.M reader (e-g., 10.7% for Llama-3.1-8B), For retrievers, Collab-RAG mostly outperforms baselines across different retrieval choices, demonstrating its robustness to varying retriever configurations.
Collab-RAG v.s. Direct Distillation from Black-box LLMs. We further study the per- formance Collab-RAG with baselines fine-tuned on synthetic question decompositions generated by GPT-4o-miní and GPT-4o models after rejection sampling. As shown in Fig- ure 4, Collab-RAG consistently outperforms SLM models trained through distillation. This suggests that simple distillation may not be the optimal solution for query decomposition.
8

---

Published as a conference paper at COLM 2025
HotpotQA EM
MusiQQe EM
ZWiKIMQQ
EM
HotpoiQQ FM
MusiQue
FM
2WikiMQA
FM
Collab RAG (Qwen-255BB) 2-38B
w/o Formal Reward
302
51.6
23.2
23.4
62.2
63.0
Collab RAG (Qwen-225-BB)
COC O-DR
304
51.6
280
25.4
63.0
w/o Accuracy
E5-Large W/
w/o Herative DPO
SET Only
4941
62.0
61.8
304
25.6
64.2
Rallab-RAG 1..ma-3.1-8B)
w/o iteaative DPO
w/o Format Reward
w/o Accuracy Reward
52.0
47.6
53,0
49.4
24.8
24.2
26.4
63.2
64.2
62.4
62.4
Collab RAG (Llama-33
w/ CTE
w/ COCOED
w/ E5-Large
30.4
306
52.0
28.4
26.6
264
1
64.2
65.2
65.0
63.2
Only
Best Baseline Table
46.6
25.8
99.0
Table 2: Different Designs in SLMs.
Table 3: Different Retriever Backbones.
Question
Piers Bizony has written articles for which magazint based in Bristol, UK?
No Decomposition
Answer: None
GPT-to-mini
Decomposed Question: What magazine based Bristol, UK?
Self-decompose
Answer: cannot identify specific magazine based in Bristol from the information given.
Decomposed Question: For which magazine has Piers Bizony written articles?
Answer: The context provided does not mention any magazine based in Bristol, UK.
Collab-RAG-3B
Decomposed Question:
What magazines has Piers Bizony written articles for?
Answer: The Independent, BBC Focus, Wired
Decomposed Question: Among the magazines mentioned in #1, which one is based in Bristol, UK? Answer: BBC Focus
Table 5: A case study comparing GPT-4o-mini's direct answer w/o decomposition, its self- decomposed questions and answers, and the decomposed responses from RRGG33B
5.5 Additional Studies
Recall Analysis. Figure 5(a) shows the passage-level answer recall on the HotpotQA dataset. We observe that while reranking can improve over retrieval, its gain is limited. Instead, for complex queries, decomposition serves more directly way improve search quality.
Performance with Different Iterations. Figure 5(b) illustrates the training dynamics of Collab-RAb across different stages. The results show consistent performance gains during the first 1-2 DPO rounds. However, by stage 3, the improvement begins to plateau. Using rounds strikes balance between performance and efficiency.
Performance with Different Question Decomposition LM. We compare Coll Rollab-RAG against frozen Qwen-2.5 2.5ns..utttt series LMs ranging from 1.5B to 72B on the question decomposition task, using GPT-4o-mini as the default LLM reader. As shown in Figure 5(c), Collab-RAN RAG with 3B backbone outperforms frozen 32B LM (10.7x larger), and with 8B backbone, surpasses frozen 72B LM (9x larger), highlighting its parameter efficiency.
Performance with Different Preference Op- timization Algorithms. Apart from DPO, we also tried to use SimPO (Meng et al., 2024) and ORPO (Hong et al., 2024) as preference optimization cilmizationnoo algorithms, yet we observe that DPO yields most robust performance.
Algurithm
HetporQA
MusiQue
2WikiMQA
collab-RNe (Qwen=2 5-38)
51.6
25,4
w/ ORPO
48.8
24.4
62.
w/ SimPO
47.6
24.2
82.
Table 4: Performance of Different Optimiza- tion Algorithm with EM the metric.
5.6 Case studies
Table 5 presents case study comparing GPT-4o-mini's direct answer w decomposition, its self-decomposed questions and answers, and the decomposed responses from Collab-RAG- 3B. We observe that, given a question from HotpotQA, GPT-4o-mini cannot infer answer from the context retrieved directly from the original question. Even after self-decomposition, it still fails to find an answer the first question is too broad, making it difficult for the decomposes the question in a more structured and human-like manner. This approach allows it to retrieve the right context step by step, ultimately leading to the correct answer.
retriever to retrieve relevant context. In contrast, Collab-RAC,

---

Published as a conference paper at COLM 2025
6 Conclusion
We introduce Collab-RAG, framework that fosters collaboration between white-box SLM and black-box LLM to enhance RAG for multi-hop muestion-answering Through iterative DPO guided by supervision signals from affordable black-box LLM (GPT--oomini)) Collab-RAG significantly enhances the SLM's question decomposition capabilities without expensive human annotations resource-intensive model distillation. Experimental results demonstrate that our training strategy consistently outperforms standard RAG models (14.2%) and strong decomposition-basse exhibiting robust generalization across various black-box LLMs. Collab-RAG presents scalable and efficient solution to improve complex retrieval-augmeened question-answering scenarios. An important line of future work is to extend Col RAG for online reinforcement learning (Jin et al., 2025).
baselines (1.8%) over 5 multi-hop QA datasets,
Acknowledgments
This research was partially supported by the Texas Advanced Computing Center (TACC) and the NVIDIA Academic Grant Program to Dr. Wenqi Shi. RX and CY were partially supported by the US National Science Foundation under Award Numbers 2319449, 2312502, and 2442172, well as the US National Institute of Diabetes and Digestive and Kidney Diseases of the US National Institutes of Health under Award Number K25DK135913. JH were partially supported by the US National Science Foundation (NSF) grant IIS-2145411.
Reproducibility Statement
The five datasets used in this study are publicly available. Detailed descriptions of these datasets and their corresponding tasks are provided in Appendix A. In section 5.1, we describe the experimental setup and key parameter settings. The implementation of Collab-RAG will be open-sourced upon acceptance.
Ethics Statement
Collab-RAG involves the usage of OpenAI APIs. We follow the data usage guidelines for interactions with Microsoft Azure's OpenAl API service and opt out of the human review process by completing and submitting the Azure Open Additional Use Case Form. We do not foresee other ethics issues.
10

---

Published as a conference paper at COLM 2025
References
Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. RQ-RAG: Learning to refine queries for retrieval augmented generation. In First Conference on Language Modeling, 2024.
Jaepill Choi, Kyubyung Chae, Jiwoo Song, Yohan Jo, and Taesup Kim. Model-based preference optimization in abstractive summarization without human feedback. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18837-18851, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, and Ji-Rong Wen. Understand what I1m needs: Dual preference alignment for retrieval-augmentee generation. Xiu preprint ar Xio::4406..187 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arviv preprint wxx::2207.21788. 2024.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arviv preprint ar arXiv:2501.12944 2025.
Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing multi-hop qa dataset for comprehensive evaluation of reasoning steps. ar arXit preprin! ar arXie:2011.01060 2020.
Jiwoo Hong, Noah Lee, and James Thorne. Orpo:
Monolithic preference optimization without reference model. Xiv preprint ar Xfo::24433,7661 2024.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-40 system card. arv preprint ar Xie:2410.21276, 2024.
Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong Park. Adaptive- RAG: Learning to adapt retrieval-augmentte large language models through question complexity. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7036-7050. Association for Computational Linguistics, 2024.
Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Wayne Xin Zhao, Yang Song, and Tao Zhang. Rag-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement. Xiv preprint ar X1::2412..,8881 2024.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yuu Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp- 7969-7992. Association for Computational Linguistics, 2023.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search- rl: Training lims to reason and leverage search engines with reinforcement learning, arviv preprint ar Xiv: 2503.09516, 2025.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. remonstrate-searcchpreeiie Composing retrieval and language models for knowledge-intensive nlp. XrXiv preprint arXtt:2212.14022 2022.
11

---

Published as a conference paper at COLM 2025
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2023.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledge-intensivee nlp tasks. Advances in neural information processing systems, 33:9459- 2020.
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. aiv preprint Xiv:2501. 003666, 2025.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang, Towards general text embeddings with multi-stage contrastive learning. arviv preprint ar axXit:2388.03281 2023.
Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz,Jimmy Lin, Yashar Mehdad, Wen- tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. In Findings of the Association for Computational Linguisties: EMNLP 2023, pp. 6385-6400, 2023.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. RA-DIT: Retrieval-augmeened dual instruction tuning. In The Twellfh International Conference on Learning Representations, 2024.
Tianci Liu, Haoxiang Jiang, Tianze Wang, Ran Xu, Yue Yu, Linjun Zhang, Tuo Zhao, and Haoyu Wang. Roserag: Robust aatrivval-augmentr generation with small-scale llms via margin-aware preference optimization. arXiv preprint ar xxXi:22022.1093 2025.
Yanming Liu, Xinyue Peng, Xuhong Zhang, Weihao Liu, Jianwei Yin, Jiannan Cao, and Tianyu Du. RA-ISF: Learning to answer and understand from retrieval augmentation via iterative self-feedback, In Findings of the Association for Computational Linguistics: ACL 2024, pp. 4730-40 4739..4......
Association for Computational Linguistics, 2024a.
Zihan Liu, Wei Ping, Rajarshi Roy, Peng Xu, Chankyu Lee, Mohammad Shoeybi, and Bryan Catanzaro. ChatQA: Surpassing GITT-4 on conversational QA and RAG. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in serieval-augmenter large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5303-5315, 2023.
Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. RaFe: Ranking feedback improves query rewriting for RAG. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 884-901. Association for Computational Linguistics, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference-free reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systemss, 2024.
Fengran Mo, Kelong Mao, Yutao Zhu, Yihong Wu, Kaiyu Huang, and Jian-Yun Nie. Convgqr: Generative query reformulation for conversational search. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4998- 5012, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2735 35:27730-27777 2022.
12

---

Published as a conference paper at COLM 2025
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason E Weston. Iterative reasoning preference optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. Is question decomposition unit all we need? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4553-4569. Association for Computational Linguistics, 2022.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, 2023.
Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp- 9248 9284.
Association for Computational Linguistics, 2023.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8371-8384. Association for Computational Linguistics, 2024a.
Zhengliang Shi, Shuo Zhang, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and generation for multi- hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7339-7353, 2024b.
Zhaochun Ren. Generate-then-ground in retrieval-augmenner
Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large lan- guage models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12991-13013, 2024.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 10014-10037, 2023.
Prakhar Verma, Sukruta Prakash Midigeshi, Gaurav Sinha, Arno Solin, Nagarajan Natarajan, and Amit Sharma. Plan x rag: sainning-guider retrieval augmented generation. Xiv preprint ar Xie:2240..20753, 2024.
Haoyu Wang, Ruirui Li, Haoming Jiang, Jinjin Tian, Zhengyang Wang, Chen Luo, Xianfeng Tang, Monica Xiao Cheng, Tuo Zhao, and Jing Gao. BlendFilter: Advancing retrieval- augmented large language models via query generation blending and knowledge filtering. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1009-1025. Association for Computational Linguistics, November 2024.
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervise contrastive pre-training. Xiv preprint ar arXin:2212.03533 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al. Chain-of tfthhught prompting elicits reasoning in large language models. Adoances in neural information processing systems, 35:24824-24883 2022.
Zhepei Wei, Wei-Lin Chen, and Yu Meng. InstructRAG: Instructing retrieval-augmenter -augmenteg generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representations, 2025.
13

---

Published as a conference paper at COLM 2025
Guangzhi Xiong, Qiao Jin, Xiao Wang, Yin Fang, Haolin Liu, Yifan Yang, Fangyuan Chen, Zhixing Song, Dengyu Wang, Minjia Zhang, et al. Ryg-gym: Optimizing reasoning and search agents with process supervision. arxiv preprint arXiv:2502.13957 2025.
Ran Xu, Hui Liu, Sreyashi Nag, Zhenwei Dai, Yaochen Xie, Xianfeng Tang, Chen Luo, Yang Li, Joyce C Ho, Carl Yang, and Qi He. Simrag: Self-improving retrieval-augmented generation for adapting large language models to specialized domains. arXiv preprint arXit:2410.17952, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint ar xxXie22412.15115 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdi- nov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Pp. 2369-2380, 2018.
Wenhao Yu, Zhihan Zhang, Zhenwen Liang, Meng Jiang, and Ashish Sabharwal. Improving language models via plug-and-pla retrieval feedback. ar Xiv preprint ar Xio:23355.1402 2023.
Wenhao Yu, Hongming Zhang. Xiaoman Pan, Peixin Cao, Kaixin Ma, Jian Li, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmented language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 14672-14688 1465. Association for Computational Linguistics, 2024a.
Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. Coco-dr: Combating distribution shift zero-shot dense retrieval with contrastive and distributionally robust learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1462-1479, 2022.
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. RankRAG: Unifying context ranking with -lugeeee- generation in LLMs. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024b.
Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang, and Michael Bendersky. Inference scaling for long-contex retrieval augmented generation. In The Thirteenth International Conference Learning Representations, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476-15488, 2022.
Ruizhe Zhang, Yongxin Xu, Yuzhen Xiao, Runchuan Zhu, Xinke Jiang, Xu Chu, Junfeng Zhao, and Yasha Wang. Knowpo: Knowledge-aware preference optimization for con- trollable knowledge selection in retrreval-augmentee language models. arxiv preprint arXiv:2408.03229, 2024a.
Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. RAFT: Adapting language model to domain specific RAG. In First Conference on Language Modeling, 2024b.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, et al. Secrets of rlhf in large language models part Ppo. Xrv preprint ar Xiv:2307 049644, 2023.
Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, et al. Rageval: Scenario specific rag evaluation dataset generation framework. arXiv preprint arXim:2408.01222,,22 2024.
14

---

Published as a conference paper at COLM 2025
A
Dataset Details
Here are the details for each dataset used in our experiments:
HotpotQA (Yang et al., 2018): A large-scale multi-hop QA dataset that requires reasoning over multiple Wikipedia passages to answer fact-based questions.
2WikiMQA (Ho et al., 2020): A dataset extending HotpotQA, featuring questions that require reasoning across two different Wikipedia articles, emphasizing cross- document retrieval.
MuSiQue (Trivedi et al., 2022): A challenging multi-hop QA dataset with questions that explicitly require reasoning across multiple sentences scattered across different documents.
Strategy (Geva et al., 2021): A dataset focusing on implicit reasoning, where answering requires strategic multi-step inference rather than direct fact lookup.
Bamboogle (Press al., 2023): A dataset designed to evaluate LLMs' ability to answer adversarial and compositional questions, requiring careful decomposition and reasoning.
B Baseline Details
Here are the details for each baseline used in our experiments:
DRAGIN (Su et al., 2024): It dynamically determines retrieval timing and content by modeling the evolving information of the language model generation.
GenGround (Shi et al., 2024b): It iteratively generates simpler single-hop questions and directly grounds their answers through retrieved external documents, combin- ing internal model knowledge and external context to solve multi-hop questions.
ChatQA (Liu et al., 2024b): It enhances retrieval-augmentee conversational ques- tion answering via a two-stage instruction-tuningg initial fine-tuning on general instruction-folloiing data followed by specialized context-enhanced tuning.
RankRAG (Yu et al., 2024b): It unifies context ranking and answer generation into single instruction-tunee LLM, by introducing a laakinn-as-generati task during training, training the model to rerank retrieved contexts before generating answers.
RAFT (Zhang et al., 2024b; Lin et al., 2024): It employs instruction tuning to train the LLM for producing chain-of-thought answers explicitly grounded in relevant retrieved contexts.
IRCOT (Trivedi et al., 2023): It interleaves retrieval and chain-of-thought reasoning, progressively refining reasoning steps and associated retrieval queries based on previous outputs.
FLARE (Jiang et al., 2023): It iteratively predicts upcoming sentences to actively de- cide when and what information retrieve during generation, enabling continuous retrieval based anticipated information needs.
RA-ISF (Liu et al., 2024a): It iteratively decomposes complex tasks and integrates self-feedback mechanisms across submodules, refining retrieval and generation steps to minimize irrelevant contexts.
BlendFilter (Wang et al., 2024): It iteratively blends query generation with knowl- edge filtering, employing LLM-generated feedback to dynamically eliminate irrele- vant information from retrieved contexts.
Search-ol (Li et al., 2025): It uses an agentic retrieval mechanism that dynamically retrieves external information upon encountering uncertain reasoning steps and employs a dedicated Reason-in-Documenn module to filter out irrelevant details.
ItAG (Yue et al., 2025): It scales inference through iterative retrieval and gen- eration steps, employing flexible test-time strategies such as increasing retrieved
15

---

Published as a conference paper at COLM 2025
documents generation steps, thereby improving the effective utilization of con- textual information in long-contex: reasoning tasks.
COT (Wei et al., 2022): It enhances LLM's reasoning abilities by guiding them to generate intermediate reasoning steps before generating the final answer.
RAG (Lewis et al., 2020): It retrieves top passages from the corpus before generating the answer.
RAG with question decomposition (Khot al., 2023): It uses the same LLM the reader model to break down complex questions into simpler sub-questions, then retrieves relevant information for each, and synthesizes the answers to address the original question.
RAG with reranking: It uses the same LLM as the reader model rerank the top-retrieved passages before generating the final answer.
RAFE (Mao et al., 2024): rewrites questions based on the LLM ranking feedback.
Iter-RetGen (Shao et al., 2023): It interleaves retrieval and generation processes in multiple iterations, allowing the model refine its queries and responses progres- sively for better accuracy.
RQ-RAG (Chan et al., 2024): explicitly enhances RAG by training the model to re- fine, rewrite, decompose, and disambiguate complex queries, addressing limitations in ambiguous or insufficiently detailed original queries.
RAG-Star (Jiang et al., 2024): It integrates RAG with Monte Carlo Tree Search, itera- tively using retrieved information to guide and improve the tree-based deliberative reasoning process of language models.
RAG-Gym (Xiong et al., 2025): It employs fine-grained, step-wise process supervi- sion and trained reward models to iteratively optimize the retrieval and generation processes.
C Additional Experimental Results
C.1 Parameter Study
Table shows the result of Collab-RAG with different B and From the result, we observe that Collab is not sensitive to the selection of B. Besides, setting K too large or too small will lead to performance degradation. This is because too small yields limited recall, yet too large can introduce many irrelevant information.
Table 6: Performance of Collab-RAG with Qwen-2.5-3B on different benchmarks under varying 6 and k settings.
Parameter
HotpotQA
MusiQue
2WikiMQA
β 0.5, k
50.8
24.6
60.6
0.5, k 10 (Our Setting)
51.6
25.4
63.0
0.5, 15
51.4
25.8
62.4
0.1, k 5
50.6
24.8
60.2
0.1, k 10
51.0
25.2
61.4
0.1, k 15
50.4
24.6
59.8
C.2 Performance with Different Size of White-box Query Decomposer
Table lists detailed performance on each dataset for directly prompting frozen white-box LMs for query decomposition, using GPT-4o-mini the L.M reader.
16

---

Published as a conference paper at COLM 2025
Table 7: Performance of LLM query decomposers across multiple QA datasets.
Model
StrategyQA Acc
HotpotQA EM
MusiQue EM
2WikiMQA EM
Bamboogle EM
Acc
F1
Acc
F1
Acc
F1
Acc
Fi
wwen-2.5-1.5D-insttee
74.2
442
32.4
43.2
25.4
16.4
25.9
49.4
40.2
49.2
46.4
38.4
51.7
Qwen-2.5-3B-instruu
75.1
53.8
44.2
56.1
40.8
22.2
37.5
62.0
50.0
60.0
52.0
43.2
53.8
wven-2.5-7B-instrune
76.0
61.2
47.2
60.3
35.8
22.6
36.0
65.2
53.8
64.6
53.6
43.2
54.1
Qwen-2.5-14B-instrun
74.2
64.6
50.4
63.6
40.6
25.8
39.2
71.8
56.8
68.0
56.0
45.6
57.7
Oven-2.5-328-mmsmmin
79.9
40.2
25.4
39.2
78.8
73.6
78.6
68.6
55.6
68.3
43.6
26.2
39.9
83.0
67.0
77.9
56.0
47.2
64.8
D Prompt Details
The detailed prompts for query decomposition as well question answering is listed in the following. It worth noting that for the decomposition step, we select two demonstrations from the training set the input to the LLM for buth our method and baselines to enable fair comparison.
Listing 1: Prompts for Query Decomposition
Please break down the given question into multiple specific sub uub-questions that address individual components of the original question.
Please generate the decomposed sub-questions for the below question. The sub-question should be labeled with a reference to previous answers (e. #1) when needed. For example, #1 means the answer for decomposed question 1.
Here [[Begin of the Example 1]]
examples:
## Question:
What is the average winter daytime temperature in the region containing Richmond, in the state where WXBX is located?
## Decomposed Question:
### Q1: Which state is WXBX located?
### Q2: In which of #1 's regions is Richmond?
### Q3: what is the average winter daytime temperature in #2?
[[End the Example 1]]
[[Begin of the Example 2]]
## Question:
How long was the place where Yongle Emperor greeted person whom the edict was addressed the capitol of the area where Guangling District was located?
## Decomposed Question:
**# Q1: Who was the edict addressed to?
**# Q2: Where did the Yongle Emperor greet #1 ?
**# Q3: Where does Guangling District locate?
### Q4: How long had #2 been the capital city of #3
[[End the Example 2]]
Now decompose the following question:
## Question: [question]
Decomposed Question:
Listing 2: Prompts for Answering Subquestions
You have the following context passages: [Retrieve top-k Context]
Please answer the question "[subquestion]" with short using the context reference.
17

---

Published as a conference paper at COLM 2025
If answer is found in the context, your knowledge.
Do not give any explanation. Your answer needs to be short as possible
Listing 3: Prompts for generating the final answer for Question Answering
For the question: [original question]
We have the following decomposed sub sub-questions and sub answers [subquestion and answers]
Based on these, provide the final concise answer to the original question Do not give an explanation.
18


Reviews:
[{"review_id": "6alNn1CSpk", "reviewer": "Reviewer_j6Ne", "summary": "The paper focuses on the task of answering complex questions using retrieved passages and a reader language model. The proposed method called \"Collab-RAG\" consists of a two step pipelined approach: (i) in the first step, a smaller language model decomposes a complex question into a series of simple (or factoid) atomic questions such that these atomic questions can be answered using a collection of relevant passages; (ii) In the second stage, these sub-questions and the relevant passages are given as input to a larger black-box LLM that processes them to generate the final answer. \n\nIn Collab-RAG, the trainable module is the smaller language model. The proposed training approach consists of first fine-tuning it using rejection sampling followed by iterative DPO (Direct Preference Optimization). Experiments are conducted on multiple multi-hop QA datasets and demonstrate comparable or new state-of-the-art results against recent baselines. The paper also contains multiple ablation studies to demonstrate the usefulness of different design decisions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is very well-written. I really enjoyed reading the paper. Almost everything is quite well done. The proposed method is elegant and carefully explained. The experiments are well conceived, and the results are compared against a lot of baselines.\n\n* The idea of finetuning a language model to breakdown a complex questions into a simpler one is not necessarily novel, the main novelty consists in the training algorithm. The training approach first trains the query decomposer language model on high-quality sub-questions dataset and then further finetunes it using preference optimization using feedback as a relevance signal from a larger language model.\n\n* The experimental methodology look sound and the results are compared against many recent approaches and strong baselines. Using their proposed training method, authors are able to get good empirical performance gains that either matches or outperforms the previous approaches.", "reasons_to_reject": "* Regarding experiments with different question decomposition language models: the paper has mainly used a couple of smaller language models for finetuning. I would like to see a discussion when black box models such as Gemini, GPT-4o, etc. are used for question decomposition.\n\n* Specificity of proposed method: Is this approach only valid for complex questions. Can the author demonstrate the effect of this framework on simple question answering tasks as well. Some of the datasets that can be used for evaluation are SimpleQA, Natural Questions etc.\n\n* Finetuning the reader models: Almost all of the results in the paper are based on using larger reader models as black-box. It will be interesting to see how the effect of finetuning readers models as well and this impacts the iterative DPO training.", "questions_to_authors": "* To my knowledge, the paper leverages a static dense retriever. It will be good to see a discussion in the paper when the retriever is further finetuned on the retrieval task for each of the datasets.\n\n* I would suggest adding this baseline to the paper: Given a complex question, retrieve all the relevant documents such as top-500, top-1000, top-5000, etc. from the retriever and give as input to the reader (large) language model to answer the question. It will be useful to understand how much the reader LM can use the long-context to understand the answer the question.\n\n* As Collab-RAG is an iterative approach during training, it will be nice to briefly include a discussion of the system time requirement of the system in the paper during training and inference and how it compares to current approaches.", "extra_scores": {}}, {"review_id": "zI8GX9L6u8", "reviewer": "Reviewer_sLb3", "summary": "The paper presents Collab-RAG, a framework for retrieval-augmented generation that combines a trainable query decomposer (SLM) with a black-box LLM reader. The decomposer generates sub-questions, which are used to retrieve context and generate answers via the black-box LLM. The SLM is trained using DPO, guided by rewards derived from the final answer quality.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* It is interesting to formulate query decomposition as a learnable task using a white-box SLM.\n* The empirical evaluation is comprehensive, spanning multiple datasets, LLM readers, and decomposer model sizes.", "reasons_to_reject": "* I find that the core motivation of this work (improving complex question answering via iterative retrieval and reasoning) largely overlaps with that of IRCoT. However, unlike IRCoT’s simple and training-free zero-shot prompting approach, Collab-RAG introduces notable system complexity and computational overhead by requiring iterative training of a white-box SLM using DPO. Also, the improvement in EM scores is sometimes marginal, which raises concerns about whether such added complexity is practically justified in real-world deployments.\n\n\n* I am concerned about the fairness of the comparison to prior black-box RAG methods, particularly IRCoT. Specifically, IRCoT relies on GPT-3 as the reader, while Collab-RAG uses GPT-4o-mini, a significantly more capable model. Given the substantial performance gap between GPT-3 and GPT-4 series models in reasoning and contextual understanding, the observed performance improvements may be partially attributed to the underlying LLM upgrade rather than the proposed collaborative framework alone. To more clearly isolate the contribution of Collab-RAG’s decomposition and optimization mechanisms, I encourage the authors to include a comparison where IRCoT-style retrieval and reasoning is applied using the same GPT-4o-mini as the backbone. \n\n\n* Related to the question above, the authors categorize IRCoT and related methods as “reference only” in Table 1, yet provide no clear explanation for this distinction. Given that IRCoT is conceptually aligned with Collab-RAG in both motivation and design and that its implementation is publicly available and easily runnable with the same black-box LLM used in this work (e.g., GPT-4o-mini), this categorization appears arbitrary. I encourage the authors to clarify their rationale for this labeling and to report the results under a unified experimental setup. In other words, for a fair and meaningful comparison, I strongly recommend rerunning IRCoT (and other black-box-based baselines) using the same retriever and LLM configuration as Collab-RAG, and reporting the full set of metrics accordingly.\n\n\n* The number and granularity of sub-questions can critically affect both effectiveness and efficiency. It would be valuable to include such analyses.\n\n\n* As the final answer accuracy is used as a proxy for sub-question quality during training, I think it would be better to include the analyses on the direct assessment of the quality of the decompositions themselves.", "questions_to_authors": "See above.", "extra_scores": {}}, {"review_id": "LLZ1nbkjSE", "reviewer": "Reviewer_vqF1", "summary": "This paper introduces a method to improve multi-hop question answering (QA) by facilitating interaction between a white-box small language model (SLM) and a black-box large language model (LLM). In this setup, the SLM is responsible for query decomposition and improved retrieval, while the black-box LLM acts as the reader and synthesizer to produce the final answer. To train the query decomposer, the authors propose a self-improving training strategy that leverages feedback from the black-box LLM. This creates a multi-turn interaction loop, which iteratively refines the decomposition strategy and ultimately improves answer generation. The approach is evaluated on five multi-hop QA datasets, showing an average performance gain of 6.6% across exact match (EM), accuracy, and F1-score metrics.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper demonstrates promising results through a comprehensive experimental setup, benchmarking against a broad range of strong baselines.", "reasons_to_reject": "While the proposed approach in interesting, the paper requires additional evidence to justify its necessity:\n\nunsupported efficiency claims: The claim regarding the \"efficiency\" of Collab-RAG in improving reasoning and retrieval is not well-supported. Given that the method involves an iterative interaction between models, it is intuitively less efficient than single-step QA approaches. The paper does not clearly define what metric is used to measure \"efficiency\" and provides no quantitative comparison to existing methods in this regard.\n\nimpact of retrieval: although the results suggest that the method is retriever-agnostic, it remains unclear whether performance gains are from improved retrieval (because of query decomposition) or from the simplification of complex queries in the generation phase. \n\nLimited evaluation metrics: The evaluation relies primarily on EM and F1-score. While standard, these metrics are limited in capturing the quality of generated answers. Also, \"accuracy\" is not clearly defined. Human judgment or an automated eval should be included.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "99muuXZmLQ", "reviewer": "Reviewer_YbMU", "summary": "This paper introduces Collab-RAG, a new framework that allows 2 LLMs to collaborate: one is large and black-box, and the other is small and open-box. The large model provides a supervision signal that allows RL training for the small model. Experimental evaluations\n14 across five multi-hop QA datasets demonstrate that Collab-RAG substantially outperforms the baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method shows strong improvement over baselines, including those using much larger LLMs, on all 5 benchmark datasets. \n2. The proposed method can help relieve the need of expensive human annotations or resource-intensive model distillation.", "reasons_to_reject": "1. While efficient, the approach still relies on potentially costly black-box LLMs for feedback\n2. The preference optimization process, while powerful, adds training complexity and may require careful tuning to avoid overfitting. In addition, the generalization ability of a model trained on one dataset is not evaluated on other datasets.", "questions_to_authors": "1. It could be possible that the decomposed questions are wrong, but the final exact match is good. Could this be a problem for the RL training of the white-box LLM? If so, how to avoid it?\n2. It is interesting that `w/o Accuracy Reward` can achieve comparable performance to Collab-RAG. I am wondering why Format Reward is a good single reward in this case. It is very likely the Format Reward is positive but the final answer is wrong, isn't it?", "extra_scores": {}}, {"review_id": "YmGXOpCAvl", "reviewer": "Reviewer_k4rg", "summary": "This paper presents an approach to complex QA, Collab-RAG, that relies on a black-box LLM for providing training signal to a smaller white-box LLM. This is done by performing iterative preference optimization after a supervised fine-tuning warm up step. The experimental results show that the proposed approach is promising, however I have concerns regarding the experimental setup and presentation issues.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Interesting approach to complex QA that utilizes iterative preference optimization as opposed to knowledge distillation.\n* Outperforms baselines under comparison.", "reasons_to_reject": "* The paper presentation could be improved. In Section 4 the training and inference stages are not clearly separated; having separate (sub)sections would help the reader. Also, the framing of the RAG system as an environment in Section 4.2. does not necessarily help in formulating the approach. \n* The compared baselines use different and sometimes older backbone architectures than the ones used in Collab-RAG; also, it is not clear which training data the baselines used and how that relates to the ones Collab-RAG uses. These raise questions regarding the fairness of the comparisons.\n* The claim that this approach is better than knowledge distillation is not convincing; the only evidence provided is in Figure 4, where the differences seem small, and important experimental details are missing, such as how much data have been generated by the teacher model, how does that training set compare to the one used by Collab-RAG, as well as an analysis on the added complexity of having to implement this framework compared to the (simpler) distillation framework.\n* The paper would benefit from quantitative/qualitative analysis on the errors of the model, and also discussion on why the proposed method is under-performed by baselines in Bamboogle.\n* Lacking discussion on relevant related work / baselines [1,2,3]\n\n[1] Jeong et al. Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity. ACL 2024\n[2] Huang et al. Question Decomposition Tree for Answering Complex Questions over Knowledge\nBases. AAAI 2023\n[3] Fu et al. Decomposing Complex Questions Makes Multi-Hop QA Easier and More Interpretable. EMNLP 2021", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents Collab-RAG, a collaborative framework for multi-hop question answering (QA), where a small open-source language model (SLM) decomposes complex questions and a larger black-box language model (LLM) synthesizes final answers. The small model is trained via supervised fine-tuning followed by iterative Direct Preference Optimization (DPO) using feedback from the black-box model. Experiments on five multi-hop QA datasets show that the approach achieves competitive or superior results compared to strong baselines.

Reviewers generally agreed about the paper’s clarity, empirical strength, and novel training procedure, and that the proposed approach offers a promising alternative to knowledge distillation and helps reduce annotation overhead.

Reviewers have shown concerns regarding the ablation study, fair baseline comparisons, and the cost for training/inference; however, the authors have responded well with requested experiments and analyses. 

I recommend that the authors reflect these new results and clarifications made in the rebuttal process in the final version of this paper, along with other reviewer suggestions, such as recommendations for presentation improvements and additional related work. **I especially think unifying the backbone reader (as reviewers SLb3, k4rg pointed out) and adding an ablation study would be important!** 

Adding one more related work regarding utilizing larger models for training another in DPO manner:
Choi et al., Model-based Preference Optimization in Abstractive Summarization without Human Feedback, EMNLP 2024
Decision: Accept