Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning
Authors: Yiqun Chen
Lingyong Yan
Weiwei Sun
Xinyu Ma
Yi Zhang
Shuaiqiang Wang
Dawei Yin
Yiming Yang
Jiaxin Mao
Venue: NeurIPS 2025 poster ()
Keywords: Retrieval-Augmented Generation
Multi-Agent Cooperation
Multi-Agent Reinforcement Learning
Multi-Module Joint Learning
Primary area: deep_learning
TL;DR: This paper is to jointly optimize mutiple modules in complex RAG pipeline using multi-agent reinforcement learning.

Abstract: Retrieval-augmented generation (RAG) is widely utilized to incorporate external knowledge into large language models, thereby enhancing factuality and reducing hallucinations in question-answering (QA) tasks. A standard RAG pipeline consists of several components, such as query rewriting, document retrieval, document filtering, and answer generation. However, these components are typically optimized separately through supervised fine-tuning, which can lead to misalignments between the objectives of individual components and the overarching aim of generating accurate answers. Although recent efforts have explored using reinforcement learning (RL) to optimize specific RAG components, these approaches often focus on simple pipelines with only two components or do not adequately address the complex interdependencies and collaborative interactions among the modules. To overcome these limitations, we propose treating the complex RAG pipeline with multiple components as a multi-agent cooperative task, in which each component can be regarded as an RL agent. Specifically, we present MMOA-RAG\footnote{The code of MMOA-RAG is on \url{https://github.com/chenyiqun/MMOA-RAG}.}, \textbf{M}ulti-\textbf{M}odule joint \textbf{O}ptimization \textbf{A}lgorithm for \textbf{RAG}, which employs multi-agent reinforcement learning to harmonize all agents' goals toward a unified reward, such as the F1 score of the final answer. Experiments conducted on various QA benchmarks demonstrate that MMOA-RAG effectively boost the overall performance of the pipeline and outperforms existing baselines. Furthermore, comprehensive ablation studies validate the contributions of individual components and demonstrate MMOA-RAG can be adapted to different RAG pipelines and benchmarks.

Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning
Yiqun Chen'
Lingyong Yan?
Weiwei Sun
Xinyu Ma² Yi Zhang?
Shuaiqiang Wang" Dawei Yin
Yiming Yang Jiaxin Mao
Renmin University of China Baidd Inc.
Carnegie Mellon University
chenyiqun9990211em frree.rrruu..c edu. cn, maojiaxin@gmail.cm com
Abstract
Retrieval-augmented generation (RAG) is widely utilized to incorporate external knowledge into large language models, thereby enhancing factuality and reducing hallucinations question-answering (QA) tasks. standard RAG pipeline con- sists of several components, such as query rewriting, document retrieval, document filtering, and answer generation. However, these components are typically opti- mized separately through supervised fine-tuning, which can lead to misalignments between the objectives of individual components and the overarching aim of gener- ating accurate answers. Although recent efforts have explored using reinforcement learning (RL. to optimize specific RAG components, these approaches often focus simple pipelines with only two components do not adequately address the complex interdependencies and collaborative interactions among the modules. To overcome these limitations, propose treating the complex RAG pipeline with multiple components as multi-agent cooperative task, in which each compo- nent can be regarded as an RL agent. Specifically, we present MMOA - Multi-Module joint Optimization Algorithm for RAG, which employs multi-agent reinforcement learning harmonize all agents' goals toward a unified reward, such as the F1 score of the final answer. Experiments conducted marks demonstrate that MMOA-RAG effectively boost the overall performance of the pipeline and outperforms existing baselines. Furthermore, comprehensive abla- tion studies validate the contributions of individual components and demonstrate MMOA-RAG can be adapted to different RAG pipelines and benchmarks.
various QA bench-
Introduction
Large Language Models (LLMMs) have been widely applied to tasks such as question answering 1 233
information retrieval 45 2 various forms of reasoning [15] [3] and evaluation [IT] g)
However, since LLMs cannot promptly update their internal knowledge after pre-training, they still prone generating outdated fabricated responses (58). To address these challenges, Retrieval-Augmented Generation (RAG) enhances the generative capabilities of LLMs by retrieving relevant information from external knowledge sources. Recent RAG systems often built as complex pipelines comprising multiple interconnected modules [10], including query rewriting
30 first-stage retrieval 255 40 re-ranking 36 37). document preprocessing 22 27 and
answer generation [40] (4)).
The complexity of RAG systems makes their optimization particularly challenging. Standard super- vised fine-tuning (SFT) optimizes each module independently using human-annotated data. However,
* "Corresponding author.
2The code of MMOA-RAG is on http://gtttuuuucccceeeeeeeeeeeee http:/gghhhg...o///eee/e//g/////
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

this often results in misalignment between the objectives of individual components and the over- arching goal of the system of generating high-quality results. For example, retrieval modules are frequently trained on human-labeled relevance data to optimize metrics such as nDCG [8]. However, this process does not address the disconnect between document relevance and response quality documents with high relevance scores do not always contribute to generating accurate answers [6]
To address this issue, existing work on end-to-ent optimization for RAG, such as 24 12 25 37 aims
to propagate rewards from the final output to intermediate modules using techniques such as attention distributions [T], generation probability 25 56], and expectation maximization (EM) iterations
[2
36). However, earlier approaches primarily focus on simplified pipelines with only two components- retriever and a generator-and fail provide generalizable framework for jointly optimizing complex systems with multiple components and richer interdependencies. More recent methods attempt to eliminate the need for module-specific rewards by leveraging algorithms like Direct Preference Optimization (DPO) [33] and Proximal Policy Optimization (PPO) Nonetheless, these methods still concentrate on optimizing individual RAG modules in isolation, without adequately modeling the collaborative dynamics between interacting components (27 30 22]. Effectively capturing interdependencies among multiple modules and jointly optimizing complex RAG architectures remains an open research challenge.
In this paper, we propose a novel approach called the Mudti-Module joint Optimization Algorithm (MMOA -AG) to enable joint optimization across multiple modules in a RAG system. Our framework treats each intermediate component in the RAG pipeline as an agent and formulates the optimization process Cooperative Multi-Agent Reinforcement Learning (Co-MARL) problem, where the agents (i.e., modules) work together to maximize a shared reward for the final outcome.
Specifically, we focus on applying MMOA-RAG to a RAG pipeline that includes four key modules: query rewriter, a fixed document retriever, a document selector, and an answer generator. Our primary objective is to optimize these modules by defining the final reward as the correctness of the generated response, measured by the Fl score against the ground-truth answer. To achieve this, we leverage the Multi-Agent PPO (MAPPO) algorithm [55]. which facilitates collaborative optimization in a fully cooperative setting. This means that all modules work in a cooperative way, with their optimization goals aligned toward producing high-quality answers. Unlike previous methods that rely on DPO 662 61 or PPO 30 22 MMOA RAG offers greater flexibility for different pipeline designs and excels at promoting collaboration among multiple modules. This end-to-end optimization ensures that each module's objectives are consistently aligned with the overarching goal of generating accurate responses.
To demonstrate the effectiveness of the MMOA-RAG modeling and optimization approach, we conducted experiments on three publicly available QA datasets, HotporQA (53) 2WikiMultihopQA
14 and AmbigQA BI], based on -lama-3-8B- --Istruct (7.. The experimental results indicate that MMOA-RAC achieves better performance than series of existing optimization methods for RAG. Additionally, we performed extensive ablation studies to investigate the effectiveness and advantages of jointly optimizing mulitple modules in the RAG system and the generalizability of MMOA-RAG across different RAG pipelines.
Our main contributions are as follows:
We innovatively model RAG as a multi-agent collaborative task, treating multiple modules within the RAG pipeline as individual agents.
We employ a multi-agent reinforcement learning algorithm to jointly optimize sophisticated RAG system with four key modules: a query writer, a fixed document retriever, a document selector, and an answer generator.
We conduct extensive experiments to verify and demonstrate the effectiveness and generalizability of the proposed framework.
2 Related Works
End-to-end Optimization in OpenQA Lewis et al. 255 introduces Retrieval-Augmentee Gen- eration as RAG, which combines pre-trained language models with pan-aaarettrc memory for improved performance on mnowledge-intensivy NI.P tasks. And some other methods 24 12 proposed end-to-end optimizing framework for OpenQA system.
56
2

---

Trainable
Freezed
Score
Individuall Pemalty
Reward of Each Agent
Adder
Workflow
Sub-questions: subq
Documents Set: D
Selected D selected
4
Ans predict
Sub-q
Initial question
Sub.q 2
Predicted answer
Sub-q 3
Query Rewriter
Retriever
Selector
Generator
RQR
PQR
Rs
Ps
R
PG
Golden answer. AnSggoddnn
Shared Reward
Optimization
Figure 1: The overall framework of MMOA-RAR
RAG without Parameters Updating Some works 20 50 44
46
design a novel RAG mechanism
without parameters updating, enhancing the performance of LLMs on question answering tasks.
RAG with Parameters Updating Some works [1 57
optimize RAG with supervised fine-tune LLMs. Specifically, Search-rl [2 and R1-Searcher [43] both use answer- based reward to improve the reasoning in RAG. used optimize LLMS in RAG task More detailed related works can be seen in Appendix A
fine-tuning. PPO [39] is used by some works [39] 30 22
Additionally, DPO [3 similar alignment algorithms
35
59 27
3
Method
3.1
Modeling RAG as Co-MARL
In this work, we conceptualize the RAG procedure within a cooperative multi-agent reinforcement learning (Co-MARL) framework. Within this framework, each module of the RAG pipeline functions an individual RL agent. The overarching objective of this multi-agent system is produce high-quality answers, which aligns with the individual goals of each module.
We define the tuple OG, R), where G denotes the set of agents in the Co-MARL system, O represents the observation information available to each agent, A constitutes the action space accessible to each agent, and R is the reward shared among all agents. The ultimate aim is to maximize this shared reward, thereby achieving higher evaluation metrics and enhancing the overall performance of the RAG system.
this paper we utilizes Multi-Agent PPO (MAPPO) [55], which is an extension of the PPO algorithm 39 for multi-agent environments, to optimize the policy for each agent in the Co-MARL framework. In fully cooperative settings, unlike PPO, which focuses on single-agent scenarios with individual reward, MAPPO employs shared global reward to promote cooperation among all agents.
3.2 Overall of MMOA-RAG
RAG systems typically follow a modular architecture composed of multiple interconnected compo- nents. Figure I illustrates the architecture of our MMOA-RO -RAG framework, which consists of four primary modules: the Query Rewriter, the Retriever, the Selector, and the Generator:
Query Rewriter reformulates the initial question which may be too complex or ambiguous to resolve with a single retrieval, into set of sub-questions denoted as subg.
Retriever retrieves relevant documents from the corpus for each sub-questions, respectively, and outputs set of candidate document D.

---

Selector further filters D to obtain subset of documents D selected that is useful for generating the final answer the initial query q.
Generator leverages selected to generate the predicted answer A to the initial question.
Since the Query Rewriter, Selector, and Generator modules can all be implemented using LLMs, they can be treated as RL agents [32], enabling parameter updates through reward signals. To optimize computational efficiency, these three modules can share the same LLM. Additionally, given the difficulty of modeling the Retriever module as an RI. agent, we use a fixed Retriever and regard it as part of the environment
The focus of the MMOA-RAG framework is the collaborative optimization of multiple modules to align their individual objectives with the ultimate goal of generating high-quality answers. We use metrics from the Generator's ss predicted answer Sntrprdicct such Fl score, as shared reward Rshored- Given the fully cooperative nature of the modules in the RAG system, Rahared can be used to train all agents, a common approach in existing MARL literature [35 55 4
Additionally, to ensure training stability and accelerate convergence in the multi- system, we design penalty terms PQR, Ps. and Pg for each agent. A more detailed explanation will be provided in Section 3.3
3.3 Detailed Configuration for Each Agent
In this section, we will provide a detailed explanation of each element in the tuple (g, O, R) mentioned in Section Here, G {Query Rewriter (QR), Selector (S), Generator (G)) represents all agents. In the following, we introduce the essential elements for each agent i E G: the observation information O E O, the action space A₁ E A, and the reward function R2
3.3.1 Elements of Query Rewriter
Observation of Query Rewriter is defined as Equation ( which contains prompt of Query Rewriter Promplon (as shown in Table 3 and the initial question q.
OQR = {
(Prompton.))
(1)
Action Space of Query Rewriter corresponds to the vocabulary of LLMs, denoted as V, as prompt the LLM to generate one or more sub-questions based on q.
AQR =
Reward Function of the Query Rewriter is defined as shown in Equation 3 Here, Rehared can be the metric for the final answer, depicted as the yellow section in Figure T In this paper, utilize the F1 score of the predicted answer, AnSprddic, as the shared reward. The term PoR serves as a penalty discourage the Query Rewriter from generating an excessive number of sub- sub-questions during training. Specifically, PQR is assigned a value of -0.5 if the number of sub-questions exceeds four, and it is set to 0 if the number of sub-questions is four or fewer.
ROR Ratared + PQR
3.3.2 Elements of Selector
Observation of Selector is defined as Equation (4) which contains prompt of Selector Prompts (as shown in Table 4 the initial question and the candidate documents set D with K documents.
Os {Prompls: q, D}
(4)
Action Space of Selector only comprises of several words as Equation (5). Since the function of the Selector is to output the IDs of candidate documents helpful to answering the initial question g, the action space is constrained to this limited set of words. This constraint can significantly reduce the exploration space of the Selector and provide more stable training process.
As = ("0", "1", "K-1", "Document", ","}
(5)
'Recent studies in generative IR (see (26 for a survey) have explored using generative models for retrieval. But we choose more traditional dense retrieval model [16] as the first-stage retriever and leave the optimization of the first-stage retriever for future work.

---

Reward Function of Selector also contains two terms, which are Rstured and Ps. And Ps is penalty term designed to prevent the Selector from generating duplicate document IDs and from outputting IDs that do not conform to the specified format (e.g., DocumentO, Document3, Selector outputs duplicate document IDs or fails to adhere to the specified format, Ps is set to -1; otherwise, Ps is set to O.
When the
Rg Rahared Ps
3.3.3 Elements of Generator
Observation of Generator is in Equation 7 containing prompt of Generator Prompte: (as shown in Table 5) the initial question q and the selected candidate documents set Dxelected given by Selector.
Og = [Prompta, q, D
Action Space of Generator Ag is the same as Query Rewriter.
AG AQR =
(8)
Reward Function of Generator contains Rehared and penalty term Pc. which is used to constrain the model from generating excessively long content. When the generated answer exceeds a certain length, Pg set to -0.5; otherwise, set to O. In fact, the values of each penalty P. (i E G) are mostly O, and they only become negative when the output does not meet the requirements.
Rg Rahared + Pg
3.4 Training Process of MMOA-RAG
3.4.1 Warm Start with SFT
In preparation for joint optimization of multiple modules using Multi-Agent PPO, it is essential to perform a warm start for each trainable module. The warm start enables the model to better adhere to instructions across diverse tasks and reduces the exploration space during MARL joint training, thereby enhancing the efficiency of exploration and exploitation.
Within the MMOA-RAG framework, there are three trainable modules: the Query Rewriter, the Selector, and the Generator. Consequently, we construct the training data for the SFT of each corresponding task and perform the SFT to get the warm-up checkpoints for each trainable modules. The details of constructing training data can be seen Appendix B
3.4.2 Multi-Agent Optimization
After undergoing SFT, the LLM demonstrates an improved ability to follow instructions while executing the functions of Query Rewriter, Selector, and Generator. The RAG system also achieves relatively satisfactory warm-start performance. To further enhance the performance of the RAG system, which modeled as a fully cooperative multi-agent system, it crucial to conduct joint training of multiple agents strengthen collaboration among them.
We adopt setup similar to Multi- Ageet PPO (55) in Starcraft II, where multiple agents share a global reward, optimizing g {QR, S. G} with Ratared- To reduce computational overhead, we apply the parameter-sharing mechanism among agents, allowing QR. S, and to utilize the same LLM.
In the multi-agent optimization process, there are three models to consider: the Actor model, the Critic model, and the SFT model. The parameters for these models are denoted 0, ø. and fSFT, respectively. The role of the Actor model to provide the response Answer, based on the observation O; for each agent i. The Critic model is responsible for estimating the state-value function Vir.t which is a classic setup in Actor-Crite architecture within RL algorithms. The SFT model serves as a baseline for the Actor model, similar InstructGPT (32). The objective update the parameters of both the Actor and Critic models. The overall loss function, L(0, Φ), consists of two terms: L (0) and (o):
L(0, x) = LActor (0) + Leaue
(10)
The Actor loss function presented in Equation (TT) similar to that used in the typical single-agent PPO (39] algorithm. The primary difference is that multiple agents are being optimized. In Equation
5

---

E G denotes the three agents: Query Rewriter, Selector, and Generator. The term ry in Equation (12) denotes the importance sampling ratio, which measures the difference between the new and old policies. The expression A&e in Equation 13) is the advantage function, estimated using Generalized Advantage Estimation (GAE) [8 The variable oj in Equation (14) is known as the
temporal difference (TD) error at time step t.
CActor (0)
ΣΣ min (rEA clip (ri, e, 1 + A::
(11)
To(a) s{)
(12)
Tom (a) si)
A Σ (12)) 6/+1
(13)
1=0
6: = R at) +
(14)
Similar to InstructGPT [32] the final reward function R(s at) is defined in Equation 15) The distinction is that our approach does not require a trained reward model, as we use the evaluation metric (FI score) of the predicted answers Areeeiddic of Generator as the shared reward Rahared for all
agents. The penalty term P; can also be easily obtained from the output of each agent, as introduced in Section 3.3 The components R; in Equation (15) are defined in Equations (3. (6) or (9. And Answer; represents the output generated by each agent based on its individual observation O,,
if K
R(s], at)
Ratured + P: -3 log
(Answer, (0.)swrrrr.)))))).)).0))))))))) Thahe (Answer, (0.)
ift=
(15)
R, and VEg
The loss function of the Critic model, as shown Equation (6)). employs a clipping operation similar to the Actor model. Here, AV,, = Vid Vangen where the cumulative return and s² is the state-value function.
Vo(st). The term Vinger represents
Cenui (Ø) = -Σ max (AVW)) (clip (vie, vélt dess + +e)
2 ]
target
(16)
The pseudocode for multi-agen optimization based on MAPPO is shown in Algorithm in Appendix C which corresponds to the overall framework of MMO depicted in Figure For specific question, the first step is to execute the Collect Rollout process. This process involves passing through the Query Rewriter, Retriever, Selector, and Generator, and the computed tuple T (OQn, subg, RQR)) (Os, IDs, Rs), (Oa, Anspredict, RG)) is stored in the replay buffer M. Next, the Policy and Value Optimization process is executed where the GAE is used to estimate the advantage function Å Subsequently, the overall loss function L(O, Φ) is calculated, and the parameters of both the Actor and Critic models are updated. Additionally. to accelerate the entire training process, we can run minibatch in parallel. Ultimately, we obtain a well-trained Actor model used for subsequent inference and evaluation.
4 Experiments
Our experiments mainly aim to explore the following research questions:
RQ1: How does our MMOA-RAC perform compared to existing RAG optimization methods?
RQ2: How does the joint optimization of individual modules in the RAG pipeline contribute to the effectiveness of MMOA-RAG framework?
RQ3: Can MMOA-RAG exhibit generalizability across different RAG systems?
4.1 Experimental Settings
Datasets and Evaluation We conducted experiments using MMOA-RAG alongside various baseline models across three open-domain QA datasets: HotpotQA 53) 2WikiMultihopQA [14, and Am- bigQA [1]. The candidate documents are all retrieved from Wikipedia passages for three datasets.

---

Table 1: Performance for different methods across datasets. All the results in this table obtained using Contriever [6]] as the retrieval model. In each dataset, the highest baseline value is underscored. The symbol A displays the improvement of MMOA-RAG over the best baseline.
Hotpo!QA EM
2WikiMuliihopOA
AmbigQA EM
Methods
Acc
F1
Acc
EM
Fl
Ace
F1
LLM wo RAG
25.08
21.31
31.18
27.78
23.68
29.47
27.21
20.96
33.42
Vanilla RAG wio train
27.99
20.62
30.67
31.94
13.91
22.84
31.09
22.42
33.56
Vanilla
44.49
39.47
44.36
SELF-RAG 0
30.42
27.77
38.93
36.32
35.39
38,86
28.35
25.70
39.04
RetRobust
[C
34,60
46.49
41.02
39.73
44.51
35.13
32.37
44.78
Rewrite-Retriove Read
30
38.03
33.93
46.32
40.40
39.17
44.17
35.94
31,90
45.92
BGM 22
36.05
32.76
44.54
39.61
38.61
43.29
36.01
32.53
45.76
RAG DDR
35.20
32.65
44.26
40.49
39.45
44,18
36.25
32.55
45.83
MMOA-RAG (ours)
36.15
42.73
41.52
48.59
A
+1.12
+1.55
+1.80
+1.71
+1.79
+1.89
+2.60
+2.20
+2.67
We employ three key evaluation metrics-Accuracc, Exact Match (EM), and F1 score to assess the performance of the RAG methods.
Implementation Details We utilize Contriever [16] the Retriever for most experiments. And the Selector consistently receives a fixed set of = 10 documents as input. Besides, we employ
-aama----B-Insttiee 7 as the foundational LLM for all the baselines and MMOA-RAG.
We compare MMOA-RAC with different baseline methods: LLM w/o RAG, Vanilla RAG w/o train, Vanilla RAG w SFT, SELF-RAG []] RetRobust [34], Cewwite-Retrreve-Reeee
30 BGM
22
RAG-DDR 27
The detailed introduction of experimental settings and baselines can be seen in Appendix D
4.2
Comparisons with Other Methods
We conducted comparative analysis of MMOA-RAG against multiple baselines, with the results presented in Table[] To ensure fairness in comparison, all methods utilized Llama-3-8B -nnstruttt as the backbone LLM, and all basclines were re-implemented according to the settings delineated in Appendix D.2
Firstly, as shown in Table [ MMOA-RAG demonstrates superior performance across all metrics and datasets, highlighting effectiveness. Additionally, is noteworthy that Vanilla RAG w/o train achieves comparable results to LLM w/o RAG across various metrics. This observation suggests that the pre-trained Llama-3-8B-Instnue struggles to effectively leverage external knowledge for answer generation, likely due to the absence of RAG-related tasks in its pre-training process, which limits its external knowledge utilization. In contrast, Vanilla RAG w SFT exhibits substantial improvements over Vanilla RAG w/o train across all evaluation metrics. This indicates that the SFT-enhanced Llama- 3-8B-Instruct is adept at utilizing external knowledge, successfully extracting valuable information from noisy candidate documents to enhance the quality of generated answers.
The
and BGM approaches enhance Vanilla RAG by respectively integrating a query rewrite module and a bridge module, each of which is trained using the PPO algorithm. As indicated in Table [ on the multi-hop datasets HotpotQA and 2WikiMultihopQA. Read surpasses BGM, suggesting that the inclusion of query rewrite module is more effective than adding bridge module for these multi-hop datasets. Conversely, on the single-hop dataset AmbigQA, the performance of Rewrite-Retrieve.
and BGM is relatively similar. Our MMOA-R AMOA-RACC AG can be conceptualized as augmenting Vanilla RAG by integrating both Query Rewriter and a Selector, whose roles are akin to the query rewrite module in Ceewite-Retrieve--e Read and the bridge module in BGM. The primary advantage of MMOA-RAS Rewriter, Selector, and Generator modules. This is achieved by aligning the objectives of these modules with the goal of generating higher-quality answers via MAPPO. The experimental results presented in Table further illustrate that MMOA-RAG significantly outperforms Read, BGM, and other baselines.
lies in its simultaneous optimization of the Query
In addition, we tested various methods based on other retriever, BGE 49 and E5 47 and the results can be seen in Table in Appendix [. And we also perform of experiments, the results can be seen in Table 8 in Appendix In summary, The results in Table [ Table 7 Table g and the
7

---

10:014 Bell
466
Vid
14.
144
Micic
Mance
Mantix
(a) HotpotQA
(b) 2WiiiuulthopQA
(c) AmbigQA
Figure Ablation about Optimizing Different Agents. In this figure, MMOA-RAC RAG w/o i (i {QR, S, G)) denote the variant where agent i is excluded from the complete optimization process of multi-agent joint optimization.
analysis in this Section jointly answer the RQ1. And the case study can be found in Appendix from which we can intuitively understand the advantages of multi-module joint optimization.
4.3 Ablation Experiments the Optimization of Different Agents
To demonstrate the necessity of multi-agent joint optimization in RAG systems, we present ablation experiments in this section. The MMOA- -AA framework, depicted in Figure L consists of three agents: E {Query Rewriter (QR), Selector (S), Generator (G)). In Figure 2 MMOA-RAS w/o denotes the variant where agent excluded from the complete optimization process of multi-agent joint optimization.
As illustrated in Figure the complete version of MMOA-RAG, where all three modules are jointly optimized, delivers the highest performance. This underscores the effectiveness multi-agent joint optimization within the RAG system and validates the importance of optimizing multiple modules concurrently. Additionally, the MMOA-RAC RAGG w/o S variant achieves the best performance among the three ablation configurations. The Selector's pri- mary function to refine the candidate document set D, yielding higher-quality subset Deelected which enhances the Generator's ability to produce superior answer AntSprdictt However, through the joint optimization by MAPPO, the Generator ac- quires some denoising capabilities. Consequently, satisfactory results can be achieved even when the Selector not optimized during joint optimization.
MHCA RAG W/S RAG MMGARAGG
Score
0.42
0.38
20000
40000
60000
Training Samples
Figure Ablation experiments on AmbigQA dataset. The horizontal axis represents the num- ber of training samples, while the vertical axis denotes the shared reward Rahared (FI score) dur- ing the training process.
We also present the trajectory of the shared reward Ratared during the training process based abla- tion experiments conducted on the AmbigQA dataset, as shown in Figure From Figure 3 is evident that the reward curve for MMOA-RAC demonstrates the fastest convergence rate and achieves the highest final convergence value. This underscores the effectiveness of joint optimization across multiple modules in significantly and efficiently enhancing the performance of the RAG system. Furthermore, the training curve for MMOA-RAN RAG w/o G in Figure 3iss noticeably slower compared to other algorithms, and the test results for MMOA-RAG w/o G the AmbigQA dataset, as shown in Figure 2 are the poorest. These findings suggest that the Generator module is the most critical component for the single-hop AmbigQA dataset.
The results in Figure 2 and Figure 3 answer the RQ.2 that it is more effective to optimize multiple modules in RAG system simultaneously.
4.4 Generality Experiments on RAG Systems with Varying Module Configurations
In this section, we evaluate the performance of MMOA-RAG optimizing RAG systems with different numbers of agents, as detailed in Table 2 In Table 2 QR+S+G represents the RAG

---

Table 2: Generality Experiments RAG Systems with Varying Module Configurations. In the second column, SFT and MAPPO refer to the current module configuration following the warm start training stage (Section 3.4.1) and the MAPPO joint training stage (Section 3.4.2), respectively. The symbol A signifies the enhancement achieved in the MAPPO stage relative to the SFT stage.
Training Stage
QA
2WikiMulthhpOAA
AmbigQA
Modules
Delta
Acc
EM
SFT MAPPO
36,00
33.04
44.69
39.54
38.50
42.97
36.55
32.60
46.71
QR+S+G
48.29
42.73
46.40
34.75
48.59
A
+3.15
+3.11
+3.60
+3.19
+3.02
+3.43
+2.30
+2.15
+1.88
SFT MAPPO
34.25
32.18
43.14
38.93
37.97
42.40
35.85
32.35
45,82
S+G
38.23
34.85
47.07
41.79
40.57
45.25
37.60
33,90
47.19
+3.98
+2.67
+3.93
+2.86
+2.60
+2.85
+1.75
+1.55
+1.37
SFT MAPPO
36.76
32.78
45,00
39.15
37,89
42.91
35,50
31,50
45.31
QR+G
38,90
35,89
47.94
42.43
41,01
46.19
37.65
33.50
47.53
+2.14
+3.11
+2,94
+3.28
+3.12
+3.28
+2.15
+2.00
+2.22
framework depicted in Figure illustrating multi-agent system composed of three agent, Query Rewriter, Selector and Generator. The configuration S+G results from omitting the Query Rewriter agent, relying solely on the initial question g for retrieval, thereby configuring the RAG system two-agent (Selector and Generator) system. Conversely, QR+G denotes the exclusion of the Selector agent, forming RAG pipeline consisting of two agents, Query Rewriter and Generator. The second column of Table 2 system through supervised fine-tuning, while MAPPO refers the joint optimization of all agents built upon SFT utilizing the MAPPO framework. The notation A is used to denote the performance enhancement achieved by MAPPO compared to SFT.
specifies that SFT refers to the warm start of all agents in the corresponding RAG
The experimental results in Table 2 reveal that RAG systems optimized using joint MAPPO con- sistently outperform those using only SFT across all datasets. This finding underscores the robust generalizability of the MMOA-RAG joint optimization approach, yiclding significant performance improvements across diverse RAG configurations. Notably. the performance gains from MAPPO over SFT are approximately three percentage points on multi-hop datasets such as HotpotQA and 2WikiMultihopAA, while improvements the single-hop dataset AmbigQA are around centage points. This difference may stem from the greater complexity inherent to multi-hop datasets, which potentially exacerbates misalignmen among different modules during the SFT stage. These results further highlight the necessity of multi-module joint optimization, especially in the context of more challenging multi-hop datasets.
per-
The results presented in Table 2 demonstrate the effectiveness of MMOA-RAG optimizing various RAG systems across different configurations, thereby answering RQ.3.
5 Conclusions and Future Works
In this paper, we model the RAG system as a multi-agent collaborative task, wherein we consider the Query Rewriter, Selector, and Generator modules as learnable RL agents. We employ a multi-agent reinforcement learning algorithm to jointly optimize these agents, aligning the optimization goals of multiple modules with the ultimate objective of generating high-quality answers.
Our experiments demonstrate the effectiveness of our modeling approach and joint optimization method. Comprehensive ablation studies confirm the necessity and generality of multi-module joint optimization, establishing MMOA-RAG as an effective approach for optimizing RAG systems.
In future works, we intend to explore the application of MMOA- RAG in more complex workflows. This exploration will include scenarios where the RAG workflows are organized as a directed acyelic graph, as well as situations involving dynamic workflows within agentic RAG. Additionally, it is important to assess the cost and latency associated with specific modules within the RAG system. In this regard, the design of the reward function should not be exclusively based on evaluation metrics, such as the score highlighted this paper. Instead, it should aim to strike a balance between effectiveness and cost in those partially cooperative RAG scenarios.

---

Acknowledgements
This research was supported by the Natural Science Foundation of China (61902209, 62377044), Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-Firs" Class" Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, the Research Funds of Renmin University of China (22XNKJ15), and Beijing Nova Program.
References
[1] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511 1511, 2023.
[2] Yiqun Chen, Qi Liu, Yi Zhang, Weiwei Sun, Daiting Shi, Jiaxin Mao, and Dawei Yin. Tourrank: Utilizing large language models for documents ranking with a tournament-nnpired strategy. arXiv preprint arXiv:2406. 16678, 2024.
[3] Yiqun Chen, Hangyu Mao, Jiaxin Mao, Shiguang Wu, Tianle Zhang, Bin Zhang, Wei Yang. and Hongxing Chang. Pide: Personalized training with distilled execution for multi-agent reinforcement learning. arXiv preprint arXiv:22 08872, 2022.
[4] Yiqun Chen, Hangyu Mao, Tianle Zhang, Shiguang Wu, Bin Zhang, Jianye Hao, Dong Li, Bin Wang, and Hongxing Chang. Ptde: Personalized training with distillated execution for multi-agent reinforcement learning. arXiv preprint arXiv:22 2022.
[5] Yiqun Chen, Erhan Zhang, Lingyong Yan, Shuaiqiang Wang, Jizhou Huang, Dawei Yin, and Jiaxin Mao. Mao-arag: Multi-agent orchestration for adaptive retrieval-augmenntr generation. arXiv preprint 10Xi::2508.0000:: 2025.
[6] Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719-729, 2024.
[7] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXi::2407.21783 2024.
[8] Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate you desire. arXiv preprint arXiv:2302.044/6 046666600066........ 2023.
[9] Jingsheng Gao, Linxu Li, Weiyuan Li, Yuzhuo Fu, and Bin Dai. Smartrag: Jointly learn rag-related tasks from the environment feedback. arXiv preprint 2r.iv:2410.18/1 2024.
[10] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmeene generation for large language models: A survey, arXiv preprint arXiv:2312. 10997, 2023.
[11] Peiyuan Gong and Jiaxin Mao. Coascore: Chain-of-aspects prompting for nlg evaluation. arXiv preprint arXiv:2312.10355, 2023.
[12] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning. pages 3929-399999 3938. PMLR, 2020.
[13] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model planning with world model. arXiv preprint ar05v: /4992, 2023.
[14] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060, 2020.
10

---

[15] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, arXiv preprint arXiv:2212.10403 2022.
[16] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprini arXiv:21 /2.09/ 18, 2021.
[17] Gautier Izacard and Edouard Grave. Distilling knowledge from reader retriever for question answering. arXiv preprini arXiv:2012.04584, 2020.
[18] Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422 446, 2002.
[19] Jinhao Jiang, Jiayi Chen, Junyi Li, Ruiyang Ren, Shijie Wang, Wayne Xin Zhao, Yang Song, and Tao Zhang. Rtg-star: Enhancing deliberative reasoning with retrieval augmented verification and refinement. arXiv preprint arXiv:2412.12881, 2024.
[20] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang. Jamie Callan, and Graham Neubig, Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 2023.
[21] Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Za- mani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516 2025.
[22] Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Bridging the preference gap between retrievers and llms. arXiv preprint vrXiv:2401.069)) 2024.
[23] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-senn seae-eearhhhhheee :predic::::i:::::: Composing retrieval and language models for knowledge-intensiee nlp. arXiv preprint arXiv:2212. 14024, 2022.
[24] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00388 2019.
[25] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
[26] Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou. From matching to generation: A survey on generative information retrieval. arXiv preprint arXiv:2404. 14851, 2024.
[27] Xinze Li, Sen Mei, Zhenghao Liu, Yukun Yan, Shuo Wang, Shi Yu, Zheni Zeng, Hao Chen, Ge Yu. Zhiyuan Liu, al. Rag-ddr: Optimizing retrieval-aus augmented generation using differen- tiable data rewards. arXiv preprint arXiv:2410.13500 2024.
[28] Yuchen Li, Hengyi Cai, Rui Kong. Xinran Chen, Jiamin Chen, Jun Yang, Haojie Zhang, Jiayi Li. Jiayi Wu, Yigun Chen, et al. Towards ai search paradigm. arXiv preprint arXiv:2506.17188, 2025.
[29] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiivee environments. Advances in neural information processing systems, 30, 2017.
[30] Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao and Nan Duan. Query rewriting for retrieval-augmeented large language models. arXiv preprint arXiv:2305. /4283, 2023.
[31] Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint 20X1v:2004.106455, 2020..
11

---

[32] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[34] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Leurning Reseurch, 21(178):1-51 2020.
[35] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning, pages 4295-4306
PMLR, 2018.
[36] Alireza Salemi and Hamed Zamani. Learning to rank for multiple eetrieval-augmenne models through iterative utility maximization. arXiv preprint wxXiv22400009442 2024.
[37] Alireza Salemi and Hamed Zamani. Towards a search engine for machines: Unified ranking for multiple retrieval retrieval-augmente -augmented large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 741-751, 2024.
[38] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 02438 2015.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[40] Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retreetal-augmmented large language models with iterative retrieval-generation synergy. arXiv preprint arXiv:2305. 2023.
[41] Zhengliang Shi, Weiwei Sun, Shen Gao, Pengjie Ren, Zhumin Chen, and Zhaochun Ren. Generate-then-gronte grroun in eetrival-auugenner generation for multi-hop question answering. arXiv preprint arXiv:2406. 14891, 2024.
[42] Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 34:25968-25999 2021.
[43] Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji- Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.0553 2025.
[44] Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. Dragin: Dynamic retrieval augmented generation based on the real-time information needs of large language models. arXiv preprint arXiv:2403. 10081, 2024.
[45] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. chatgpt good at search? investigating large language models as re- re-ranking agents. arXiv preprint --Xiv:2334.095555 2023.
[46] Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan Ö Ank. Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. arXiv preprint arXiv:2410.07 2024.
[47] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 212.:3221...3333,,200
12

---

[48] Zhepei Wei, Wei-Lin Chen, and Yu Meng. Instructrag: Instructing retrieval augmented gen- eration via self-synthesized rationales. In Adaptive Foundation Models: Evolving Al for Personalized and Efficient Learning.
[49] Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C- pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641-649, 2024.
[50] Shicheng Xu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. Search-in-the-chain: Interactively enhancing large language models with search for knowledge-intensive tasks, In Proceedings of the ACM on Web Conference 2024, pages 1362-1373, 2024.
[51] Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou. Unsupervised information refinement training large language models for retrieval auggeeted generation. arXiv preprint arXiv:2402.188155 2024.
[52] Wei Yang and Jesse Thomason. Learning to deliberate: Meta-policy collaboration for agentic Ilms with multi-agent reinforcement learning. arXiv preprint arXiv:2509.03811 2025.
[53] Zhilin Yang, Peng Qi. Saizheng Zhang. Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: /00,009000000000 2018
[54] Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making language models robust to irrelevant context. arXiv preprint arXiv:2310.0155. 2023.
[55] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:2461 1 246244 2022.
[56] Hamed Zamani and Michael Bendersky. Stochastic rag: End-to-end retrieval-augmented generation through expected utility maximization. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2641- 2646, 2024.
[57] Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, and Jie Tang. Longrag: A dual-perspective retrieval-augmeened generation paradigm for long-context question answering. arXiv preprint arXiv:2410. 18050, 2024.
[58] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
[59] Xinping Zhao, Dongfang Li, Yan Zhong, Boren Hu, Yibin Chen, Baotian Hu, and Min Zhang. Seer: Self-aligned evidence extraction for retrieval-augmented generation. arXiv preprint arXiv:2410. 11315, 2024.
[60] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403. 13372, 2024.
[61] Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, and Lei Sha. Atm: Adversarial tuning multi- agent system makes a robust 2024.
generator. arXiv preprint 2xXx:2405..11111
[62] Kun Zhu, Xiaocheng Feng, Xiyuan Du, Yuxuan Gu, Weijiang Yu, Haotian Wang, Qianglong Chen, Zheng Chu, Jingchang Chen, and Bing Qin. An information bottleneck perspective for effective noise filtering on retrieval-augmented generation. arXiv preprint arXiv: 544, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's ss contributions and scope?
Question:
Answer: [Yes]
Justification:
We claimed the main contribution and the scope in our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected generalize other settings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss some limitations of our proposed method in Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification:
We provide all the details and the anonymous code url in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may that access to the model is limited some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We provide all the details and the anonymous code url in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
Baallicgguides/Coodeetiiei siionooliyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results, See the NeurlPS code and data submission guidelines (https
ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide many details of the experiments and the key experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We don't 't do the statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar the standard deviation or the standard error of the mean.
16

---

It is OK to report 1 Issimaa error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of emors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We provide the computational resource requirements in anonymous code.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't ' make into the paper).
Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics
Question:
Answer: [Yes]
Justification: No ethics problem.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special considera- tion due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
This discussion is not necessary for work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, legitimate to point out that improvement in the quality of generative models could be used to generate
17

---

deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our paper meets this requirement.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssii thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer:
[NA]
Justification:
We don't 't release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/revieww based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, non- standard component of the core methods this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the methodology, scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: We only checked for syntax emrors using LLM.
Guidelines:
19

---

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htttgp/s//Buuuiie.ecc//ooforrrrs LLMM) for what should or should not be described.
20

---

Appendix
A Detailed Related Works
A.1 End-to-end Optimization in OpenQA
ORQA 24 is an open-domain QA system that learns end-to-end evidence retrieval and answer generation using only question-answer pairs, enabled by pretraining with Inverse Cloze Task. REALM [2] is an end-to-end optimizing framework that enhances language model pre-training with a retrieval-augmentedd approach. [25] introduces Retrieval-Angmented Generation RAG, a model that combines pre-trained language models with non-parametric memory for improved performance knowledge-intensivy NLP tasks. [17] propose knowledge distillation method to train retriever models using synthetic labels derived from reader model attention scores. Stochastic RAG [56] introduces a novel end-to-end optimization framework for RAG through expected utility maximization.
A.2 RAG without Parameters Updating
These methods typically involve designing novel RAG mechanism to enhance the performance of LLMs question answering tasks. For example, DSP 23 leverages sophisticated interactions between retrieval and language models to address knowledge-intensiin NLP tasks. FLARE [20]. an active retrieval augmented generation method, enhances text generation by dynamically retrieving relevant information throughout the process, showing superior performance across various long-form knowledge-intensee tasks. TTE-REETGEN 40) is iterative retrieval-generration synergy method that enhances and generation in an iterative manner. Search-in-the-Chaill [50] is framework that interactively enhances Large Language Models with search capabilities to improve performance on complex, mhowledge-intensiig tasks. SELF-RAG [ enhances language model quality and factuality through self-reflective retrieval and generation. DRAGIN [44] is a dynamic RAG framework that addresses the real-time information needs of LLMs during text generation, enhancing performance knowledge- intensive tasks. GenGround [41 synergizes large language model knowledge with external documents to enhance multi- -hop question answering through an iterative process of generating answers and grounding them in evidence. Astute RAG Ho is a approach that enhances the robustness of Retrieval- Augmented Generation for Large Language Models by adaptively integrating internal and external knowledge while resolving knowledge conflicts.
large language models by synergistically combining retrieval
A.3 RAG with Parameters Updating
A.3.1
Optimizing RAG with SFT
INFO-RAG [5]] an unsupervised training method, enhances the capacity of large language models integrate and refine information from retrieved texts. LongRAG [57] introduces a dual-perseective retrieval-auu -aggeeeed generation system to enhance understanding of complex long- ong-context knowledge for improved performance in long-contex question answering tasks. In INSTRUCTRAG [48). generation accuracy and trustworthiness are enhanced by explicitly denoising retrieved information through self-synthesized rationales, outperforming standard RAG approaches without additional supervision.
A.3.2 Optimizing RAG with RL
Some existing works use PPO (39] algorithm to fine-tune LLMs. In Rewrite-Retrievee framework 30 small language model trained with reinforcement learning rewrite queries for RAG.
BGM [22 proposes novel bridge mechanism between retrieval model and LLMs and uses PPO to optimize the parameters of the bridge filter for more helpful documents. SMARTRAG 9 optimizes an iterative RAG framework with reward, which includes decision maker and a policy network. RAG-Star [19 a reasoning approach that combines Monte Carlo Tree Search (MCTS) to improve the complex reasoning abilities of LLMss. Additionally, concurrent work Search-rl 21 and R1-Searcher 43 both use answer-hased reward to improve the reasoning RAG. MAO-ARAG proposes hicrarchical RAG framework that achieves balance between effectiveness and efficiency
21

---

in RAG by dynamically orchestrating executors through optimizing the planner agent with RL, thereby achieving Pareto optimality. SoftRankPO [52] introduces meta-cognitive multi- multi-agent reinforcement learning framework that enables LLM agents to dynamically deliberate, coordinate, and improve reasoning performance. Additionally, 28 propose an Al search paradigm which can be trained with RI. algorithms.
Some other works use DPO [33 similar alignment algorithms to optimize LLMs. A noise-filtering method [2] is proposed by optimizing mutual information between compressed data and output while minimizing with the retrieved passage. ATM [6]], an Adversarial Tuning Multi-agent system, enhances the robustness and performance of eetrieval-augmente
generators in question answering by iteratively tuning against an adversarial attacker agent to better discriminate useful documents and resist fabricated content. SEER [59] proposes novel self-aligned evidence extraction learning framework aimed at enhancing RAG performance by optimizing the extraction of high-quality, concise, and relevant evidence. RAG-DDR [27] optimizes RAG systems by aligning data preferences between modules through DDR, resulting in enhanced performance on knowledge-intensii tasks.
B How to Construct Training Data for the SFT Training Process
Before the multi-module joint training process of MAPPO, the warm-up to each trainable modules is necessary. The warm-up can make LLMSs to follow the instructions better and reduce the exploration space for the MAPPO training process to stabilize and accelerate the joint training process.
In this section, introduce the details of constructing training data for each modules.
B.1 Query Rewriter
Query Rewriter is to reformulate the initial query q, which may be too complex or ambiguous to resolve with single retrieval, into a of sub-questions denoted subg. The prompt of Query Rewriter in Table 3
In Rewrite-Retrieve- Read 30). small language model was trained using PPO to effectively rewrite queries for RAG. Building on this approach, we utilize the publicly available query rewriting data from Rewrite-Retrieve Retrieve-Read the SFT dataset warm start the Query Rewriter in MMOA-RAG.
Table 3: The prompt of Query Rewriter agent.
system: You professional assistant skilled at rewriting complex unclear questions into simpler, more searchable subquestions.
assistant: Okay, I will provide the rewritten sub-questions.
Please help me rewrite or decompose the given questions into sub-questions making them easier to search for answers in a search engine. The rewritten sub-questions must have logical connections and dependencies, without being overly repetitive in meaning. Additionally, avoid using vague demonstrative pronouns and similar terms.
user:
assistant: Okay, will provide the rewritten sub-questions.
Original question (content of Question). Now rewrite or decompose the original question into sub-questions according to the above requirements, and only output the rewritten subquestions in the format one subproblem per line without any additional content. Addi- tionally, avoid using vague demonstrative pronouns and similar terms, avoid the duplicate subquestions.
user:
22

---

B.2 Selector
The task of the Selector is to choose subset Deelected that are helpful for answering a question from a given set D with K candidate documents.
The output format of the Selector is the IDs of the documents in Dsetected (e.g.. DocumentO, Doc- ument4, Document6, Document7), as shown in the prompt of Selector in Table 4 Therefore, to construct SFT data for the Selector, the ground truth should be the IDs of documents that are truly useful for answering the question. One method to obtain the ground truth is to employ advanced LLMs, such as GPT-4o, to provide the ground truth. However, we have found that this approach does not yield results as good expected. Additionally, BGM [22) introduced and optimized bridge module which similar to the Selector module. They proposed a method called synthesis silver passage sequence (Synthesis SPS) to construct the ground truth for SFT data. However, the Synthesis SPS method requires examining each candidate document di, E D, (candidate documents of question i), invoking the LLM for each check, and comparing the utility values before and after the check, making a complex and costly method.
Table 4: The prompt of Selector agent.
system: You are helpful, respectful and honest assistant. Your task is to output the IDs of the candidate Documents (0.1.2....,K-I]] which are helpful in answering the Question.
assistant: Okay, will provide the ID of candidate Documents which are helpful in answering the Question.
user: Question: (content of Question)
DocumentO: (content of Document0) Documentl: {content of Documentl)
Document(K (-22)) (content of Document(K-2)
Document(K-)) (--1): (content of Document(K-I))
assistant: OK, received the Question and the candidate Documents.
user: Now, output the IDs of the candidate Documents (0,1,2,..,,K-I) which are helpful in answering the Question: (content Question), for example, in the following format: Docuumett,.Docummm66...
We propose a convenient heuristic approach for constructing SFT data, aimed at LLMs to effectively follow instructions and output in the desired format. As illustrated in Figure for given question q and its golden answer, there are K candidate documents denoted as dij. where E (0, 1, First, by removing certain insignificant stop words and punctuation marks from g, and its golden answer, and converting the words to lowercase, we obtain the set Set,,- Similarly, we perform the same operation on the K candidate documents d:jj to obtain Seta.,, Finally, any word from Set,, appears in Setd.,, the ID of corresponding document, j is included in the final output as the Label of SFT. With this approach, we can rapidly and cost-effectively construct the Selector's s ground truth labels during the SFT stage. Given our focus on the subsequent joint optimization of multiple modules, this straightforward data construction method can adequately meet our requirements.
K K-11 1}.
B.3 Generator
The Generator is responsible for producing the final answer, Arsppedditts based on the Dxelested provided by the Selector. Therefore, the ground truth for the SFT data Generator is the golden answer Ansgolden- And the prompt of Generator is in Table 5
23

---

Initial Question:
younger, rouslart lariko, dumitru. dediu
Seta
di
Who is younger Roustam
Tarike
Dumitru Dediu?
answer
Golden Answer
of q:
Roustam Tariko
Label SFT.
DocumentO 1
Doe 0: Dumitru Dediu (May
dumltin, dedia, may, 12, 1942, gala, 2013, july, commonaut, foree
d,o
12 19:2 in Gala 2013 July
Set
commonatt
Romanian Force
Doe
Roman Smishko
romen, smishio, retired. ukrainiam, professional foodballer played. goelkeeperr
d
retired Ukmannin professional footballer who played
Set
goalkeepeer
Doc K- Tarike
roustam, Larkko born. march, 1962, founder, mussian, stanclard, vodka
d,x-1 founder of Russian Standard Vodka .....
Iborn March 17. 1962 the
Setdig-1
Figure 4: The convenient approach to construct the SFT data for Selector.
Table The prompt of Generator agent.
system: You are a helpful, respectful and honest assistant. Your task is to predict the answer to the question based on the given documents. If you don't know the answer to a question, please don't share false information. Answer the question as accurately as possible.
assistant: Okay, I will provide the answer to the question based on the corresponding documents. Please provide the question and the corresponding documents.
user: Question: (content of Question)
Document0: [content of DocumentO) Document3: (content of Document3)
Document7:
(content Document7)
assistant: OK, received the Question and the corresponding Documents.
user: Given the Question and the corresponding Documents, predict the answer to the Question briefly and accurately possible based the Documents. Only give the brief and accurate answer with the form of **answer** and nothing else.
With these approaches, the SFT data used for the warm start training of these three modules Query Rewriter, Selector, and Generator-can be obtained. All modules can be fine-tuned using the typical loss function of SFT presented in Equation [7).
(SF
Σ log P(Y: X;;
(17)
In Equation (17 N represents the number of samples in the SFT dataset, while 0 denotes the parameters the L.L.M The variable X, corresponds to the input content of each module. Meanwhile, Y, signifies the output content of each modulc.
24

---

C The Pseudocode of Multi-Agent Training Process of MMOA-RAG
Algorithm 1 is the pseudocode for multi-agent optimization based on MAPPO.
Algorithm 1:
The Training Process of Multi-Agent Optimization
Initialize: The parameters of the Actor model 8, the Critic model Ø, the SFT model ÓSFT, and a replay buffer M 2.
Inputs: Dataset with initial questions q and corresponding golden answers Ans for epoch <- 1 to N _epoch do for batch <- to N j_batch do
// Collect Rollout for each question E batch do
// Query Rewriter (QR)
Construct observation Oqn according to Equation
Get sub uub-questions subq for initial question q
Calculate the penalty term of Query Rewriter PQR
// Retriever
Retrieve candidate documents to construct D
// Selector (S)
Construct observation Os according to Equation 4
Select Iss of helpful documents, and get Dselected Calculate the penalty term of Selector P's
// Generator (G)
Construct observation OG according to Equation Predict the AnSpredict initial question q Calculate the penalty term of Generator Pa // Getting Reward and Storing Tuple Calculate the F1 score of Ansppeedic as the shared reward Rehared
Get reward for each agent R,, E {QR, S, G), according Equation 1 6 and 9
Store tuple T = ((OqR, subq, RQR), (Os, Ds, Rs), (Oa, AnSpredicts RG) in the replay buffer M
// Policy and Value Optimization
for each question q E batch do
Compute the advantage function A using GAE
Calculate the loss of the Actor Actor and Critic model Cerie (ő)
Update the parameters of models through the overall loss function L(0, Φ) in
Equation (10)
Clear the replay buffer M to Ø
Output Well-trained Actor model with parameters Omained
D
The Detailed Introduction of the Implementation Details
D.I Baselines
The methods detailed below serve as baseline models:
LLM w/o RAG: This approach answers questions solely based on the internal knowledge embedded within LLMs, without employing any retrieval mechanisms.
Vanilla RAG w/o train: This method leverages a retrieval model to obtain relevant documents, thereby augmenting the LLM's internal knowledge with external sources. Here, the LLM remains in a pre-trained state and has not undergone any fine-tuning.
Vanilla RAG SFT: Building on the Vanilla RAG framework, this variant involves fine-tuned L.M to improve the integration of retrieved external knowledge with the LLM''s internal knowledge, potentially enhancing the quality of final answers.
25

---

This innovative framework advances LLM performance by incorporating both adaptive retrieval mechanisms and self-reflection processes, aiming to produce precise and dependable answers.
SELF-RAG [
RetRobust (54) This approach fortifies the RAG architecture against irrelevant contexts, thereby boosting its effectiveness in open-domain question-answent scenarios.
Rewrite- Retrieve-Read 30 A small-scale query rewriter model is trained using reinforcement learning, optimizing the interaction between retrieval and answer generation.
BGM [2]]:
Utilizing PPO, this method trains bridge component to filter and identify documents that are more likely be helpful, thus refining the quality of the retrieved context.
RAG-DDR 27 This approach utilize DPO to align data preferences between different modules, enhancing performance and reducing hallucinations in RAG.
D.2 Implementation Details
To ensure fairess in comparison, all baselines were re-implemented using Llama-3-8B SB-Instruct as the backbone architecture. Within these methods, the untuned modules employed the pre-trained version of Llama-3-8B-Instruct whereas the trainable modules were derived through specific SFT processes on Jlama---8-IInstuuce Notably, in the rewrite module, which trainable, was optimized using the PPO algorithm applied Llama-3- 8B-Instruct; meanwhile, answer generation was implemented based on the SFT-refined backbone. Regarding the BGM method, rebuilt the bridge to connect the retrieval model and the generation model, leveraging Llama-3-88 -aae--3---I-strueee Instuutruc,, with this bridge being trained using the PPO algorithm. The generation model for BGM was similarly obtained from the SFT-refined backbone.
Rereee-eeeriiieeeeeeeeee framework, the query
We utilize Contriever [6] as the Retriever. Regardless of how many sub-questions subq the Query Rewriter generates from the initial question q, the Selector consistently receives a fixed set of K = 10 documents as input. For example, the Query Rewriter yields 2 sub-questions, each sub-question is used for retrieval, with the top-5 documents from each retrieval being selected part of the candidate documents D for the Selector. Furthermore, it is important to emphasize that we do not utilize any support facts or positive passages that come with the official datasets to generate answers. Instead, we only use the K candidate documents from the retrieval model as external knowledge for answer generation.
Besides, we also employ Llama-3 3-8---Instruec 77 as the foundational LLM for the baselines and MMOA-RAG, Building the PPO code from LLama-Factory [60], we have developed MMOA- RAG, which optimizes the RAG multi-agent system using Multi-Agent PPO. And the critical hyperparameterss of MMOA-RAG are detailed in Table
Table 6: Key hyperparameters in the training process of MMOA-RAG.
Name
Explanation
Values
8maz
Maximum in Equation
15
0.2
Bmin
Minimum in Equation 15
0.06
Key hyperparameter in GAE
1.0
Key hyperparameter in GAE
0.95
Clip range in MAPPO
0.2
Coefficients in Equation 10
Maximum learning rate
2e-5
bueffer_size
Buffer size in MAPPO
128
1r ar_scheduler
I Learning rate scheduler
cosine
top-P
Sampling parameters in training
0.9
*Some QA datasets inherently include supportive texts that aid in answering questions. And some studies incorporate these supportive texts alongside retrieved candidate documents as input to LLMs for answer prediction, which significantly enhances answer quality. However, adhere the natural RAG process by using only candidate documents provided by retriever answer prediction, as annotated supportive texts are not present the natural RAG workflow.
https: / //gi thub. www./ooyoooooolll/lo.o wwwpr//uugggLLLLa----e Factory
26

---

Table 7: The performance of different methods based on Retriever BGE 147 and E5 449
Llama-
3-8B-Inste ISB-Instruct and its the fine-tuned version is the backbone for all methods. The highest baseline value in each dataset is underscored. The A displays the improvement of MMOA-RAG over the best baseline.
HotpotQA EM
2WikiMulthhoPAA EM
AmbigQA EM
Methods
Acc
F1
Acc
F1
Acc
F1
Retriever:
BGE [47
Vanilla RAG w/o train
37.72
28.99
41.62
28.86
18.54
27.87
48.05
35.10
52.31
Vanilla RAG
45.47
41.16
54.32
42,79
41.50
46.70
47.10
42.45
57.41
Self-RAG 1
33.41
30.59
42.12
37.39
36.41
40.91
30.00
27.00
40.77
RetRobust
S
55.77
Rewrite: Retrieve Read
10
45.58
40.95
54.35
43.91
42.48
47.83
46.30
41.75
57.19
BGM 22
41.49
54.73
41.88
40.67
45.53
48. 15
42.70
RAG DDR
45.13
41.51
54.32
42.49
41.21
46.00
48.50
42.95
58.71
MMOA-RAG
56.45
A
+0.56
+0.72
+0.68
+0.11
.0.10
+0.32
+0.15
+4.50
+0.39
Retriever:
ES (49)
Vanilla RAG w/o train
36.92
28.01
40.80
28.57
17.50
27.67
50.50
37,00
53.93
Self
1
33.66
30.84
42.36
38.39
37.38
42.03
30.10
27.20
40.83
RetRobust
49.27
Rewrite-Retriee Read
30
46,62
42,00
55.54
46.41
44.83
50.35
47.85
43.55
58.36
BGM 22
45.81
41.72
55,32
43.24
41.99
46.83
48.35
44.05
59.04
RAG DDR
27
44.50
41.09
53.96
43,89
42.49
47.29
49.60
43.75
59.75
MMOA-RAG
43.73
57.23
47.53
53.13
A
+0.80
+0.95
+1.38
+0.05
-0.06
+0.38
-0.05
+0.70
+..05
E The Performance of Different Methods Based on Retriever BGE and E5
We firstly perform the training of different methods based on retrieval model Contriever 16 and the results are shown in Table
We further test the trained models different retriever BGE 47 and E5 44). From Table see that our MMOA-RAG also surpasses most of baselines on each datasets.
Out-of-Domain Experiments
We also conducted out-of-domain (OOD) experiments evaluate the generalization capabilities of MMOA-RAG compared to the baselines. We trained LLM on HotpotQA dataset and evaluated on the AmbigQA dataset. The experimental results shown in Table
Table 8: Out-of-domain experimental results: The model is trained the HotpotQA dataset and tested on the AmbigQA dataset.
Methods
Acc
EM
F1
SE..F-RA
26.70
24.25
36.38
RetRobust
34.19
31.75
44.08
Retrrte--eeriieeeeee Read [30
33,91
30.73
43.61
BGM 22
32.58
29.77
42.07
MMOA-RAG (ours)
35.45
32.43
45.62
From Table 8, it is evident that MMOA-RAC demonstrates superior performance in the OOD experiments, underscoring its notable generalization capabilities and effectively. Additionally, noteworthy that RetRobust outperforms all other baselines. This can be attributed to its strategy of integrating both relevant and irrelevant data during the SFT process, which significantly enhances the robustness and generalization abilities of the RAG system.
27

---

G Discussion about Training Time and Inference Time
Training Time In our MMOA-RAG framework, multiple agents (modules) are optimized simul- taneously, which results in increased training time as the number of agents rises. When we have agents, the time required for rollouts and updating model parameters becomes n times that single agent. However, since this increase in training time is linear relative the number of agents, it does not lead to an excessively large overhead. Therefore, this is entirely manageable.
Inference Time The inference process of MMOA-RO ARAG follows the workflow outlined in Figure Training with multiple agents does not introduce any additional overhead during the inference phase.
H Reasons for Using Reward Sharing and Parameter Sharing Strategies
H.1 Reward sharing
It is through the way of reward sharing that the optimization objectives of multiple modules can be unified and aligned to optimize the quality of the final predicted answer. Multiple modules in MMOA-RAG naturally fit into fully cooperative relationship, and reward sharing is near-standard setting in fully cooperative MARL algorithms [34 29 55 31. as fully cooperative modeling only allows all agents have the same goal (here, Fl score). Therefore, reward sharing reasonable and necessary here.
H.2 Parameter sharing
Parameter sharing is also a common and effective strategy in multi-agent reinforcement learning 29 55 3]. Parameter sharing means that multiple agents share the parameters of the same policy network instead of maintaining a separate policy network for each agent. The biggest advantage of this approach is that it can reduce the computational and storage overhead: sharing parameters can significantly reduce the total number of parameters of the model, thus reducing the computational complexity and storage requirements. This is particularly important for RAG systems with multiple modules based on LLM, which is an important reason why we chose to use the parameter sharing strategy.
However, undeniable that parameter sharing does have its limitations: it can lead to instability when the number of agents increases. However, in framework, there three agents (modules) that are jointly optimized, and the problem of instability generally does not occur when the number of agents is small. The experimental results also prove that under the setting of parameter sharing, the functions of Query Rewriter, Selctor and Generator are not affected at all, and they can complete the task well and exceed the baselines without parameter sharing such RRR and BGM.
To summarize, we adopt the Settings of parameter sharing and reward sharing commonly used in previous MARL frameworks, which are common and versatile multi-agent reinforcement learning frameworks. Agents are distinguished by different prompts, which correspond to different obser- vations of different agents. Therefore, there doesn't 't have to be a distinction at the architectural or parameter level to be called different agents.
I
Case Study
In this section, we analyze the question from the HotpotQA dataset: "The Vermont Catamounts men's soccer team currently competes in conference that formerly known as what from 1988 to 1996?" The golden answer to this question is "the North Atlantic Conference." We compare two inference cases, Inference Case involves the response process of MMOA-RAC after MAPPO training, while Inference Case 2 pertains to the response process before MAPPO training, where only SFT was conducted:
In Inference Case 1, we observe that the LLM reformulates the initial question & into two sub-questions subg. In Candidate Documents D. Document 0 to 4 are retrieved based on Sub-question 1, and Documents
are retrieved based on Sub-question Notably, the answer highlighted in red appears in Document 5, which the Selector accurately identifies. The
28

---

final answer generated based on the Initial question and Document 5 is "North Atlantic Con- ference," " consistent with the Golden answer Andggeden- This process effectively demonstrates how, after MAPPO training, multiple modules can collaborate accurately to provide the most precise answer possible.
In contrast, Inference Case 2 shows that the LLM only reformulates the initial question & into single Sub-question 1, All 10 candidate documents are retrieved based on this Sub-question 1. Upon inspection, none of these documents contain text that can answer the initial question. The Selector also chooses multiple documents, which clearly does not aid in generating the correct answer. Consequently, the predicted answer preditt is the incorrect "Yankee Conference."" Hence, prior to MAPPO training, despite each module being well-optimized through SFT, there is lack of effective collaboration among the modules, potentially leading to inaccurate responses.
These examples provide an intuitive understanding of the advantages of joint multi-module optimiza- tion within the complex RAG framework.
Inference Case I: Training with MAPPO
Initial question q:
The Vermont Catamounts men's soccer team currently competes in conference that was formerly known what from 1988 to 1996?
Golden answer Anegeiies
the North Atlantic Confereace
Sub-questions subg:
Sub-question name the conference that the Vermont Catamounts men's soceer ream currently competes in Sub question 2: name of the conference from 1988 to 1996
Candidate Documents
Document 0:
Title: Vermont Catamounts foothall
Content: The Vermont Catamounts football program were the intercollegiate American football team for the University of Verment located Burlington, Vermont. The team competed in the NCAA Division and were members of the Yankee Conference. The school's lirst football lean lielded 1886. The fooiball program was discontinued at the conclusion of the 1974 season. Vermont fields team at the cluh foothall level, in conference that also the Yankee Conference name. The first intercollegiate game happened on November 6, 1886 between Dartmouth and Vermont in Burlington. Dartmouth won
of Vermont
Document I:
Titl: Vermont Catamounts men's ice hockey
Content: The Vermunt Catamounts men's ice hockey team is National Collegiate Athletic Association (NCAA) Division college ice hockey program represents University Vermont. The Catamounts are member of Hockey East, joining in 2005 after competing in ECAC Hockey from 1974-2005. They play home games at Gutterson Fieldhouse in Burlington, Vermont. Vermont has appeared in the NCAA Men's 's Hockey Division in 1974-75 including trips the Frozen Four
Championship five times since making the 1996 and 2009. Prior to moving to Division I, UVM competed in ECAC Division
Document 2:
Title: Vermont Catamounts foothall
Content: Members of the conference. Verment began Yankee Conference play in 1947 with Connecticut, Maine, Massachusetts, New Hampshire,
Rhode Island. Although they played UMass they didn't play Maine until 1950, Rhode Island until 1955, and UConn until 1966. Boston University began league play 1973, Notable alumai include: Vermont Catamounts football The Vermont Catamounts football program were the intercollegiate American football team for the University of Vermont located in Burlington. Vermont, The leam competed in the NCAA Division I and were members of the Yankee Conference. The school's first foothall team was fielded in
the first season,
Document
Title: Vermont Catamounts men's basketball Content: The Vermont Catamounts Basketball is the basketball team represents the University Vermont in Burlington. Vermont. The schooll' team currently competes in the America East Conference plays home games
Busketball Toumament six times, in 2003, 2004, 2005, 2010, 2012, and 2017. UVM famously upser Symcuse University in the first round of the 2005 loumameni, The Catamoonts are coached by John Becker. America East Canch the Year America East Player of the Year America Fast Defensive Player of the Year
Gym. team reached the NCAA Division
Document
Title: Vermont Calamounts
Content: The Vermont Catamounts the varsity intercollegiate athletic programs of University of Vernonnt hased in Burlington, Vermont, United States. The school sponsors 18 athletic prorams (8 men's 10 women's is), Conference (AEC), which school has member since 1979. The men's 's and women's ice bockey programs compete in Hockey Fast. The men's and women's alpine and nordic skiing teams compete in the Easter Intercollegiate Ski Association (EISA), The school's athletic director is Robert Corran. The Catamounts have won six national championships. all in skiing.
most which compete in
Division America
29

---

Docunent S:
Titl: America East Conference
Content: The America East Conference is a collegiate athletic conference affiliated with the NCAA Division Northeastern United States, specifically members include the public flgshhi universities of three stales, and one private university. The America East Conference was
whose members located mainly
England.
as the Fastern College Athletic Conference- Citrenee--oth,,,, a men haskerhall-onf athletic conference in 1979. The conference known the Easlem College Athletic Conference North from 1979 to 1988 and the North Atlantic Conference from 1988 1996. The charter members were the University Rhode Island, the College of
Document
Tille: Northwest Conferencee
Content: The Northwest Conference (NWC) is an athletic conference which competes in the NCAA's Division III. Member
localed in the states of Oregon and Washington. The NWC formed in 1926, making one of the oldest athletics conferences in the westem United States. For 60 years, the Narthwest Conference but 1984 joined with the Women's Conference Independent Colleges to becomee the Northwest Conference of Independent Colleges, shortening the name to its current joined NCAA. The charter members included Willamette University, Pacific
sponsored sports exclusively
moniker in 1996 University,
Document
Title: Big Conference
conference institutions participate in the National Collegiate Athletic Associaion's 's Division L. The conference was Coast Athletic Association (PCA in 1988 renamed the West Conference. The conference stopped sponsoring college lootball after the 2000
Content: Big Conference (BWC) is American collegiate
originally formed
The Big West Conference was formed June 1968 as the Pacific Coust Athletic Association. The five original charter members Fresno State, Jose State, Santa Barbara, Son Diego State, and Long Beach State. Two other schools, Cal State
Document 8:
Title: Big Fight Conference Men's Baskathall Tournaent
Content: The Big Eight Conference
NCAA athletic conference that existed from 1907 1996, when and the Southwest Conference dishanded to create the Big Conference. The post season conference and
conference's automatic Tournamen bid. From 1977 until 1985, the quatterfialls were played on the campus sites of the higher seeded teams.
loumament was instituted
toumammntt through conference's Missouri won the most Big Eight toumament titles with six. Colorado was the
Big Eight
senson in
Document
Title: Big Conference
Tournament
Basketball Tournament (popularly known Big Tournament) is the conference championship tournmennt in basketball for the Big South Conference. The toumament has held every year 1986. It elimiiaamnntooonn tournament
Content: The Big Conference
seeding hased regular records. The winner, declared conference champion. receives the conference's automatic bid the NCAA men's 's buskethall tournament. However, the conference did not have an automatic the NCAA toumament from 1986 1990, From 2003 through 2012, the tournament held mostly campus sites.
IDs of Selected Documents Diractee
Document
Predicted answer Ansppsdis:
North Atlantic Conference
Inference Case 2: Training with SFT
Initial question q:
Vermont Catamounts men's soccer team currently competes conference that was formerly known what from 1988 to 1996?
Golden answer Avtäggattee:
the North Atlantic Confereace
Sub-questions subg:
Sub-question name the conference that the Vermont Catamounts men's soceer team currently competes in
Candidate Documents
Document
Title: Vermont Catamounts
Content: Vermont Catamounts Football
Catamounts tootball program were intervollegiate
American football team for the University of Vermoni located in Burlington, Vermont. The team competed
team was fielded 1886. The lootball program discontinued at the conclusion of the 1974 season. Vermont fields leam the cluh foothall level, in a conference that also uses the Yankee Conference name. The first intercollegiate game the state Vermont happened November 1886 between Dartmouth Vermont in Burlington. Dartmouth won
NCAA Division
were members
Yankee Conference. The school's first
Document 1:
30

---

Title Vermont Calumounts men' 's ice hockey
Content: Vermont Catamounts men's lcc hockey The Vermont Catamounts men's 's ice bockey team is a National Collegiate Athletic Association (NCAA) Division college ice hockey program that represents the University of Vermant. The Catamounts are member of Hockey East, joining in 2005 after competing in ECAC Hockey from 1974-2005. They play home games Gutterson Fieldhouse in Burlington, Vermont. Vermont has appeared in the NCAA Men's Hockey Championship live times since making the move to Division in 1974-75 75 including trips to the Frozen Four 1996 2009. Prior moving to Division Division
competed in ECAC
Document 2:
Title Vermont Calamounts men's buskethall
Content: Vermont Catamounts men's basketball The Verment Catamounts Basketball team the basketball team that represents the Univensity of Verment in Burlington, Vermont. The school's 's' team currenily competes America East Conference and plays home games Patrick Gym. The team has reached the NCAA Division I Men's Basketball Toumament six times, in 2003, 2004, 2005, 2010, 2012, and 2017. UVM famously upset Syracuse University in the first round the 2005 toumament. The Catamounts are coached by John Recker. America Fast Crach of the Year America East Player of the Year America Fast Defensive Player of the Year
Document 3:
Title: Vermont Catamounts foothall
Content: members of the conference. Vermont began Yankee Conference play in 1947 with Connecticus, Maine, Massachusetis, New Hampshire, and Rhode Island. Although they played UMas and UNII in the first seuson, they didn't play Maine until 1950, Rhode Island until 1955, and UConn unril 1966. Boston University hegan leogue play in 1973. Notable alumai include: Vermont Catamounts football The Vermont Catamounts football program were intercollogiate American football team for the University of Vermont located in Burlington, Vermont. The team competed in the NCAA Division I and were members of the Yankee Conference. The school's first football team was fielded in
Document
Title: Vermont Catamounts
Content: Vermont Calamounts The Verment Catamounts are the varsity intercollegiate athletic programs of the University Vermont, based Burlington, Vermont, United States. The school sponsors atlletic programs (8 men's, 10 women's, ss, most of which compete in the NCAA Division I America Fast Conference (AFC), of which the school has been a member since 1979. The men's and women's 's iee hockey programs compete Hockey East. The men's and women's alpine and nordie skiing
compete in the Eastern Intercollegiate Ski Association (EISA). The school's atbletic director is Robert Corunn The Catanounts have won six national championships, in skiing.
Doeument
Tirle 2018-19 Vermont Catamounts men's 's baskethall team
Content: to win their conference touraament, the Catamounts received an automatic bid to the National Invitation Tournament where lost Middle Tennesser the First Round. 1colsaan= 12 style=l Exhibition Icolspan= 12 stylesl Non-conforenice regular season Icolspan=9 syleel America East Conference regular scason loolspan=12 12 style= America East Toumament Source 2018-19 Vermont Catamounts men's basketball team The 2018-19 Vermoni Calamounts men's basketball team will represent the University of Vermont in the 2018 NCAA Division
's baskethall season. They will play their home games Patrick Gym Burlington, Vermont and will be led by 8th-year head coach John Becker. The Catamounts
Document
Title:
Vermont
Content: based in Burlington. They named the Vermont Expos before 2006. Up until the 2011 scason, they were the affiliate of the Washington Nationals (Tormerly the Montreal Expos). Currenily the highest teams basketball, representing Vermont are the NCAA's Vermont Catamounts male female. The Vermont Frost Heaves, the 2007 and 2008 American Besketball Association national champions. were franchise of the Premier Basketball League, and were based in Barre and Burlington from the fall of 2006 through the winter 2011. The Vermont Bucks, indoor foothall team, based in Burlington and began play in 2017
Document
Title: 2017-18 1-17-18 Vermont Catamounts men's basketball team
Content: 2017-18 Vermont Catamounts men's 's baskethall team The 2017-18 Vermont Caramounts men's basketball team represented the University of Verment during the 2017-18 NCAA Division men's isss basketball season. The Catamounts, led by seventh-year head coach John Becker, played their home games at Patrick Gym Burlington, Vermont members of the America East Conference. They finished the season 27 8, 15- in America East play to win the America East regular season championship. The Catamounts defeated Maine Stony Bruck to advance
championship game the America East Tournament where they lost UMRC. As regular season conference champion
Document 8:
Title: Vermont Catamounts
Content: basketball team performed well America East Tournament. In 2013, it made conference- record 18th appearance in the semufinals. It has the most wins in tournament play, with 35 (2013 field.. It has advanced least one round
of 24 loumuments. The Catamounts were first women's buskeiball program to undefeated during the regular season in back ha-kk seasons (1991. 1992 and 1992. 1993), a feat matched only by Connecticut (2008-2009 and 2009-2010), In the 2012-2013 season, the program total attendance mark of 10,579. After the 2009 season, the Vermont bascball program, which played
Document
Title: 2018-19 Verment Catamounts men's basketball team
Content: 2018-19 Verment Catamounts basketball team will represent the University of Vermont in the 2018-19 NCAA Division I men's 's basketball season. They will play their home games at the Patrick Gym in Burlington, Vermont and will be led by
basketball team The 2018-19 Vermont Catamounts men's
31

---

Bihryeur bead couch John Becker. The Catamonnts linished the 2017-18 season 27-8, 15-1 in America East Conference play finish first place. In the America Fast Toumament, they defeated Maine and Stony Brook to advance to the championship game, where they lost to UMBC. As a regular scason conference champion
Dss Selected Documents Dulested
Document Document 2. Document 3. Document Document Document 6, Document 8, Document 9
Predicted unswer pusdii
Yankee Conference
32


Reviews:
[{"review_id": "BIM13bbaiy", "reviewer": "Reviewer_DCcd", "summary": "The authors propose MMOA-RAG: Multi-Module joint Optimization Algorithm for RAG, which employs multi-agent RL and formulate  the optimization process as a Cooperative Multi-Agent Reinforcement Learning (Co-MARL) to optimize the 4 sub-components (rewrite, retrieve, select, generate) and demonstrate the effectiveness of the approach on , HotpotQA, 2WikiMultihopQA, and AmbigQA, on Llama-3 8B. The proposed algorithm includes 2 components - a warm start SFT, and then Multi-Agent Optimization.", "questions": "1. From the workflow in figure 1 page 3, I wonder what was the thought process to isolate out \"selector\" with \"generator\", given both of them can be the same LLM - what is the motivation to separate them out? \n2. How did you arrive at 4 as the absolute threshold in \"PQR is assigned a value of -0.5 if the number of sub-questions exceeds \nfour, and it is set to 0 if the number of sub-questions is four or fewer\".. While 4 or N can be an arbitrary hyperparameter, what I am interested in is the choice to make it a step as opposed to a continuous loss? \n3. This statement in the README of the codebase \"Therefore, we deployed the retrieval model on a separate machine using Faiss and leveraged GPU acceleration to ensure fast retrieval results.\" is not clear to me. Are the authors mentioning that FAISS lookup is also GPU memory bound? I was under the impression in your system, the retrieval system stays on the host (NOT device) up until D_{selected}.", "limitations": "Yes, but the authors mention \"Justification: We discuss some limitations of our proposed method in Appendix G\"\nBut Appendix G reads \"G Discussion about Training Time and Inference Time\". I would encourage the authors to consider the system as a whole and think about the limitations from algorithmic, and system perspectives. Surely, the proposed algorithm cannot be with \"NO limitations\".", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[+] The authors propose clear techniques to modify each of the loss functions for each agent (Query re-writer, generator, selector, retriever). \n\n[+] The proposed system MMOA-RAG is simple to implement and can benefit RAG systems across the stack. \n\n[+] The paper is very well written\n\n[-] The evals are the weakest link of the paper. I.e., datasets like HotPotQA and others, which tend to be factual, usually are already baked into the pre-training of LLMs such as Llama - I would strongly encourage the authors to consider evaluating their technique on datasets that are post the model release. \n\n[-] To better understand the benefit of each components, For table 1, what would the table look like with just the SFT from \"3.4.1 Warm Start with SFT\" part of the training. Given, the Rewrite-Retrieve-Read dataset was used for training, it should be a significant contributor, perhaps?", "was_revised": false, "final_justification": "", "rebuttal": "First of all, thank you for your review. Here is our response:\n# Weakness 1\n[-] The evals are the weakest link of the paper. I.e., datasets like HotPotQA and others, which tend to be factual, usually are already baked into the pre-training of LLMs such as Llama - I would strongly encourage the authors to consider evaluating their technique on datasets that are post the model release.\n\n# Response to W1\n\nThanks for your suggestions, we conducted experiments on a QA dataset that is updated regularly, called **FreshQA** (version of **updated on July 21, 2025**) and a finance decision-making dataset, **LOFin**, **released in 2025**. These dates ensures that the FreshQA data and LOFin were not included in the pre-training data for llama3-8b-instruct. Our retrieval model uses E5, and **the corpus is the same Wikipedia dataset used in the main experiments of the paper**.\n\n| Method                | LOFin | Fresh Dataset |\n|-----------------------|----------------|----------------|\n| Vanilla RAG w SFT     | 12.23          | 17.78          |\n| Self-RAG              | 14.08          | 16.44          |\n| Rewrite-Retrieve-Read | 14.91          | 17.94          |\n| BGM                   | 10.41          | 17.97          |\n| MMOA-RAG              | **15.02**          | **18.82**          |\n\nFrom the table, we can see that MMOA-RAG still achieves the best performance compared to other baselines on both new datasets. This indicates that MMOA-RAG can still provide relatively the best results for questions that were not seen during the LLM pre-training phase. Since the Wikipedia data used in our main experiment is from the 2018 version (which is the version used in most RAG papers), the overall performance on FreshQA might be relatively low. However, if a more updated version of Wikipedia were used, the performance should likely improve further.\n\n# Weakness 2\n[-] To better understand the benefit of each components, For table 1, what would the table look like with just the SFT from \"3.4.1 Warm Start with SFT\" part of the training. Given, the Rewrite-Retrieve-Read dataset was used for training, it should be a significant contributor, perhaps?\n\n# Response to W2\n**We present the metrics after training with ‘3.4.1 Warm Start with SFT’ in the table below.**\n\n| **Training Stage & Delta** | **HotpotQA Acc** | **HotpotQA EM** | **HotpotQA F1** | **2WikiMultihopQA Acc** | **2WikiMultihopQA EM** | **2WikiMultihopQA F1** | **AmbigQA Acc** | **AmbigQA EM** | **AmbigQA F1** |\n|----------------------------|------------------|-----------------|-----------------|------------------------|------------------------|------------------------|-----------------|----------------|----------------|\n| SFT                        | 36.00           | 33.04           | 44.69           | 39.54                 | 38.50                  | 42.97                  | 36.55           | 32.60          | 46.71          |\n| MAPPO                          | 39.15           | 36.15           | 48.29           | 42.73                 | 41.52                  | 46.40                  | 38.85           | 34.75          | 48.59          |\n| &Delta;                       | **+3.15**       | **+3.11**       | **+3.60**       | **+3.19**             | **+3.02**              | **+3.43**              | **+2.30**       | **+2.15**      | **+1.88**      |\n\nFrom the above table, it can be observed that applying SFT to all modules already yields good results, and further improvement (&Delta;) is achieved after multi-module joint optimization with MAPPO.\n\nIn order to facilitate comparison, we put the performance of baselines in Table 1 of the original paper below for easy reference.\n\n| **Method**                      | **HotpotQA Acc** | **HotpotQA EM** | **HotpotQA F1** | **2WikiMultihopQA Acc** | **2WikiMultihopQA EM** | **2WikiMultihopQA F1** | **AmbigQA Acc** | **AmbigQA EM** | **AmbigQA F1** |\n|--------------------------------|------------------|-----------------|-----------------|------------------------|------------------------|------------------------|-----------------|----------------|----------------|\n| LLM w/o RAG                    | 25.08            | 21.31           | 31.18           | 27.78                  | 23.68                  | 29.47                  | 27.21           | 20.96          | 33.42          |\n| Vanilla RAG w/o train          | 27.99            | 20.62           | 30.67           | 31.94                  | 13.91                  | 22.84                  | 31.09           | 22.42          | 33.56          |\n| Vanilla RAG w SFT              | 36.18            | 32.30           | 44.49           | 39.47                  | 38.28                  | 43.36                  | 34.41           | 30.74          | 44.36          |\n| SELF-RAG                       | 30.42            | 27.77           | 38.93           | 36.32                  | 35.39                  | 38.86                  | 28.35           | 25.70          | 39.04          |\n| RetRobust                      | 37.69            | 34.60           | 46.49           | 41.02                  | 39.73                  | 44.51                  | 35.13           | 32.37          | 44.78          |\n| Rewrite-Retrieve-Read          | 38.03            | 33.93           | 46.32           | 40.40                  | 39.17                  | 44.17                  | 35.94           | 31.90          | 45.92          |\n| BGM                            | 36.05            | 32.76           | 44.54           | 39.61                  | 38.61                  | 43.29                  | 36.01           | 32.53          | 45.76          |\n| RAG-DDR                        | 35.20            | 32.65           | 44.26           | 40.49                  | 39.45                  | 44.18                  | 36.25           | 32.55          | 45.83          |\n\nIn addition, regarding the data from Rewrite-Retrieve-Read, we used the query rewrite data they released, which has been a significant contributor to our work.\n\n# Question 1\nFrom the workflow in figure 1 page 3, I wonder what was the thought process to isolate out \"selector\" with \"generator\", given both of them can be the same LLM - what is the motivation to separate them out?\n\n# Response to Q1\n**The motivation to separate them out**: During the RAG process, the introduction of external knowledge can easily lead to noise or conflicting information. Therefore, a \"selector\" is explicitly introduced to pick out useful text from the candidate documents and filter out useless and harmful information. This helps the generator produce more accurate answers. \n\nIn existing work, **BGM [1] and RAG-DDR [2] also have employed similar selector modules** within the RAG pipeline.\n\n[1] (**ACL 2024**) Bridging the Preference Gap between Retrievers and LLMs \n\n[2] (**ICLR 2025**) RAG-DDR: OPTIMIZING RETRIEVAL-AUGMENTED GENERATION USING DIFFERENTIABLE DATA REWARDS\n\n# Question 2\nHow did you arrive at 4 as the absolute threshold in \"PQR is assigned a value of -0.5 if the number of sub-questions exceeds four, and it is set to 0 if the number of sub-questions is four or fewer\".. While 4 or N can be an arbitrary hyperparameter, what I am interested in is the choice to make it a step as opposed to a continuous loss?\n\n# Response to Q2\nYou’re referring to a specific issue in RL training. During the RL training process, the model continuously explores new states and action spaces, which can sometimes lead the Query Rewriter to start rewriting the original question into multiple useless or repetitive sub-questions at a certain point in training. This can be detrimental to the overall training process, so we impose penalties for excessive sub-questions to **stabilize training.**\n\nIn most cases, this penalty is set to 0, and only when the number of sub-questions exceeds our set threshold (such as N=4) do we impose a penalty of -0.5. This ensures that the exploration by the Query Rewriter remains within a relatively controlled range. When the number of sub-questions is below this threshold, it can generally be considered as normal exploration by the Query Rewriter, so no penalty should be applied. **Therefore, we opted for a step penalty rather than a continuous one.**\n\n# Qustion 3\nThis statement in the README of the codebase \"Therefore, we deployed the retrieval model on a separate machine using Faiss and leveraged GPU acceleration to ensure fast retrieval results.\" is not clear to me. Are the authors mentioning that FAISS lookup is also GPU memory bound? I was under the impression in your system, the retrieval system stays on the host (NOT device) up until D_{selected}.\n\n# Response to Q3\nThank you for providing such detailed questions. Since our training process involves optimizing the Query Rewriter, the rollout process in RL requires real-time retrieval of new sub-questions.\n\nWe have deployed the retriever on another machine using FAISS specifically for retrieval purposes, and we have placed the corpus index on the GPU for acceleration (using faiss.index_cpu_to_gpu). We refer to this as **Machine 1**. Therefore, when performing rollouts on the machine used for RL training (referred to as **Machine 2**), when real-time retrieval is required for a given question, we call the retriever deployed on Machine 1 to quickly retrieve the top-k most relevant documents for the given question. **In other words, Machine 2 sends the question text to Machine 1, and the retriever deployed on Machine 1 retrieves the top-k most relevant documents and sends them back to Machine 2.**\n\n# Response to Limitations:\nThank you for your suggestions. I will update limitations, from algorithmic and system perspectives, in the next version of the paper. For example, I will focus on the following points: 1. Validation was not performed in the dynamic RAG system.  2. The design of the penalty term requires experience.\n\n\nHere is my complete response to your questions, and I hope it addresses your concerns. I look forward to your reply."}, {"review_id": "KKgcz9YVMb", "reviewer": "Reviewer_Vxtd", "summary": "This paper proposes MMOA-RAG, a multi-agent reinforcement learning (MARL) framework to optimize multiple modules within a Retrieval-Augmented Generation (RAG) pipeline—namely the Query Rewriter, Selector, and Generator—by modeling them as cooperating agents optimized using a shared reward (F1 score). The paper claims improvements over prior work by extending end-to-end optimization to more components and aligning their objectives via a unified reward using MAPPO.\nWhile the general motivation—addressing misalignment among modules in RAG—is sound and well-motivated, the core contributions and empirical validations raise several concerns regarding novelty, methodological clarity, and evaluation rigor.", "questions": "Have you considered training each agent with separate LLM heads (e.g., adapter tuning) to allow for more expressive role specialization?", "limitations": "see the weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Well-motivated problem: The paper tackles an important and known issue in RAG pipelines—disjoint training of components—and seeks to unify the optimization process.\n\n2. Systematic framework: The authors clearly formalize each RAG component as an agent and describe how shared rewards are used to encourage cooperation.\n\nWeaknesses\n\n1. Limited Novelty Beyond Extended Scope: While the authors position the work as an end-to-end optimization framework, this line of work has already been well explored in prior literature ([1], [2]). The only substantial difference here is the inclusion of additional modules (Query Rewriter and Selector) into the training loop. This extension, while practical, is relatively incremental, and the contribution is overstated as novel multi-agent collaboration.\n\n2. All “agents” share the same underlying LLM, differentiated only through prompts. There is no architectural or parameter-level separation between agents, which makes the claim of multi-agent learning weak. Without decoupled policies or distinct learning dynamics, the formulation seems closer to a multi-task learning setup with shared parameters and shared reward, rather than a genuine multi-agent RL framework.\n\n3. The ablation analysis is underdeveloped. The current results only test the impact of removing modules from joint optimization, which is insufficient to support the claim of agent-level collaboration. Specifically: There is no analysis on how each module performs when trained independently and then frozen. There is no comparison across different LLMs or analysis of whether shared vs. non-shared parameterization impacts performance.", "was_revised": true, "final_justification": "This is a solid piece of work. The authors have conducted numerous additional validation experiments during the rebuttal phase in response to the requests, which justifies a positive score.", "rebuttal": "First of all, thank you for your review. Here is our response:  \n\n# Weakness 1\n\nLimited Novelty Beyond Extended Scope: While the authors position the work as an end-to-end optimization framework, this line of work has already been well explored in prior literature ([1], [2]). The only substantial difference here is the inclusion of additional modules (Query Rewriter and Selector) into the training loop. This extension, while practical, is relatively incremental, and the contribution is overstated as novel multi-agent collaboration.  \n\n# Response 1 to W1\n\nFirst of all, there may be some problems. I did not see the \"prior literature ([1], [2])\" you mentioned in your review.\n\nFurthermore, we are extending end-to-end optimization framework to more modules (including Query Rewriter, Selector, Generator), which means that we align the optimization goals of multiple modules across the RAG pipeline with the ultimate unified goal of generating high-quality answers. This greatly alleviates the problem that the goals of multiple modules in RAG are inconsistent with the actual optimization goals, and our experimental results in Table 1 and Table 7 prove that our approach can indeed improve the effect of RAG. **This is our main contribution, which is not a simple extension of existing work.**\n\nAnd we model the entire RAG pipeline as a cooperative multi-agent reinforcement learning (Co-MARL) framework and optimize it with MAPPO algorithm. **Our modeling way and optimization method are never covered by previous works on RAG, so in essence our method is significantly different from the previous works.**\n\n# Weakness 2\n\nAll “agents” share the same underlying LLM, differentiated only through prompts. There is no architectural or parameter-level separation between agents, which makes the claim of multi-agent learning weak. Without decoupled policies or distinct learning dynamics, the formulation seems closer to a multi-task learning setup with shared parameters and shared reward, rather than a genuine multi-agent RL framework.\n\n# Reponse 2 to W2\n\n### Reward sharing:\n\nFor the reward sharing you mentioned, this is one of the core of this paper. It is through the way of reward sharing that the optimization objectives of multiple modules can be unified and aligned to optimize the quality of the final predicted answer. Multiple modules in MMOA-RAG naturally fit into a fully cooperative relationship, and **reward sharing is a near-standard setting in fully cooperative MARL algorithms [1,2,3,4], as fully cooperative modeling only allows all agents to have the same goal (here, F1 score). Therefore, reward sharing is reasonable and necessary here.**\n\n### Parameter sharing:\n\n**Parameter sharing is also a common [1,2,3,4] and effective strategy in multi-agent reinforcement learning.** Parameter sharing means that multiple agents share the parameters of the same policy network instead of maintaining a separate policy network for each agent. The biggest advantage of this approach is that it can reduce the computational and storage overhead: sharing parameters can significantly reduce the total number of parameters of the model, thus reducing the computational complexity and storage requirements. This is particularly important for RAG systems with multiple modules based on LLM, which is an important reason why we chose to use the parameter sharing strategy.\n\nHowever, it is undeniable that parameter sharing does have its limitations: it can lead to instability when the number of agents increases. However, in our framework, there are three agents (modules) that are jointly optimized, and the problem of instability generally does not occur when the number of agents is small. The experimental results also prove that under the setting of parameter sharing, the functions of Query Rewriter, Selctor and Generator are not affected at all, and they can complete the task well and exceed the baselines without parameter sharing such as RRR and BGM.\n\n**To summarize, we adopt the Settings of parameter sharing and reward sharing commonly used in previous MARL frameworks, which are common and versatile multi-agent reinforcement learning frameworks.** Agents are distinguished by different prompts, which correspond to different observations of different agents. **Therefore, there doesn't have to be a distinction at the architectural or parameter level to be called different agents.**\n\n[1] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning\n\n[2] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments\n\n[3] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games\n\n[4] PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning\n\n# Weakness 3\n\nThe ablation analysis is underdeveloped. The current results only test the impact of removing modules from joint optimization, which is insufficient to support the claim of agent-level collaboration. Specifically: There is no analysis on how each module performs when trained independently and then frozen. There is no comparison across different LLMs or analysis of whether shared vs. non-shared parameterization impacts performance.\n\n# Response 3 to W3\n\nThe current results only test the impact of removing modules from joint optimization\" you mentioned corresponds to Section 4.4, And **that's not the only kind of ablation experiments we did.**\n\nIn addition, the \"how each module performs when trained independently and then frozen\" mentioned by you is carried out **in Section 4.3, We did perform ablation experiments by freezing each module separately during MARL training**. It can be seen from Figure 2 that the effect of freezing a certain module is worse than jointly optimizing all modules, which proves the advantage and necessity of jointly optimizing all modules.\n\nAs for \"There is no comparison across different LLMs\" you mentioned, it is true that we only fine-tuned the model based on llama3-8b-instruct, which has some damage to prove the generality of our method. However, the experiments in Section 4.4 demonstrate the generality of our approach across different workflows. And the focus of our verification is whether the performance of the RAG system can be steadily improved after the joint optimization of MARL.\n\n**Analysis of whether shared vs. non-shared parameterization impacts performance**: In practice, parameter sharing is widely accepted in multi-agent reinforcement learning (MARL) because it significantly reduces the computational overhead associated with maintaining independent parameters, while incurring only a minimal loss in performance. Thus, the impact of non-shared parameters on performance is quite small (compared to shared parameters), and good performance can still be achieved with parameter sharing. Our experimental results were obtained by training on a machine with 8\\*A800 GPUs, whereas using a non-shared parameter approach would require 24\\*A800 GPUs to achieve the same batch size during training. Therefore, adopting parameter sharing is an ideal solution in LLM-based multi-module RAG scenarios. **And we will add the analysis of whether shared vs. non-shared parameterization impacts performance in our paper.**\n\n# Question 1\n\nHave you considered training each agent with separate LLM heads (e.g., adapter tuning) to allow for more expressive role specialization?\n\n# Response to Q1\n\nYes, thank you very much for your advice. I think fine-tuning for each agent via adapter tuning is really a great approach and I will try your suggestion in the future. But as argued earlier, this does not mean that full parameter sharing is a bad solution.\n\nHere is my complete response to your questions, and I hope it addresses your concerns. I look forward to your reply."}, {"review_id": "9RwbS70J5s", "reviewer": "Reviewer_xDLX", "summary": "This paper presents MMOA-RAG, a Multi-Module Joint Optimization Algorithm for Retrieval-Augmented Generation (RAG) systems. The authors model the RAG pipeline—consisting of a query rewriter, document selector, and answer generator—as a cooperative multi-agent reinforcement learning (Co-MARL) framework. Each module is treated as an agent optimized via Multi-Agent Proximal Policy Optimization (MAPPO), with a shared reward based on the final answer’s F1 score. The approach is evaluated on three open-domain QA datasets (HotpotQA, 2WikiMultihopQA, AmbigQA), demonstrating improved performance over several baselines.", "questions": "1. Can you clarify the distinct motivation for each experiment and what unique insight each offers?\n2. How sensitive is the final performance to the choice of these penalties? Are they critical for convergence or only auxiliary?", "limitations": "Consider mentioning compute cost, inference latency, or implications for real-time deployment of large RAG systems. Being transparent about such limitations will not be penalized and can improve the paper’s credibility.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. End-to-End Optimization: The method directly optimizes downstream answer quality, addressing the common misalignment between retriever and generator objectives.\n2. The formulation of RAG modules as cooperative agents within a MARL framework is conceptually neat and modular. The design of observations, actions, and rewards for each agent is well-specified and logically structured.\n\nWeakness:\n1. The core method is largely a direct adaptation of MAPPO to the RAG context, with minimal innovation tailored to the unique challenges of RAG systems.\n2. The related work is not sufficiently comprehensive and lacks a clear positioning of the proposed method relative to existing RAG optimization frameworks.\n3. Although the paper defines penalty terms to regularize agent behavior, their effectiveness is not empirically validated.\n4. All modules share the same underlying LLM parameters, but the paper lacks any discussion or experiment addressing potential issues like functional interference or catastrophic forgetting.\n5. The generalization (Section 4.3) and ablation (Section 4.4) experiments are overly similar, both aiming to show the benefits of joint optimization, leading to redundancy.", "was_revised": true, "final_justification": "Response to W1:\nThe authors reiterate their motivation but ultimately rely on a direct application of MAPPO, without any notable algorithmic adaptation to RAG-specific challenges. Modeling RAG modules as cooperative agents is conceptually reasonable but not novel, and without tailored techniques to handle the unique structure, dependencies, or reasoning patterns in RAG systems, the methodological contribution remains limited.\n\nResponse to W3:\nFormat violations during RL fine-tuning are a well-understood and solvable issue, especially with proper initialization from SFT checkpoints, as seen in models like DeepSeek. The reliance on a penalty term to maintain format integrity appears ad hoc and unnecessary, addressing a minor technical hurdle rather than a core challenge. The justification overstates the importance of this design choice.\n\nResponse to W4:\nThe claim that specialization eliminates the risk of forgetting overlooks the real concern: highly specialized agents likely require distinct parameter adaptations. Sharing a single LLM backbone across them introduces a clear risk of functional interference. The authors provide no empirical analysis or ablation to evaluate this, nor consider alternatives like partial sharing, which weakens the rationale behind their chosen setup.", "rebuttal": "First of all, thank you for your review. Here is our response:  \n\n# Weakness 1\n\nThe core method is largely a direct adaptation of MAPPO to the RAG context, with minimal innovation tailored to the unique challenges of RAG systems.\n\n# Response to W1\n\n**We have indeed addressed one of the unique challenges inherent in RAG systems.**\n\nLet us reiterate the research motivation of our paper (which is clearly stated in the Abstract): current RAG systems are largely modular, consisting of multiple modules. However, the optimization objectives of these modules are not aligned with the ultimate goal of generating high-quality answers. Therefore, we modeled each module within the RAG system as an agent and conceptualized the modular RAG system as a fully cooperative multi-agent scenario. We used the Multi-Agent RL algorithm MAPPO for optimization. Through this approach, we can unify and align the optimization objectives of all modules within the RAG system towards generating high-quality answers.\n\nThus, **the primary issue we address is the unification and alignment of optimization objectives across multiple modules within RAG systems.**\n\n# Weakness 2\n\nThe related work is not sufficiently comprehensive and lacks a clear positioning of the proposed method relative to existing RAG optimization frameworks.\n\n# Response to W2\n\nDue to space constraints in the main text, it is difficult for us to provide a comprehensive overview of related works in Section 2 \"Related Work\". We have mentioned in the end of Section 2 that “**More detailed related works can be seen in Appendix A.**”\n\nAppendix A provides a detailed explanation of the related works. We divided RAG into two major categories: RAG without Parameters Updating and RAG with Parameters Updating. The latter is further divided into Optimizing RAG with SFT and Optimizing RAG with RL. Since we use a MARL algorithm to optimize the RAG system, **our approach falls under the Optimizing RAG with RL category.**\n\n# Weakness 3\n\nAlthough the paper defines penalty terms to regularize agent behavior, their effectiveness is not empirically validated.\n\n# Response to W3\n\n**The purpose of designing a penalty term for each agent is to prevent any agent from producing outputs that do not meet the requirements during the MARL training process, thereby avoiding disruptions in the joint optimization of multiple modules.**\n\nInitially, we did not include a penalty term, and each agent’s reward was based solely on the F1 score of the final generated answer. Since RL agents need to continuously explore new state and action spaces, this exploration can lead to situations where an agent’s output does not conform to the required format, which in turn can halt the entire joint training process. However, after introducing the penalty term, when an agent explores outputs that do not meet the requirements, the penalty term can penalize such behavior, allowing the entire training process to continue running smoothly and stably.\n\nIn summary, **the inclusion of the penalty term transformed the multi-module MARL joint training system from being prone to failure to operating stably, which is the best evidence of the penalty term’s effectiveness.**\n\n**In addition, due to the rebuttal policy prohibiting the display of any images, I will describe the difference between using and not using penalty terms**: We conducted training on HotpotQA for a total of 600 steps. At around 300 steps, the test F1 score for both with and without penalty terms was approximately 0.475. However, around the 370-step mark, the Query Rewriter without penalty terms began outputting repeated sub-questions, which directly led to the collapse of multi-module training. At 400 steps, the test results showed that the F1 score with penalty terms was close to 0.48, whereas without penalty terms, due to the training collapse, the F1 score was below 0.2. This is the difference in using penalty terms. **We will include related experiments and analysis in future versions of the paper.**\n\n# Weakness 4\n\nAll modules share the same underlying LLM parameters, but the paper lacks any discussion or experiment addressing potential issues like functional interference or catastrophic forgetting.\n\n# Response to W4\n\n**Parameter sharing is a very common setting in cooperative multi-agent reinforcement learning [1,2,3,4]**. When the number of agents, N, is relatively small, it can ensure that the performance among agents is almost unaffected while significantly reducing the number of parameters and computational overhead (reducing the parameters from O(N) to O(1)). This is why we adopt parameter sharing. Moreover, the experimental results in Table 1 and Table 7 demonstrate that our method achieves better performance across multiple datasets, proving the effectiveness of parameter sharing and indicating that there is no interference between the functions of different agents.\n\nRegarding the issue of catastrophic forgetting: In the RAG scenario, the functions of different modules are highly customized. For example, the Query Rewriter only needs to focus on decomposing and rewriting the question, while the Generator only needs to provide accurate answers based on the candidate text. **Therefore, in a modular RAG system, the function of each agent needs to be highly specialized rather than comprehensive and general-purpose, making the issue of catastrophic forgetting inapplicable in a modular RAG scenario.**\n\n[1] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning\n\n[2] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments\n\n[3] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games\n\n[4] PTDE: Personalized Training with Distilled Execution for Multi-Agent Reinforcement Learning\n\n# Weakness 5\n\nThe generalization (Section 4.3) and ablation (Section 4.4) experiments are overly similar, both aiming to show the benefits of joint optimization, leading to redundancy.\n\n# Response to W5\n\nIn the paper, we explicitly states that **Section 4.3** and **Section 4.4** correspond to **RQ2** and **RQ3**, respectively:\n\n**RQ2**: How does the joint optimization of individual modules in the RAG pipeline contribute to the effectiveness of the MMOA-RAG framework? **RQ3**: Can MMOA-RAG exhibit generalizability across different RAG systems?\n\nSpecifically, **in Section 4.3**, the pipeline for each RAG method is the same, consisting of agents i in {Query Rewriter (QR), Retriever (R), Selector (S), and Generator (G)}. “MMOA-RAG w/o i” means freezing agent i during MARL joint optimization. Therefore, Section 4.3 investigates whether the performance declines when only two of the three modules are optimized during MARL joint optimization.\n\nIn contrast, **in Table 2 and Section 4.4**, the three pipelines are different. The QR+S+G pipeline consists of QR, R, S, and G. The S+G pipeline comprises R, S, and G, while the QR+G pipeline includes QR, R, and G. These represent three different modular RAG systems, and all LLM agents undergo MARL joint optimization. Here, we aim to study whether there is a significant improvement in the effectiveness of these differently composed RAG systems before and after MARL joint optimization. This would demonstrate that our MMOA-RAG approach can be applied to optimize various RAG systems (generalizability).\n\n**The research questions we aim to verify in Section 4.3 and Section 4.4 are different, so there is no issue of experimental redundancy.**\n\n# Question 1\n\nCan you clarify the distinct motivation for each experiment and what unique insight each offers?\n\n# Response to Q1\n\nPlease see the **Response to Weakness 5**.\n\n# Question 2\n\nHow sensitive is the final performance to the choice of these penalties? Are they critical for convergence or only auxiliary?\n\n# Response to Q2\n\nPlease see the **Response to Weakness 3**.\n\n# Limitations:\nConsider mentioning compute cost, inference latency, or implications for real-time deployment of large RAG systems. Being transparent about such limitations will not be penalized and can improve the paper’s credibility.\n\n# Response to Limitations\n**We have discussed issues regarding training and inference time in Appendix G.**\n\nAdditionally, regarding the limitations of the paper, **We will further consider the limitations of our algorithm from both the algorithmic and systems perspectives and update them in the next version of the paper.** For example, I will focus on the following points:\n(1) Our method’s pipeline is fixed, and we need to explore the effectiveness of multi-module joint optimization in a more general and dynamic RAG system.\n(2) The penalty term is a crucial component for maintaining training stability, but its design relies on experience and cannot fully rely on the F1 score to update the entire system.\n\nHere is our complete response to your questions, and I hope it addresses your concerns. I look forward to your reply."}, {"review_id": "XeL7CRx2mT", "reviewer": "Reviewer_zfzg", "summary": "In this paper the authors present a way of treating a RAG pipeline as a multi-agent collaborative task, where each component (query rewriting, document selection, answer generation) becomes a RL agent. Through multi-agent reinforcement learning, they are harmonised to operate toward the same goal (e.g. improving F1 score) and show its usefulness in a series of evaluations (examining also the benefit of combining all three components together).", "questions": "Are the penalties described in section 3.3 really needed? Eg. the one about not writing too many sub-questions (with the hardcoded value 4), what happens if they are not applied an have you tested other values?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I have overall enjoyed reading this paper and I found the idea of combining multi-agents in a cooperative task to perform RAG via reinforcement learning insightful. I would suggest the authors to give more details about the computational infrastructure used and the overall length/cost of training at the end of Section 3, as I believe readers would be interested to know more about that step.\n\nThe main weakness I have noticed is about the choice of evaluation, as I have found the datasets picked very limited to only QA settings for a method that should in theory be very impactful across many RAG applications. I would strongly recommend the authors to consider benchmarking their approach in other settings where RAG would play a role beyond QA, for instance decision-support (e.g. medical retrieval), summarisation, etc.", "was_revised": true, "final_justification": "The authors have addressed my main feedback showing how our method applies beyond Q&A to other domains", "rebuttal": "First of all, thank you for your review. Here is our response:\n# Weakness 1\nThe main weakness I have noticed is about the choice of evaluation, as I have found the datasets picked very limited to only QA settings for a method that should in theory be very impactful across many RAG applications. I would strongly recommend the authors to consider benchmarking their approach in other settings where RAG would play a role beyond QA, for instance decision-support (e.g. medical retrieval), summarisation, etc.\n\n# Response to W1\nDespite most existing RAG work being tested on QA datasets, your suggestion to test in other scenarios is valuable. \n\nTherefore, we tested the F1 score of different methods on the **LOFin**, **a financial decision-making dataset**. And we also test methods on **PubMed**, **a novel biomedical decision-making dataset**. Additionally, we conducted tests based on **a regularly updated dataset**, the **FreshQA**, which is updated weekly with new versions to incorporate potential new test data. \n\nAs seen in the table below, MMOA-RAG also demonstrates an advantage over baselines in decision-making and latest questions.\n\n| Method                | LOFin | PubMed | FreshQA |\n|-----------------------|----------------|---------|----------------|\n| Vanilla RAG w SFT     | 12.23          | 37.50    | 17.78          |\n| Self-RAG              | 14.08          | 50.5   | 16.44          |\n| Rewrite-Retrieve-Read | 14.91          | 38.00    | 17.94          |\n| BGM                   | 10.41          | 39.5   | 17.97          |\n| MMOA-RAG              | **15.02**          | **55.5**  | **18.82**          |\n\n# Question 1\nAre the penalties described in section 3.3 really needed? Eg. the one about not writing too many sub-questions (with the hardcoded value 4), what happens if they are not applied an have you tested other values?\n\n# Response to Q1:\n**Firstly, it is clear that penalty terms are necessary.**\n\n**Why design penalty terms**: During the RL training process, the agent continuously explores new state and action spaces during rollout and utilizes the obtained data. If an agent’s output becomes repetitive or deviates from the expected paradigm during exploration and exploitation, penalty terms are needed to constrain the agent’s exploration scope. This helps prevent an agent’s output from crashing, which could adversely affect the integrated optimization of the entire RAG system’s multiple modules.\n\n**What happens if penalty terms are removed**: Initially, in MARL training, no anomalies will be observed. However, after a certain step, an agent’s output might not meet expected requirements, potentially causing the entire training process to crash (e.g., a significant drop in reward). For instance, after a certain step, the Query Rewriter might output redundant and repetitive sub-questions, harming the training process. Similarly, after a particular step, the Selector might repeatedly output the IDs of candidate documents, thus losing its function of selecting useful documents. The Generator may, as exploration progresses, tend to output overly long and ambiguous answers. If any agent experiences a breakdown and is not penalized promptly, it can lead to a rapid decline in the overall performance of the RAG system.\n\n**Have I tested values other than 4**: Yes, I have tested other values. The purpose of setting this threshold is to prevent the Query Rewriter from outputting redundant or repetitive sub-questions. Through case studies, it is known that almost all questions can be decomposed into four or fewer sub-questions. Therefore, I tried N=4, 5, and 6. These values can all prevent the Query Rewriter from exhibiting crashing behavior during MARL training. In other words, the hyperparameters in these penalty terms are not sensitive, as long as they serve the purpose of constraining the agent’s exploration space to prevent it from going out of control.\n\nHere is our complete response to your questions, and I hope it addresses your concerns. I look forward to your reply."}]

Author rebuttal:

Meta-review: This paper proposes a multi-agent reinforcement learning (MARL) framework to optimize multiple modules, Query Rewriter, Selector, and Generator, in a Retrieval-Augmented Generation (RAG) pipeline. In specific, it treats each module as an RL agent and applies individual SFT for warm-up followed by Multi-Agent PPO (MAPPO) for cooperative MARL with a shared reward. Experimental results with LLaMA-3-8B show that the proposed multi-agent joint optimization improves RAG performances over the SFT baseline.

Overall, the proposed MARL for RAG seems to be technically sound with clear motivation. However, as the reviewers’ comment, the technical novelty is somewhat incremental in that it adopts the existing MARL algorithm without any specific modification. Therefore, the main contribution seems to come from the extensive empirical validation. In this respect, the current analysis and ablaltion studies in the current paper are insufficient as the reviewers concerns. However, most concerns are well addressed by the authors during the rebuttal phase including the effects of penalty rewards, shared parameters, and different tasks. Based on the consensus of all positive ratings between the reviewers, I would recommend the paper to be accepted.

However, further empirical validation is needed for the joint optimization with a unified reward (goal) in MARL. For example, it would be important to include a comparison against the case where each module is separately trained by RL with its own dedicated reward. In addition, experiments on various backbone LLMs would be necessary.
Decision: Accept (poster)