RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards
Authors: Xinze Li
Sen Mei
Zhenghao Liu
Yukun Yan
Shuo Wang
Shi Yu
Zheni Zeng
Hao Chen
Ge Yu
Zhiyuan Liu
Maosong Sun
Chenyan Xiong
Venue: ICLR 2025 Poster (2025)
Keywords: Retrieval-Augmented Generation
Primary area: generative models

Abstract: Retrieval-Augmented Generation (RAG) has proven its effectiveness in mitigating hallucinations in Large Language Models (LLMs) by retrieving knowledge from external resources. To adapt LLMs for the RAG systems, current approaches use instruction tuning to optimize LLMs, improving their ability to utilize retrieved knowledge. This supervised fine-tuning (SFT) approach focuses on equipping LLMs to handle diverse RAG tasks using different instructions. However, it trains RAG modules to overfit training signals and overlooks the varying data preferences among agents within the RAG system. In this paper, we propose a Differentiable Data Rewards (DDR) method, which end-to-end trains RAG systems by aligning data preferences between different RAG modules. DDR works by collecting the rewards to optimize each agent in the RAG system with the rollout method, which prompts agents to sample some potential responses as perturbations, evaluates the impact of these perturbations on the whole RAG system, and subsequently optimizes the agent to produce outputs that improve the performance of the RAG system. Our experiments on various knowledge-intensive tasks demonstrate that DDR significantly outperforms the SFT method, particularly for LLMs with smaller-scale parameters that depend more on the retrieved knowledge. Additionally, DDR exhibits a stronger capability to align the data preference between RAG modules. The DDR method makes the generation module more effective in extracting key information from documents and mitigating conflicts between parametric memory and external knowledge. All codes are available at https://github.com/OpenMatch/RAG-DDR.

Published as a conference paper at ICLR 2025
RAG-DDR: OPTIMIZING RETRIEVAL-AUGMENTER GENERATION USING DIFFERENTIABLE DATA RE- WARDS
Xinze Li Sen Mei
Zhenghao Liu
Yukun Yan
Shuo Wang"
Shi Yu
Zheni Zeng? Hao Chen
Ge Yuu
Zhiyuan Liu
Maosong Sun?
Chenyan Xiong
Northeastern University
Tsinggua University
CCrnegie Mellon University
ABSTRACT
Aatrieval-Augeente Generation (RAG) has proven its effectiveness in mitigating hallucinations in Large Language Models (LLMs) by retrieving knowledge from external resources. To adapt L.MMs for the RAG systems, current approaches instruction tuning to optimize L.Mss, improving their ability to utilize retrieved knowledge. This supervised fine-tuning (SFT) approach focuses on equipping LLMs to handle diverse RAG tasks using different instructions. However, it trains RAG modules to overfit training signals and overlooks the varying data prefer- ences among agents within the RAG system. In this paper, we propose Dif- ferentiable Data Rewards (DDR) method, which end-to-end trains RAG systems by aligning data preferences between different RAG modules. DDR works by collecting the rewards to optimize each agent in the RAG system with the roll- out method, which prompts agents to sample some potential responses as pertur- bations, evaluates the impact of these perturbations on the whole RAG system, and subsequently optimizes the agent produce outputs that improve the perfor- mance the RAG system. Our experiments on various knowledge-intensiin tasks demonstrate that DDR significantly outperforms the SFT method, particularly for LLMs with smaller-scale parameters that depend more on the retrieved knowl- edge. Additionally, DDR exhibits stronger capability align the data preference between RAG modules. The DDR method makes the generation module more effective extracting key information from documents and mitigating conflicts between parametric memory and external knowledge. All codes are available at cwt https://ggthbb..cmmm/eeaaaaaaa/t
INTRODUCTION
Large Language Models (LLMs) have demonstrated impressive capabilities in language under- standing, reasoning, and planning across wide range of natural language processing (NLP) tasks (Achiam al. 2023 Touvron et al. 2023 Hu al. 2024). However, LLMs usually gen-
erate incorrect responses due to hallucination (Ji et al. 2023 Xu et al. 2024c). To alleviate this
problem, existing studies employ Retrieval- Augmented Generation (RAG) (Lewis et al.) 2020) [hhi al. 2023 Peng et al.) (223) to enhance the ability of LLMs and help LLMs access long-tailed knowlede ledge and up-to-date knowledge from different data sources (riveddi et al. 2023 He et al.
2021
Cai et al. 2019 Parvez et al. 2021). However, the conflict between retrieved knowledge and
parametric memory usually misleads LLMs, challenging the effectiveness the RAG system
al. 2022
Chen et al. 2023 Asai et al. 2024).
To ensure the effectiveness of RAG systems, existing research has focused on developing various agents to obtain high-quality external knowledge (Gao al. 2024 Xu et al.] 2024a Jiang al.)
2023 Xu et 20224b), which can refine retrieval documents through query reformulation, reranking candidate documents, summarizing the retrieved documents, or performing additional actions to
obtain more relevant information for LLMs (Yan et al. 2024 Trivedi et al. 2023 Asai et al. 2023
Equal Contribution. Corresponding Authors.

---

Published as a conference paper at ICLR 2025
Yu et al. 2023a To further optimize the RAG system, some methods independently optimize
different RAG modules by using the EM method (Singh et al. 2021 Sachan et al. (021) or build the
instruction tuning dataset for Supervised Fine-Tuming (SFT) these LLMs-based RAG modules (Lin)
et al. 2023 Asai et al. 2023). However, these SFT-based methods usually make LLMS overfit the training signals and face the catastrophic forgetting problem (Luo et al. (023b).
Furthermore, current research aims to optimize RAG modules by aligning their data preferences and primarily focuses on optimizing a two-agent RAG system, which consists of retrieval and generation modules. Typically, these methods train only the retrieval module by using the preference signals from the generation module, making the retrieval module supply more accurate documents to satisfy
the data preference of the generation module (Shi et al.] [223]] Yu et al. (023b). However, these methods only optimize the retrieval module and overlook that the generation module still faces the
knowledge conflict (Xie et al., 2024), making the generation module's ' outputs do not align with the
data preferences of the RAG system. Thus, optimizing and aligning the data preferences of each
module in the RAG system is essential for building more tailored RAG system.
This paper introduces a Differentiable Data Rewards (DDR) method for end-to-end optimizing each agent in the RAG system using the DPO (Rafailov et al. 2024) method. DDR uses a roll- out method (Koesis & Szepesvári 2006) to collect the reward from the overall system for each agent and optimizes the agent according to the reward. Specifically, we follow follow[Asai et al. (2023) and build a typical RAG system to evaluate the effectiveness of DDR model. This RAG system consists of knowledge refinement module for selecting retrieved documents and generation module for producing responses based on the query and refined knowledge. Then we conduct the RAG-DDR model by optimizing the two-agent based RAG system using DDR. We use the reward from the en- tire RAG system and iteratively optimize both the generation and knowledge refinement modules to align data preferences across both agents, enabling the RAG system to generate accurate responses.
Our experiments on various Large Language Models (LLMs) demonstrate that Differentiable Data Rewards (DDR) outperforms all baseline models, achieving significant improvements over the pre- vious method (Lin et al. 2023) in range of LLMs for the RAG modeling and help LLMs generate higher-quality responses of an appropriate length. Our further analyses show that the effectiveness of our RAG-DDR model primarily derives from the generation module, which is optimized by the reward from the RAG system. The DDR- optimized generation module is more effective in capturing crucial information from retrieved doc- uments and alleviating the knowledge conflict between external knowledge and parametric memory. Further analyses show that the effectiveness of DoRooptimized RAG systems can be generalized even when additional noisy documents are incorporated during response generation.
tasks. DDR can effectively retrofit
RELATED WORK
Retrieval-Augmentt Generation (RAG) is widely used in various real-world tasks, such as open- domain QA (Trivedi et al., 2023), language modeling (He al. [02I), dialogue (Cai et al. 2019), and code generation (Parvez et al., 2021). RAG models retrieve documents from external cor- pus (Karpukhin et al., 2020 Xiong et al. (021) and then augment the LLM'' generation by incor- porating documents as the context of input (Ram et al. 2023) or aggregating the output probabilities from the encoding pass of each retrieved document Shi et al. 2023}. They help LLMs alleviate hallucinations and generate more accurate and trustworthy responses (Jiang et al. 2023 Xu et al.)
2023
Luo al. 2023a Hu al.
2023 Kandpal et al. 2023). However, retrieved documents
inevitably incorporate noisy information, limiting the effectiveness of RAG systems in generating
accurate responses (Xu et al. 2024a]
Longpre et al.) 2021: Liu et al. 2024b).
Some studies have demonstrated that the noise from retrieved documents can mislead LLMs, some- times resulting in degraded performance even on some knowledge-intenss inteee-ieteesivee tasks (Foulds et al., 2024 (Shuster et al. [202]] Xu et al.) 2024b). This issue primarily derives from the knowledge con- flict between the parametric knowledge of LLMs and external knowledge (Jin et al. 2024 Longpre al. 1 Chen et al., 2022 Xie et al. 2024 Wu et al.) 2024). Xie et al. (2024) have demon- strated that LLMs are highly receptive to external evidence when external knowledge conflicts with the parametric memory (Xie et al. 2024). Thus, lots of RAG models focus on building modular RAG pipelines to improve the quality of retrieved documents (Gao et al., 2024). Most of them aim to conduct more accurate retrieval models by employing retrieval evaluator to trigger different
2

---

Published as a conference paper at ICLR 2025
knowledge refinement actions (Yan al., 2024), prompting LLMs to summarize the query- query-related knowledge from retrieved documents (Yu et al. (023a) or training LLMs to learn how to retrieve and utilize knowledge on-demand by self-reflection (Asai al. 2023).
Optimizing the RAG system is a crucial research direction to help LLMs generate more accurate re- sponses. Previous work builds RAG system based on pretrained language models and conducts an end-to-end training method (Singh et al. [222 Sachan et al. 2021). They regard retrieval decisions as latent variables and then iteratively optimize the retriever and generator fit the golden answers. Recent research primarily focuses on optimizing the LLMs in RAG systems. INFO-RAG (Xu et al.) 2024a) focuses on enabling LLMs with the in-context denoising ability by designing an unsuper- vised pretraining method to teach LLMs to refine information from retrieved contexts. RA-DIT (Lin et al. 2023) builds a supervised training dataset and then optimizes the retriever and L.M by in- struction tuning. However, these training methods focus on training LLMS to fit the training signals and face the issue of catastrophic forgetting during instruction tuning (Luo et 2023b)
Reinforcement Learning (RL) algorithms (Schulman et al. 2017), such Direct Preference Opti- mization (DPO) (Rafailov et al. 2024), are widely used to optimize LLMs for aligning with human preferences and enhancing the consistency of generated responses (Putta et al. 2024). Agent Q inte- grates MCTS and DPO to allow agents to learn from both successful and unsuccessful trajectories, thereby improving their performance in complex reasoning tasks (Putta et al. 2024). STEP-DPO further considers optimizing each inference step of a complex task as the fundamental unit for pref- erence learning, which enhances the long-chain reasoning capabiliites of LLMs LLai et al. 2024). While these models primarily target the optimization of individual agents to improve response ac- curacy at each step, they do not focus on the effectiveness of data alignment within the multi-agent system. Instead of using SFT methods for RAG optimization (Lin et al. (023). this paper focuses on using the DPO method to avoid overfitting the training signals and aligning data preferences across different agents, which is different from the above RL-based optimization methods.
3 RAG TRAINING WITH DIFFERENTIABLE DATA REWARDS
This section introduces the Differentiable Data Rewards (DDR) method. We first introduce the DDR method, which optimizes the RAG system by aligning the data preferences between different agents (Sec. B.Ip. Then, we employ knowledge refinement and generation modules to build the RAG system and utilize the DDR method optimize the agents in this RAG system (Sec. 3.2)
3.1 DATA PREFERENCE LEARNING WITH DIFFERENTIABLE DATA REWARDS
In a RAG system V = ..,,..., V...., }r agents exchange and communicate data. To optimize this system, first forward propaaate data among agents and then evaluate the performance of the RAG system. Then we backward- propagate rewards to refine the data preferences of each agent.
nackward-propagate
Data Propagation. During communication, the t-th agent, Vi, acts both sender and a receiver. Agent V/2 receives data from agent V and simultaneously passes data to agent V4+1:
V12 V, "5 V411.. Vr-1 pT-2 Vi yT
where V. Mi, Vi+1 denotes that the agent generates one response y/ of the maximum prediction probability and sends it the agent V4+1. x and ~ yr represent sending the input æ to the RAG system V and getting the final output yr from V. The performance of the RAG system V can be evaluated by calculating the quality score S(yr) of the final output yr-
Differentiable Data Reward. Unlike conventional supervised fine- fine-tuning approaches (Lin et al., 2023). DDR tries to optimize the agent Vú to align its data preference with V4+1.7, making the RAG system produce a better response with a higher evaluation score S(vr).
To optimize the agent V,, DDR aims to propagate the system reward train the targeted agent. Specifically, we first instruct V; to sample multiple outputs St. which incorporate some perturbations into the RAG system. Then, we calculate the reward r(x, Ut) with rollout process (Koesis & Szepesvári 2006). In detail, we regard the agents V.-1.T the evaluation model, feed De to this subsystem 1+1. and calculate the evaluation score S(yT) of the final output yr:
0t V4+1 M1. V4+2 VT-1
VT
VT yr, r(z, 0c) S(yr).
3

---

Published as a conference paper at ICLR 2025
Retreeval-Augmented Generation (RAG)
RAG Training with Differentiable Data Rewards
Training Knowledge Refinement
q: who are the artists in the traveling willbryss
ykn: + q
d1 +
Ven' Tom Lynne
d2 0 9
d2 ④
Pkn YES
Yoen Jeff Lynne
Retriever
VIRR
dg 1
Refinement
dg +
ydu: YES
Gen Bob Griffin
D:[dy, dg3
Max P(YES NO1q. dz), P(NO YESIq. dg)
de: Traveling Wilburys were an English American supergroup consisting Bob Dylan, George Harrison, Jeff Lynne...
Reward
r(22 1 q.
1 g.y
G.y
d3: The group was organized by former Beatle George Harrison, former Electric Light Orchestra leader, Bob Griffin, Tom Lynne...
Training Generation
d1 +
Yka! NO
Sampling
d2 ④
yku: YES
d2 + d.
M Jeff Lynne
Yom The Traveling Wilburys consisted of George Harrison, Jeff Lynne, Bob Dylan, Roy Orbison, and Petty.
D D
dg ④
>da YES
2 Gen Bob Griffin
Max P(y Gen
Vtenl 4. (d2, dz))
Reward
Ground Truth:
leff Lynne
G(ycen)
r(d2 2 dg q. r
dg q. 9.en))
Voen
Evaluation
Figure 1: The Illustration of End-to-End Retrieval-Augmente Generation (RAG) Training with Our Differentiable Data Reward (DDR) Method. During training, we iteratively optimize the Generation module (Voea) and Knowledge Refinement module (VKR).
Finally, we maximize the probability of generating Øे over Dr. where øe wins higher reward than Dr (r D.") r(z, )):
P > (.(x)) (r((,, r(x, D.
where a is the Sigmoid function. The parameters of V, can be trained using the DPO (Rafailov et al. 2024) training loss, aiding the DDR model in identifying optimization directions by contrastively learning from the positive (3,,) and negative (j, outputs:
Vi x)) Vree x))
Vi x) Vrel
L(V:; Viret -E (a.st S1 D>Dgoog (3 log
3 log
x
where is a hyperparameter. D the dataset containing the input and its corresponding preference data pairs (Øit 07 Virl is the reference model, which is frozen during training.
3.2 OPTIMIZING A SPECIFIC RAG SYSTEM THROUGH DDR
Given a query and a set of retrieved documents D = (di,...,dy, we build a RAG system by employing knowledge refinement module (VxR) filter unrelated documents and generation module (Veen) to answer the question. These two modules can be represented by two-agent system:
{q. D} ~ VKR (g.D),
Vaen ~nt yGens
(5)
where δ C D and VKR produces the select actions to filter out the noise documents in D to build D. Voen generates the answer according to the query with filtered documents D. As shown in Figure [ we iteratively tune different modules to conduct the RAG-DDR model, beginning with the generation module optimization and subsequently focusing tuning the knowledge refinement module during training this RAG system. In the rest of this subsection, we will explain the details of how to optimize VER and Veen using DDR.
Knowledge Refinement Module. We follow Asai et al. (023) and build the knowledge refinement module VRR estimate the relevance of each document d, to the query q for refinement. We feed
4

---

Published as a conference paper at ICLR 2025
both query q and document de to the VKR and ask it to produce the action ykR ("YES", "NO"),
which indicates whether d, is retained (yKR = "YES") or discarded (yKR = "NO"):
ykR LI.MIInttructk d; @ q),
(6)
where @ denotes the concatenation operation, and InstructıR is a prompt designed for knowledge
refinement. Then the refined document collection δ = [di,..., d&} is constructed, where k <= n.
The document d, in D leading the RAG system to achieve the highest evaluation reward r(z, VKR "YES") is considered positive, while the document d, that results in the lowest reward r r(3r, VKR =
"YES") is regarded as negative. VKR is trained to maximize the probability P(VKR = "YES" > UdR "NO"lq, di) to retain the positive document and maximize the probability P(yKR "NO"> UKR "YES" q, dj) to filter out irrelevant documents.
Generation Module. After knowledge refinement, the query q and filtered documents δ {d1..... ] are fed to the generation module Vo-n The response Yeen is sampled from VGen:
MGen LLM(Instructean d1 + ... + d& + q).
where Instructgen is a prompt for generating a tailored response. To reduce the misleading knowl- edge from the retrieved documents, we also sample responses using only the query as input:
üGenn LLM (Instrruucct): 9)-
The response that achieves the highest evaluation score S(iGen) is considered positive (ite)) while the lowest evaluation score is considered negative (TGen). The generation module VGen optimized to maximize the probability of generating the positive response P(ū > VGenlg D) to win a higher reward. By generating responses jeGen based documents only the query, LLMs can learn to balance internal and external knowledge, alleviating the knowledge conflict problem.
4 EXPERIMENTAL METHODOLOGY
This section first describes datasets, evaluation metrics, and baselines. Then, we introduce the im- plementation details of our experiments. More experimental details are shown in Appendix A.1
Dataset. In our experiments, we follow RA-DIT (Lin et al., (023) and use the instruction tuning datasets for training and evaluating RAG models. For all datasets and all baselines, we use bge- large (Xiao et al. 2023) to retrieve documents from the MS MARCO 2.1 (Bajaj et al., 2016).
During the training of DDR, we collect ten datasets covering two tasks, open-domain QA and rea- soning. Specifically, we randomly sample 32,805 samples for the training set and 2,000 samples for the development set in our experiments. Following previous work (Lin et al. 22033 Xu et al. 2024a), we select the knowledge-intent intensive tasks for evaluation, including open-domain question answering, multi-hop question answering, slot filling, and dialogue tasks. The open-domain QA tasks consist of NQ (Kwiatkowski et al., 2019). MARCO QA (Bajaj et al., (016) and TriviaQA (Joshi et al. 2017). which require models to retrieve factual knowledge to help LLMs answer the given question. For more complex tasks, such as multi-hop QA and dialogue. we use HotpotQA dataset (Yang et al.) 2018 and Wikipedia of Wizard (WoW) (Dinan al. 2019) for evaluation. Besides, we also employ T-REx (Elsahar et al. 2018) to measure one-hop fact look-up abilities of models.
Evaluation. Following Xu al. (2024a), we utilize Rouge-L and Fl as evaluation metrics for the MARCO QA task and WoW task, respectively. For the rest tasks, we use Accuracy.
Baselines. In our experiments, we compare DDR with five baseline models, including zero-shot models and supervised finetuning models. We first treat the LLM as a black box and conduct three baselines, including L.LM w/o RAG, Vanilla RAG and REPLUG. For the L.M w/o RAG model, we directly feed the query to the LLM and ask it to produce the answer according to its memorized knowledge, To implement the vanilla RAG model, we follow previous work (Lin et al. 2023), use retrieved documents as context and leverage the in-context learning method to conduct the RAG modeling. REPLUG (Shi et al. 2023) is compared, which ensembles output probabilities from dif- ferent passage channels. Belf-RAG (Asai et al., 2023) is employed LLMs to retrieve documents on-demand and reflect on the retrieved documents, and generate re- sponses using reflection token. RA-DIT (Lin al. 2023) is also compared in our experiments,
baseline, which trains the
5

---

Published as a conference paper at ICLR 2025
Table 1: Overall Performance of Different RAG Models. The best and second best results highlighted. In our experiments, we employ Llama3-8B as the knowledge refinement module and utilize LLMs of varying scales (Llama3-8B and MiniCPM-2.44B as the generation module.
Open- QA
Multi-Hop QA
Slot Filling
Dialogue WoW
Method
NQ
TriviaQA
MARCO QA
HotpotQA
T-REx
MiniCPM-2.4B
LLM w/o RAG
20.1
45.0
17.1
17.7
22.6
14.9
Vanilla RAG 2003
42.2
79.5
16.7
26.7
22.1
14.4
REPLUG 2023
39.4
77.0
24.7
26.7
15.7
Self-RAG 2026)
28.8
48.6
15.2
23.9
RA-DIT 2026
41.8
78.6
19.6
25.2
15.7
RAG-DDR (w7 I-Round)
47.0
82.7
28.1
32.5
32.6
RAG-DDR (w/ 2-Round)
47.6
83.6
29.8
33.2
32.1
18.2
Llama3-8B
LLM w/o RAG
35.4
78.4
17.0
27.7
16.0
14.6
Vanilla RAG 2023
46.2
84.0
20.6
30.1
26.9
12.5
REPLUG 2025
45.1
83.1
22.5
29.4
23.1
14.0
Self-RAG 2023
39.6
78.2
12.5
24.3
29.5
14.2
RA-DIT (2023
87.4
20.3
34.9
41.7
RAG-DDK (w/ I-Round)
88.2
25.1
37.3
37.3
RAG-DDI LGGDDDR (w/ 2-Round)
52.1
89.6
27.3
39.0
40.6
16.8
which optimizes the RAG system using the instruct-tuning method. our experiments, we reim- plement REPLUG and RA-DIT baselines and don't finetune the retriever during reproduction process, as the retriever we use has already been trained with massive supervised data and suffi- ciently strong.
Implementation Details. our experiments, we employ Minicpm-2.. 1e4omm-2.BBsstt (Hu et al. 2024 and
Llama3 Alaama3-BBIing 8B-IIntructt (Touvron al. (2023) as backbone models to construct the generation modules and employ Llama3-88B-Instruue (Touvron er al. 2023) to build the knowledge refinement module. More training implementation details of RAG-DDR are presented in Appendix A.1
5 EVALUATION RESULTS
In this section, we first evaluate the performance of different RAG methods and then conduct abla- tion studies show the effectiveness of different training strategies. Then, examine the effec- tiveness of DDR training strategies on the generation module (VGen) and explore how balances internal and external knowledge through DDR. Finally, we present several case studies.
5.1 OVERALL PERFORMANCE
The performance of various RAG models is presented in Table As shown in the evaluation results, RAG-DDR significantly outperforms these baseline models on all datasets. achieves improve- of 7% compared to the Vanilla RAG model when using MiniCPM-2.4B and Llama3-8B to construct the generation module (VGen).
Compared with LLM w/o RAG, Vanilla RAG and REPLUG significantly enhance LLM perfor- mance on most inowledge-intensive tasks, indicating that external knowledge effectively improves the accuracy of generated responses. However, the performance of RAG models decreases on di- alogue tasks, showing that LLMs can also be misled by the retrieved documents, Unlike these zero-shot methods, RA-DIT provides a more effective approach for guiding L.LMs to filter out noise from retrieved content and identify accurate clues for answering questions. Nevertheless, RA-DIT still underperforms compared to Vanilla RAG on certain knowledge-intensive tasks, such as NQ and HotpotQA, showing that overfitting to golden answers is less effective for teaching LLMs to capture essential information for generating accurate responses. In contrast, RAG-DDR surpasses RA-DIT achieving 5% improvement. This highlights the generalization capability our DDR training method, enabling LLMs of varying scales to utilize external knowledge through in-context learning effectively.
almost all tasks, particularly with smaller LLMs (MiniCPM-2.4B3

---

Published as a conference paper at ICLR 2025
Ablation Study. Both Vanilla RAG (w/o VRR) and Vanilla RAG evaluated in zero-shot setting without any fine-tuning, We then use DDR to optimize the knowledge refinement module
Table 2:
(VKR), the generation module (Van). and both modules, resulting three models: RAG-DDR (Only
Training VKR). RAG-DDR (Only Training VGen) and RAG-DDR (All Training).
Open-Domain QA
Multi-Hop QA
Slot Filling
Dialogue
Method
NQ
TriviaQA
MARCO QA
HotpotQA
T-REX
WoW
MmniCPM-2.4h
Vanilla RAG (w/o ViK))
42.1
78.0
16.6
24.9
22.0
15.1
Vanilla RAG
42.2
79.5
16.7
26.7
22.1
14.4
RAG-DDR (Only Training ViRR)
42.5
79.6
16.8
27.3
21.9
15.8
RAG-DDR (Only Training Vae)
46.8
81.7
28.3
31.2
32.2
17.0
RAG-DDR (All Training)
47.0
82.7
28.1
32.5
32.6
17.2
Llama3-8B
Vanilla RAG (w/o ViR)
45.4
83.2
20.8
28.5
26.6
12.3
Vanilla RAG
46.2
84.0
20.6
30.1
26.9
12.5
RAG-DDR (Only Training Vicn)
46.8
84.7
20.7
30.7
28.5
12.5
RAG-DDR (Only Training Vas)
50.2
87.8
25.2
36.9
36.2
14.8
RAG-DDR (All Training)
50.7
88.2
25.1
37.3
37.3
14.9
5.2 ABLATION STUDIES
we conduct ablation studies to explore the role of different RAG modules and evaluate different training strategies using DDR.
As shown in Table 2
This experiment compares five models, utilizing MiniCPM-2.4B and Llama3-8B to construct the generation module. The Vanilla RAG (w/o VKR) relies solely on the generation module (VGen) to produce answers based on the query and retrieved documents. The Vanilla RAG adds knowledge refinement module (VKR) to filter the retrieved documents and then feeds query and filtered docu- ments to the generation module (VGe). RAG-DDR (Only Training VKR) indicates that we tune the Vanilla RAG model using DDR by only optimizing the knowledge refinement module (VKR). RAG- DDR (Only Training Viea) only optimizes the generation module (Voe)). RAG-DDR (All Training) optimizes both VKR and VGen modules.
Compared with the Vanilla RAG (w/o VKR). Vanilla RAG improves the performance on almost all evaluation tasks, demonstrating the effectiveness of the knowledge refinement module in improving the accuracy of LLM responses. In contrast, RAG-DDR (Only Training VGen) shows greater im- provements over than Vanilla RAG, indicating that the primary effectiveness of RAG-DDR comes from optimizing the generation module (Voen) through DDR. When begin with the RAG-DDR (Only Training VGea) model and subsequently optimize the knowledge refinement module, the per- formance further improved. It shows that filtering noise from retrieved documents using feedback from the generation module is effective, which is also observed in previous work Yuu et al. 2023b) Izacard & Grave 2020). However, the improvements from optimizing the knowledge refinement modules are limited, highlighting that enhancing the generation module's ability to leverage exter- nal knowledge is more critical for the existing RAG system.
5.3 CHARACTERISTIC, OF THE GENERATION MODULE IN RAG-DDR
In this experiment, we explore the characteristics of the generation module (VGen) by employing various training strategies, including zero-shot (Vanilla RAG), the SFT method (RA-DIT), and DDR (RAG-DDR). As illustrated in Figure 2 we present the performance of Vien w/o RAG and Veen w/ RAG. These experiments evaluate Voen's ability to memorize knowledge and utilize external knowledge. Additionally, we report the average length of responses generated by VGen module.
As shown in Figure 2(a) we compare the performance of the generation module that relies solely on internal knowledge of parametric memory. Compared to the Vanilla RAG model, RA-DIT demon- strates decline performance on the NQ and HotpotQA tasks. Such a phenomenon reveals that the model loses previously acquired knowledge while learning new information during SFT (Luo et al., 20236). contrast, DDR not only outperforms the RA-DIT method but also achieves con- sistent improvements
the Vanilla RAG model across all evaluation tasks. This indicates that DDR can help LLMs learn more factual knowledge during training while also preventing the loss of previously memorized information through reinforcement learning-base based training approach. Then

---

Published as a conference paper at ICLR 2025
70
NO
Vanilla RAG
NQ
Vanilla RA-DIT
60
RAG DDR
RA-DIT
50-
RAG-DDR
#Totens
48.5
30
40
32.4
wow
MARCO QA
Wow
MARCO
30
27.4
0
0
10
13.8
12.5
7.6
WoW
MARCO QA
NQ
HotaotQA
T-Rex
HotpotQA
Different Datasets
(a)
VGen w/o RAG.
(b) Vaea w/ RAG.
(c) Average Length of Responses Generated by Vaen-
Figure 2: Characteristics of the Generation Module in RAG Optimized with Different Training Strategies. We use MiniCPM-2.4B build the generation module (VGmn) and then train it using dif- ferent strategies. The performance of VGen is shown in a zero-shot setting, along with the generation module optimized using the RA-DIT and DDR methods.
Table 3: Experimental Results on Evaluating the Knowledge Usage Ability of the Generation Module (VGen) of Different RAG Models.
Ilswer
Miss-Answer nswer
Internal Knowledge
Method
NQ
HotpotQA
T-REx
NQ
HotpotQA
T-REx
NQ
HotpotQA
T-REx
MimCPMM2.44B
LLM
27.6
26.7
2.6
11.7
100.0
100.0
Vanilla RAG
39.1
51.7
36.8
[.1
99.1
2.5
7f1
70.
50.9
RA-DIT
58.3
47.6
41.7
1.7
10.8
4.2
76.9
73.7
73.4
RAG-DDR (w/ I-Round)
65,5
56.9
52.6
2,4
10.8
5.9
82.9
81.0
78.5
RAG-DDR (w/ 2-Round)
65.3
59,4
51.7
2.4
14.2
5.5
84.0
82.3
74.8
Llama3-8B
LLM
44.7
25.5
6.7
15.9
100.0
Vanilla RAG
64.2
58.0
43.5
2.9
10.5
3.0
80.0
67.3
66.3
RA-DIT
64.1
59.7
65.3
3.4
17.9
10.8
81.0
79.2
77.9
RAG-DDR (w/ I-Round)
69.5
64.3
59.9
4.1
18.0
5.4
88.4
82.0
79.2
RAG-DDR (w/ 2-Round)
71.6
66.2
65.4
5.6
18.9
7.7
89.3
83.9
89.7
we feed retrieved documents to the generation module and show the generation performance in Figure 2(b). The evaluation results indicate that RA-DIT marginally outperforms the Vanilla RAG model, while RAG-DDR significantly improves generation accuracy by utilizing factual knowledge from the retrieved documents. Additional experiments showing the general capabilities of RAG- DDR are presented in Appendix A.7 A.7
Finally, we show the average length responses generated by VGen in Figure 2(c) Compared the Vanilla RAG model, the average length of responses generated by RA-DIT decreases significantly, indicating that the SFT training method tends to cause LLMs to overfit the training dataset. On the contrary, RAG-DDR shows the model to generate responses more appropriate length. It demonstrates that training LLMS to learn data preferences from generated responses help align the output format of RAG models more closely with that of the original LLMs.
similar length distribution with the Vanilla RAG model, enabling
5.4 EFFECTIVENESS OF RAG-DDR IN USING EXTERNAL KNOWLEDGE
In this section, we investigate the capability of the generation module Vaen in the RAG model to leverage external knowledge for response generation. We first evaluate the ability of VGea to bal- ance the internal and external knowledge. Next, we evaluate the denoising ability by VGen feeding additional unrelated documents as the external knowledge.
As shown in Table we first show the effectiveness of Viee in balancing internal and external knowl- edge during producing responses. We compare three training strategies: zero-shot (Vanilla RAG), SFT (RA-DIT), and DDR (RAG-DDR). Our experiment establishes three testing scenarios to evalu- ate the effectiveness of different RAG models by categorizing the evaluation data into three distinct
8

---

Published as a conference paper at ICLR 2025
89
87
55
Accuracy
60
Accuracy
54
85
55
0 RAG-DDR
49
8 RA-DIT
40
Vanilla
44
1 Noise Documents
3 Noise Documents
1 2 Documents
Noise Documents
NQ.
(b) HotpotQA.
(c) WoW.
(d) T-REx.
Figure 3: Effectiveness of Different RAG Models in Defending Noisy Information. We use MiniCPM-2.44 4B to build the generation module (Vae). Then we retain one informative document and randomly replace n top-retrieved documents with noisy ones.
scenarios: Has-Answer, AisweA and Internal Knowledge. The Has-Answer scenario indicates that the retrieved documents contain the golden answer, which help the generation module to answer the question accurately. The Miss-Answer scenario indicates that the retrieved documents do not contain the golden answer and fail provide sufficient support for LLMs generate accurate responses. Lastly, the Internal Knowledge scenario further evaluates the ability of LLMMs in dealing with the conflict between internal and external knowledge. The test cases of the Internal Knowledge scenario indicate that LLMs can generate accurate answers using only parametric memory, whereas RAG models may produce incorrect responses.
In the Has-Answer scenario, RAG-DDR and RA-DIT outperform the Vanilla RAG model on all datasets. This indicates that training the generation module enables it capture essential knowledge facts, improving the accuracy of the generated responses. Compared with RA-DIT, RAG-DDR achieves consistent improvements over the Vanilla RAG model, which shows that DDR can better generalize the external knowledge usage ability to different tasks. In the Miss-Answer scenario, all RAG models perform significantly worse compared to the Has-Answer scenario, showing that the retrieved knowledge fails to provide sufficient information for generating accurate responses. Compared to the Vanilla RAG, RAG-DDR and RA-DIT mitigate the performance drop caused by incorporating retrieved documents by fine-tuning the generation module, illustrating their ability to effectively leverage internal knowledge and avoid being misled by irrelevant information. For the Internal Knowledge scenario, we evaluate the ability of RAG models in handling the conflict from external knowledge. The Vanilla RAG model decreases the generation accuracy more than 20% on all tasks, showing that the knowledge conflict can significantly affect the generation result. DDR cxhibits strong effectiveness in mitigating the knowledge conflict in the Vanilla RAG model, resulting in reduction of more than 10% in performance drop. This indicates that DDR effectively balances the internal and external knowledge, facilitating the robustness of the RAG model.
In the second experiment, we extend the Has-Answer setting and further investigate the denoising effectiveness of VGen by adding the different number of noise documents. As shown in Figure B we increase the noise by randomly replacing n documents from the top-5 retrieved set with the last documents from the 10p-100 retrieved candidates, while ensuring that the ground truth document remains in the set. RA-DIT exhibits inconsistent performance and degrades in some RAG scenarios. In contrast, RAG-DDR consistently outperforms the Vanilla RAG model, maintaining consistent improvement even as the number of noisy documents increases. It further confirms the effectiveness of our RAG-DDR approach in defending noise.
5.5 CASE STUDIES
In Table 4 we present three randomly selected examples from the NQ, HotpotQA, and TriviaQA datasets to show the generated responses and evaluate the effectiveness of the RAG-DDR model.
For the first case, the query asks about the "daylight savings time of Canada" and the retrieved documents contain detailed information about when the daylight savings time begins each year in Canada. However, since the exact date of daylight savings time changes annually, the most accurate answer is "the second Sunday in March", RAG-DDR shows its effectiveness in accurately answer-

---

Published as a conference paper at ICLR 2025
Table 4: Case Studies. These matching phrases are indicated in Red if the model answers correctly and in Blue if it answers incorrectly. We present the generated responses of different RAG models, which use MiniCPM-2.44 build the generation module (VGen).
Case I: When does Canada switch to daylight savings time
Document: Daylight Saving Time Canada. Daylight Saving Time in Canada. In most of Canada Daylight Saving Time begins at 2:00 a.m. local time on the second Sunday in March. Canada Daylight Saving Time Start and End Dates 2020 March 8 November 1 2019 March 10 November ... November 2010 March 14 November 2009 March 08 November
Vanilla LLMs: Canada does not switch to or from daylight savings time. Instead, the clocks are automatically set forward by one hour during the spring.
Vanilla RAG: March 14 RA-DIT: March 8
RAG-DDR: Canada switches Daylight Saving Time the second Sunday of March and returns to
Standard Time on the first Sunday November.
Case 2: The manager who recruited David Beckham managed Manchester United during what timetrame
Document: Ferguson's unprecedented achievements at Aberdeen led to managerial offers from some of the most prestigious clubs in Europe over the years, and he signed with Manchester in November 1986... Ferguson retired the end of the 2012-13 Premier League season but stayed on with Man a...
Vanilla LLMs: 1992 2003 Vanilla RAG: 1986-1993
RA-DIT: 1992 to 2013
RAG-DDR: The football manager who recruited David Beckham was Sir Alex Ferguson, and he managed Manchester United from 1986 to 2013.
Case 3: Following success at 1979 election whose party conference speeech included the lines "you turn you want to, the lady's 's not for turning"? ?
Document: The lady's 's not for turning- Wikipedia The lady's not for turning From Wikipedia, the free encyclopedia Jump to navigation Jump to search 1980 Margaret Thatcher speech ... phrase used by Margaret Thatcher, then Prime Minister, in her speech to the Conservative Party Conference...
Vanilla LLMs: The speaker in question is Margaret Thatcher, who was the leader of the Conservative Party and later became the Prime Minister of the United Kingdom. Vanilla RAG: Conservative Party Conference RA-DIT: The Conservative Party
RAG-DDR: The 1979 Conservative Party conference speech by Margaret Thatcher included the lines "you turn if you want to, the lady's not for turning".
ing the question, while both Vanilla RAG and RA-DIT are misled by noisy information such as "14th March" and "10th March", leading to incorrect responses. This demonstrates that RAG-DDR model has the ability to distinguish the most accurate knowledge from ambiguous misleading in- formation in the retrieved documents. In the second case, the model must integrate multiple pieces of knowledge from the provided documents to answer the question. While the Vanilla RAG model and RAG-DIT only correctly answer half of the questions using partial knowledge, RAG-DDR suc- cessfully identifies the correct start time and end time. This indicates that RAG-DDR has stronger capacity to integrate factual knowledge from different document segments. As shown in the third case, Vanilla L.M can answer the question correctly only depending the parametric memory. Nevertheless, both Vanilla RAG and RA-DIT are misled by the confusing information from these retrieved documents and generate the response "the Conservative Party Conference", which is en- tirely unrelated to the given question. In contrast, RAG-DDR accurately follows the intent of the question for generating the response, demonstrating the ability of RAG-DDR to mitigate the negative influence of external knowledge.
CONCLUSION
This paper proposes Differentiable Data Rewards (DDR), aiming at end-to-end optimizing the Retreeval-Aummented Generation (RAG) model using the DPO method. DDR optimizes each agent by collecting the reward in a rollout way and aligns data preferences among these communicative agents. We build two-agent RAG system and optimize it using DDR to implement the RAG-DDR model. Our experiments demonstrate that DDR helps the generation module produce responses of an appropriate length and avoids overfitting the training signals during SFT. Our further analyses reveal that the DDR optimized generation model can better capture key information from retrieved documents and mitigate the conflict between external knowledge and parametric memory.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partly supported by the Natural Science Foundation of China under Grant (No. 62206042, No. 62137001, and No. 62272093), CCF-zhipu Large Model Innovation Fund (No. 202403).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. ArXiv preprini, 2023. UtL/https://aat //axxiv org/ hatg///b///// 2303. 08774
Shourya Aggarwal, Divyanshu Mandowara, Vishwaject Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. Explanations for Commonsense@A:: New Dataset and Models. In Proceedings of ACL, pp. 2050-3065, 2021. URL httts://acllantholoo org/ 2021 acl-alng.....oa.... 238
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorsii Yejin Choi, and Hannanch Hajishirzi. MathQA: Towards interpretable math word problem solving with operation bbased for- malisms. In Proceedings of NAACL-HLT, pp. 2357 2367, 2019. URL
org/N19- 1245.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. ArXiv preprint, 2023. URL https: //arxiw.org/abs/2311 11511
Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, and Wen-tau Yih. Reliable, adaptable. and attributable language models with retrieval. ArXiv preprint, 2024. hRtp::tss://arxi http:://aawivv.ooabb//////a
03187
Payal Bajaj, Daniel Campos. Nick Craswell, Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, al. Ms marco: A human generated machine reading comprehension dataset. ArXiv preprint, 2016. httthttss///xiiiv
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of EMNLP, pp. 1533-1544, 2013. URL https:
//aclanthologyyorrg/111--- www.ooogyyyoog/1111116111
Deng Cai, Yan Wang, Wei Bi, Zhaopeng Tu, Xiaojiang Liu, Wai Lam, and Shuming Shi. Skeleton- to-response: Dialogue generation guided by retrieval memory. In Proceedings of NAACL-HLT, pp. 1219-1228, 2019. /tL/httts////ccatttoto
Hung-Time Ting Chen, Michael Zhang, and Funsol Choi. Rich knowledge sources bring complex knowl- edge conflicts: Recalibrating models reflect conflicting evidence. In Proceedings of EMNLP, pp. 2292-2307, 2022. URL httpt:tts//:/iiii http:///aaaxiioooopo/////10//
13701
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in reureeedl-augmeented generation. ArXiv preprint, 2023. URL.
01431
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yumxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fimetuneed language models. Journal of Machine Learning Research, 25(70): 2024. URL https:
//www. www.:jjmr..orgppeee. www..orppppppr////2.. 0870. html
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, 2018. https://t////rxxii wrg//bs//////a// /ttp//b./1180.0555)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, 2021. URL
14168

---

Published as a conference paper at ICLR 2025
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In Proceedings of ICLR, 2019. URL
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018. URL
org /ogg/111 1trgg/L181111 1544
Philip Feldman Foulds, R James, and Shimei Pan. Ragged edges: The double-edged sword of retrieval-uggmented chatbots. ArXiv preprint, 2024. URL
01193
Yunfan Gao, Yun Xiong, Meng Wang, and Haofen Wang. Modular rag: Transforming rag systems into lego-like reconfigurable frameworks, ArXiv preprint, 2024. UtL/htpp:///rxxvoorrr//// 2407. 21059
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, pp. 346-361, 2021. URL /tttts:///axiiii .org/
pdf 2t0122111.020 02235
Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick. Efficient nearest neighbor language models. In Proceedings of EMNLP, pp. 5703-5714, 2021. URL httpp:://axxv..org///a
2169. 04212
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv preprint, 2020. URL
attps://arxiiy http:///aaxxv.oooabbb/////ss 2009.03300
Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. Chatdb: Aug- menting iims with databases as their symbolic memory. ArXiv preprint, 2023. URL https:
//arxiv.orge .rg/ h/g//bb//20655 03901
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In Proceedings of ICLR, 2022. URL https://arxiv. org/pdf 221///211000000 09685
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. ArXiv preprint, 2024. URL https://arxivvorr//bbb. 06395
Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering. ArXiv preprint, 2020. URL http://://aaxiv..rraaa//////////
04584
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii. Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023. URL
2220.003299
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of EMNLP, pp. 7969-7992, 2023. URL
orw.hhhooog..ogg//2/// wmn.p-main. 495.
pdf
Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Li Qiuxia, and Jun Zhao. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval- augmented language models. Proceedings of COLING, pp. 16867-16878, 2024. https::
//aclanthology. org/ 2024. Irec-main. 1466.pdd
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of ACL, pp. 1601-1611, 2017. URL htttos://adlanntoollgyoo.////1/3
12

---

Published as a conference paper at ICLR 2025
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail tail knowledge. In Proceedings of ICML, pp. 15696-15707, 2023. URL https::/proceeedinggt wtw.s.ess..rrsppeeeeeeeeee..e.le wtt.reeeeeee///eeeeeeeee.e v2222/aamdaaa22i//aaaaa22222222
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP, pp. 6769-6781, 2020. URL
emmnlpmminn.55 550
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. of ECML. pp. 282-293, 2006. URL htttps://ciieseee ist..su............ http.///occcmeeeerepprrreeeg//// pdffdoi1666115523377 0137776e677777722444464444473333
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, pp. 452-466, 2019. URL http:: org/019-1026)
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of llms. ArXiv preprint, 2024. URL https:
/aarxiv.org/abs/// 2406. 18629
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- -uggented generation for knowledge-intensitn NLP tasks. In Proceedings of NeurIPS, 2020. URL httptpppp//opcceeinngs..stt neurips httpp//psp...ppapp//////////////
Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-augmentte text generation. ArXiv preprint, 2022. URL https:///xxi hwtp:/ii.wwgg//.....ee/ees wwwgg/////g/////// wtwp/a/.....e2/ee/ 01110
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: serrreeal-augmeneeed dual instruction tuning. ArXiv preprint, 2023. URL https: htaps:://arxivy wttp:iiwwrrg/b///////e 01352
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gen- eration: Learning to solve and explain algebraic word problems. In Proceedings of ACL, pp. 158-167. 2017. URL http::::///aalaatooloyy////// wttp////ooo....///1111153
Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang. Dahua Lin, and Kai Chen. Mathbench: Evaluating the theory and application proficiency of llms with a hierarchical mathematics benchmark. arXiv e-prints, pp. arXiv-2405, 2024a. httht:tts///..tttt wwt////gg///////oo/o//o
org/ http//dff22440 /200. 12209
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, pp. 157-173, 2024b. URL https:///axiv..
pdf/2307.05 03172
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. basety-based knowledge conflicts in question answering. ArXiv preprint, 2021. URL |
//arxiv.org/abs/2100 2005225512000222
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. ArXiv preprint, 2023a. URL BtLLhhtts://aaatii org hwrg///bs/2305.. 15225
Yun Luo, Zhen Yang, Fandong Meng. Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. ArXiv preprint, 2023b.
13

---

Published as a conference paper at ICLR 2025
Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai Wei Chang. Retrieval augmented code generation and summarization. ArXiv preprint, 2021. URL https:
//axxiv.org/abs/22100 11601
Baolin Peng. Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. ArXiv preprint, 2023. URL https:
//arxiv.org/abs a200g 2302 12813
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. ArXiv preprint, 2024. URL /ettps://arxiv. http:///axiio.o/ab////////
07199
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Pro- ceedings of NeurIPS, 2024. URL htttttsss///piiiii /ttt:://///xxi..or///dd
org/ htte:ddf/23000 2305. 18290.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- and Yoav Shoham. In-context retrieval-augmentte language models. Transactions of the Association for Computational Linguistics, pp. 1316-1331, 2023. URL https://arxxwworrg
00083
Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamil- ton, and Bryan Catanzaro. End-to-end training of neural retrievers for open-domain question 2021. URL
answering. In Proceedingy of ACL, pp. 2021. 1o1glong.519
org//
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, 2017. httph wrtp://////bb//...oomo 06347
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Sco, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retrieval-augmmened black-box language models. ArXiv preprint, 2023. URL https:://aaxii httt::://arxiv.org/bb/////33
12652
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Proceedings of EMNLP Findings, pp. 3784-3803, 2021. URL
hwwplllooy.ogggg/// http://222......nngeeee findings-emnlp. 320 .ddf
Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. In Proceedings of NeurlPS, pp. 1-968-25981. 2021. URL
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. ArXiv preprint, 2022. URL https: larxiv.org//be. 2210.22120 09261
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQ:: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT, pp. 4149-4158, 2019. URL httttsss/aclaanholooyyy.oooooeee http:/oooyyoo/NNN9111111
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, 2023. URL https:://axiv..orr org.abs./rrg///////
2302. 13971
Harsh Trivedi, Niranjan Balasubramaniann, Tushar Khot, and Ashish Sabharwal. Interleaving re- trieval with chain-of-thought reasoning for knowledge-intensivee multi-step questions. In Pro- ceedings ACI, pp. 10014-10037, 2023. 557. 557.pdf
attp//222...1l11111.ee acl-
Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to filter context for retrieval-augmented generation. arXiv preprint arX1v:2311.08377, 2023. URL
14

---

Published as a conference paper at ICLR 2025
Zhepei Wei, Wei-Lin Chen, and Yu Meng. Instructrag: Instructing retrieval-augmmente generation with explicit denoising. arXiv e-prints, pp. arXiv- 2406, 2024. URL 2406. 13629
Kevin Wu, Eric Wu, and James Zou. Clasheval: Quantifying the tug-of between Ilm's internal prior and external evidence. ArXiv preprini, 2024. URL /tRLhttpss//arrvi..og////////// 10198
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023. URL httpss//:///axiiii https///aaxiv..o/////////// 07597
Jian Xic, Kai Zhang, Jiangjic Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In Proceedings of ICML., org httg 2205 13300
2024. httphhtts:///r//t
Lee Xiong, Chenyan Xiong. Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive lcarning for dense text retrieval. In Proceedings of ICLR, 2021. URL htttps://arxiii owtp://..rrg///////////o http://rxxi.voor//////0/////// 88880000088
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented Ims with
compression and selective augmentation. ArXiv preprint, 2023. URL URL https://arxii.orggaas. 2310.04408
Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou. Unsupervised information refinement training of large language models for retrieval-augmentte generation. ArXiv preprint, 2024a.
Zhipeng Xu, Zhenghao Liu, Yibin Liu, Chenyan Xiong, Yukun Yan, Shuo Wang, Shi Yu, Zhiyuan Liu, and Ge Yu. Activerag: Revealing the treasures of knowledge via active learning. ArXiv preprint, 2024b.
13547
Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. ArXiv preprint, 2024c. URL
11817
Shi-Qi Qi Yan, Jia-Chen Gu, Yun Zhu, and Then- Ling. Corrective retrieval augmented generation.
ArXiv preprint, 2024. URL
15884
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.151155 2024.
Yi Yang. Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. Proceedings of EMNLP, pp. 2013-2018, 2015. URL https://
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of EMNLP, pp. 2369-2380, 2018. URL hittps://aclanthonn
org/D18- 1259.
Wenhao Yu, Hongming Zhang. Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. Chain- of-note: Enhancing robustness in retrieval- augmented language models. ArXiv preprint, 2023a.
URL https://arxxv.orr/abb////
09210
Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. Augmentation-adaated retriever improves generalization of language models as generic plug-in. ArXiv preprint, 2023b. URL. https:
//axxiv.org/abs/200.1733
15

---

Published as a conference paper at ICLR 2025
Table 5: Data Statisties.
Split
Task
Dataset
Metric
Total
Commonsense QA (2019)
Accuracy
4,200
Math QA (0199)
Accuracy
4,200
Web Questions 2013
Accuracy
3,778
QA
Wiki QA 2015
Rouge-I
1,040
Yahoo! Answers QA
Rouge-L
4,200
Training
MARCO QA (2016)
Rouge-L
4,200
Algebra QA with Rationales (2017
Accuracy
2,727
Explanations for Commonsense 2021
Accuracy
4,200
Reasoning
Grade School Math 8K 2021
Accuracy
4,200
StrategyQA 22021
Accuracy
2,060
Natural Questrons 2019
Accuracy
2,837
Open-domain QA
TriviaQA (0017)
Accuracy
5,359
MARCO QA (2016)
Rouge-L
3,000
Evaluation
Multi-Hop QA
HotpotQA (2018)
Accuracy
S,600
Slot Filling
T-REx (2018)
Accuracy
5,000
Dialogue
Wizard of Wikipedia (2019)
F1
3,000
A APPENDIX
A.1 ADDITIONAL EXPERIMENTAL DETAILS
In this subsection, first outline the process of constructing the training data. Then show the prompt templates used in our experiments. Then, we describe the maximum generation length of RAG-DDR during inference. Finally, provide details on the implementation of RAG-DDR training. To ensure a fair comparison, we train both RA-DIT and RAG-DDR using the same dataset, hyperparameters, and number of epochs. For Self-RAG, we adhere respective training data formats and hyperparameter settings.
Data Preprocessing for DDR. The quantity of our training and evaluation data, along with the corresponding evaluation metrics, are presented in Table Then we describe the details of data preprocessing during training both knowledge refinement (RR) and generation (Veen) modules using our DDR method.
Optimizing the knowledge refinement module is rollout process, which individually feeds the top-100 retrieved documents into the generation module (VGen) with the query to calcu- late the reward. Then, we apply the evaluation metrics shown in Table 5 to calculate the reward scores, identifying the documents that result in the highest scores ments and those with the lowest scores as negative ones. Finally, we obtain the triplet data (query, positiveenegative documents, "YES"/NOO") To construet the dataset for training the gen- eration module, we concatenate the refined documents with the query and then feed them into the generation module for sampling responses. We apply five different temperature settings (0.5, 0.6, 0.7, 0.8, 0.9) to sample responses and conduct the five-round sampling for each tem- perature. Afterward, we compute the reward score for each output using evaluation metrics to identify the positive and negative responses for each query. This process yields the triplet data (query, positive response, negative response).
positive docu-
Maximum Generation Length. For the evaluation dataset, we set the maximum generation length for MARCO QA task 100, while the maximum generation length for the rest tasks set to 32.
Prompt Templates. For RA-DIT. we use the same instruction tuning template as Lin et al. (2023) and leverage top-5 retrieved documents as augmented knowledge during training. For Self-RAC
we use the same instruction tuning template as Asai et al. (2023) during training. Then we de- scribe the prompt templates used in RAG-DDR. As shown in Figure we refer to the prompt designs of RA-DIT (Lin et al. 2023) and Self-RAG (Asai et al. 2023) to conduct tailored task prompts for different LLMs, helping LLMS generate better responses. In addition, we design sepa- rate prompts for LLMs w/ RAG and LLMs w/o RAG: Background: Documents in {Instruction) } and [nstrruction) where Documents indicates the external documents provided for LLMs and In- struction represents the task instructions. As illustrated in Figure 5 we design the prompts for
16

---

Published as a conference paper at ICLR 2025
Prompts of Different Tasks (MiniCPM)-2.0g
Prompts of Different Tasks (Llama3-88)
[TASK]: Comonnsnss 04, Math Q4
[TASK]: Comonnsense Q4, Math Qa [Instruction]]: Please questions below and output only the choice. (Question)
[Instruction]: The following is multiple choice question. Please choose best answer which can answer the fallowing question. (Question)
the nultiple choice
Answer:
Answer:
[TASK]: Meb Questions, Miki QA, Yahoo! Q4, NARCO MQ, Triviage, HotpatQA [Instruction]: Q: (Question)
[TASK]: Neb Questions, kiki QA, Yahoo! 04, MS MARCO QA NO, TriviaQA, HotpotQ4 [Instruction]: q: (Question)
A:
[TASK]: Algebra QA with Rationales, Explanations CommonsenseQ,
[TASK]: Algebra with Rationales, Explanations for CommonsenseQ
[Instruction]: Please answer nultiple choice question and choose best answer choice first. Then give your explanation between [<car] and [COT>].
[Instruction]: Please answer the multiple choice questions below output only choice. (Question)
Answer:
question: (Question)
Answer:
[TASK]: Grade School Math 8K, Strategy Q4 [Instruction]: Please answer the question and only output answer. Then give your explanation between [<cor] and [caT>]. question: (Question)
[TASK]: Grade School Math 8K, Strutegy QA [Instruction]:
question. give your explanation between [<cOT] and [COT>]. question: (Question)
Answer:
Answer:
[TASK]: Soft Filling
[TASK]: Soft Filling
[Instruction]: Given the input format "Subfect Entity [SEP] Relationship Type," predict target entity.
[Instruction]: Given the input format 'Subject Entity [SEP] Relationship Type, predict the target entity. (Question)
(Question)
Answer:
Answer:
[TASK]: Dialogue
[TASK]: Dialogue
[Instruction]: (Question_i)
[Instruction]: Q: [Question_
A: {Answer_ 1}
A: (Answer_x)
Q: (Question_n)
Q: (Question_n)
A:
A:
(a) MiniCPM-2.4B.
(b) Llama3-8B.
Figure 4: Prompt Templates of Different Training and Evaluating Tasks.
Prompts of Knowledge Refinement
[Instruction]: Given the following question and context, return YES if the context is relevant to the question and NO if it isn't.
Question: (question)
> Context:
>>>
{context)
>>>
Relevant (YES / NO):
Figure 5: Prompt Templates Used for Knowledge Refinement.
the knowledge refinement tasks by referring LangChair enabling L.LMs correctly generate "YES" or "NO" to retain or discard retrieved documents.
Training Details. For DDR training, we use automatic metrics such Rouge-L and Accuracy to calculate the reward and set 3 = 0.1. The learning rate set to 5e-5, and each model is trained for one epoch. For the generation module, we feed 5 retrieved passages as external knowledge for augmenting the generation process. To optimize both the knowledge refinement module and generation module, we use LoRA (Hu al. 2022 efficient training.
17

---

Published as a conference paper at ICLR 2025
Tabbl 6: Overall Performance of RAG-DDR and Additional Baselines. The evaluation scores of InstructRAG on the MARCO QA and WoW tasks are usually low. The reason may lic in that the responses generated by InstructRAG are long and contain complex reasoning processes, making
difficult to conduct fair evaluation for these string matching based evaluation metrics.
Open-Domain QA
Multi-Hop QA
Slot Filling
Dialogue WoW
Method
NQ
TrivaQA
MARCO QA
HotpotQA
T-REx
Short Generation Length
Filco (2023
46.6
81.9
14.6
25.3
44.1
14.4
InstrucTRAG 2024
4.2
21.1
14.3
11.0
17.1
9.2
RAG-DDR (w7 2-Round)
52.1
89.6
27.3
39.0
40.6
16.8
Long Generation Length
InstructRAG (2024)
60.8
92.0
12.4
42.3
43.1
5.3
RAG-DDI (w7 2-Round)
58.0
91.9
26.1
43,9
45.6
10.9
Table Additional Ablation Study Results on RAG-DDR.
Open-Domain QA
Multi-Hop QA HotpotQA
Slot Filling
Dialogue
Method
NQ
TriviaQA
MARCO QA
TREX
WoW
MiniCPM-2.44
Independent Tuning
46.6
82.0
28.0
32.3
31.7
17.3
RAG-DDR (ViR First)
47.4
83.2
27.2
33.3
35.0
17.3
RAG-DDR (Vien First)
47.0
82.7
28.1
32.5
32.6
17.2
1lama3-88
Independent Tuning
49.9
88.3
25.1
37.4
37.0
14.9
RAG-DDR (ViR First)
88.6
25.6
36.9
36.7
14.9
RAG-DDR (Vaen First)
50.7
88.2
25.1
37.3
37.3
14.9
A.2 ADDITIONAL BASELINE COMPARISON RESULTS
This section presents the comparison results between RAG-DDR (w/ 2-Round) and other baseline models. We employ InstructRAG (Wei et al. 2024) and Filco (Wang et al. (023) as baseline models. InstructRAG guides the I.IM to self-synthesize denoising instruction tunigg data and trains the generation module to denoise the retrieved documents. Filco finetunes Flan-T5-XI. (Chung et al.) 2024) as a context filtering module to refine the retrieved documents. Specifically, the responses generated by InstructRAG include long reasoning and analysis, thus InstructRAG sets the maximum generation length as 4,096. For a fair comparison, we compare the performance of RAG-DDR and InstructRAG the two maximum generation length settings: the short generation length (MARCO QA task 100 and the rest tasks are 32), and the Long generation length (4,096). In our experiments, employ Bammaa----Instruun as the knowledge refinement and generation modules.
As shown in Table 6, RAG-DDR outperforms Filco almost all tasks. This suggests that the SFT method leads modules in the RAG system overfit the training signals, making less effective to optimize the entire RAG system to generate accurate responses. In contrast, in the long gen- eration length setting, InstructRAG slightly outperforms RAG-DDR but performs slightly worse than RAG-DDR on T-REX and HotpotQA tasks. This illustrates that high-quality synthetic fine-tuning data can effectively optimize the performance the generation module. However, in the short generation length setting, InstructR AG performs significantly worse than RAG-DDR, which indicates that compared to InstructRAG. RAG-DDR can generate accurate answers with shorter responses, without complex and long reasoning processes.
NQ and TriviaQA tasks
A.3 ADDITIONAL ABLATION STUDIES ON RAG-DDR
As shown Table we conduct additional ablation studies to explore the effectiveness of differ- ent training strategies: Independent Tuning, RAG-DDR (VeR First) and RAG-DDR (VGen First). Independent Tuning indicates that the knowledge refinement module VKR and generation module Voe trained independently. RAG-DDR (VRR First) and RAG-DDR (Vasa First) cascaded optimization models, indicating that we first train VER or Vaen and subsequently optimize the other module by initializing the RAG model with the already optimized module.
18

---

Published as a conference paper at ICLR 2025
Table Effectiveness of RAG-DDR with More Agents. Llama3-8B is used the backbone model for both the knowledge refinement and summarization modules, while MiniCPM-2244B is used as the backbone model for the generation module.
Open-Domain QA
Muld-Hop QA
Slot Filling
Dialogue
Method
NQ
TriviaQA
MARCO QA
HotpotQA
T-REx
WoW
RAG (w/o Vsun)
79.5
16.7
26.7
14.4
Vanilla RAG
44.1
82.3
17.5
28.9
25.4
15.4
RAG-DDR (Only Training Vom)
46.6
83.6
26.9
32.1
28.3
16.9
RAG-DDR (Training VGen & Viun)
47.6
83,9
26.4
32.0
28.6
17.0
RAG-DDR (All Training)
47.9
84.8
26.4
32,9
28.7
16.9
Table 9: Effectiveness of RAG-DDR Using Large-Scale I.LMs. In our experiments we utilize
Qwen2.5-14B 1wen2.5-14B-Instructt the generation module and Llama3-8B 8B-Instruct the refinement module.
Open-Domain QA
Multi-Hop QA
Slot Filling
Dialogue
Method
NQ
TriviaQA
MARCO QA
HotpotQA
T-REx
WoW
Vanilla RAG
48.1
81.1
18.5
29.4
37.3
14.3
RAG-DDR
51.2
84.1
23.0
35.1
44.6
16.6
Compared to Independent Tuning, the effectiveness of RAG-DDR is enhanced on all tasks. It shows that Independent Tuning results misalignment of data preferences between the optimized modules, thereby impacting the overall performance of the RAG system. In contrast, the performance of RAG- DDR (VER First) and RAG-DDR (Veen First) is indistinguishable across different datasets and Voen demonstrating that the DDR method is robust to the training orders.
A.4 EFFECTIVENESS OF RAG-DDR WITH MORE AGENTS
In this experiment, extend RAG-DDR more agents evaluate its effectiveness in optimizing complex RAG systems. Specifically, we utilize the knowledge refinement (VKR), summarization (Vsum), and generation (VGea) modules construct a new RAG system. In this system, the knowl- edge refinement module refines the retrieved documents D get the refined documents D, the summarization module summarizes the refined documents δ into concise summary $ based the query 4, and the generation module uses this summary $ to answer the query q. This system can be represented as a threc-agent system:
{g. D} ViR (g.D), Vun
14.51
VGen
YGen:
where D C D. {q. D} ~ and ~* represent sending the input fq. D} to the RAG system and getting the final output yr. In implementation, use Llama3-8B Instrrutt the backbone model to construct the knowledge refinement and summarization modules, and Miniepm--244B-sh the backbone model to construct the generation module.
As shown Table g we employ five models: Vanilla RAG (w/o Vsum. Vanilla RAG, RAG-DDR (Only Training VGen RAG-DDR (Training VGen & VSum) and RAG-DDR (All Training) evaluate the effectiveness of the DDR method in RAG systems with more agents. RAG-DDR (Only Training VGen indicates that we tune the Vanilla RAG using DDR by only optimizing the generation module (Veen). RAG-DDR (Training Vaen & Vsum) indicates that we use DDR to optimize both the gener- ation module (VGen) and summarization module (Vsum) in Vanilla RAG. RAG-DDR (All Training) indicates that we optimize all three modules in the Vanilla RAG.
Compared to Vanilla RAG (w/o Vusm), Vanilla RAG demonstrates performance improvements across all evaluation tasks, showing the effectiveness of the summarization module in extracting relevant information from multiple documents. In contrast, RAG-DDR (Only Training VGen) shows greater improvements than Vanilla RAG, indicating that the primary improvements of RAG-DDR come from optimizing the generation module (Vien). When we subsequently optimize the summarization and knowledge refinement module based RAG-DDR (Only Training Vaea), the performance is further improved. This indicates that DDR not only improves the performance of the RAG system but also exhibits strong scalability, allowing it be extended to different RAG systems.
19

---

Published as a conference paper at ICLR 2025
100
92.5
87.8 88.1 88.2
87.5
Vanilla RAG
Vanilla RAG
80
RA-DIT
RA-DIT RAG-DDR
RAG-DDR
Accuracy
Accuracy
60
57.2
58.0
40
42.2
34.5
42.4
31.8
40
36.9 37,0 37.0 37.3
36.2 37.3
20
20
TriviaQA
HotpotQA Different Datasets
T-REX
TriviaQA
HotpotQA Different Datasets
T-REx
(a) The Accuracy of Knowledge Refinement.
(b) The Accuracy of Generation Using Different Re- fined Document Sets.
Figure 6: Effectiveness of the Knowledge Refinement Module (VKR) Optimized Using Different Methods, Including Zero-shot (Vanilla RAG), SET (RA-DIT), and DDR (RAG-DDR).
Mathematical
w/o
Mathernttiaal 70
w/o RAG
RAG-DDR
RAG-DOR
65
50
Common Sense
50
Disciplinary Knowledge
Common Sense Reasoning
33
Disciplinary Knowledge
Reasoning
10
Logical
World Knowledge
Logical Reasoning
World Knowledge
Reasoning
(a) Llama3-8B.
(b) MiniCPM-2.4B.
Figure 7: The Ability of the Generation Module (VGen) in RAG-DDR.
A.5 EXTENDING RAG-DDR TO LARGER-SCALE LLMS
In this section, we explore the effectiveness of the DDR method in the RAG system that em- ploys a larger-scale LLM as the backbone model of the generation module. Specifically, we utilize Dwen2.5-14BB- 1nB-tnstrut YYang et al.) 2024) as the generation module and refinement module build new combination for the RAG system. As shown in Table compared to vanilla RAG, RAG-DDR shows significant improvements across multiple tasks, highlighting the effectiveness of the DDR method the RAG system with the larger-scale LLM.
as the
A.6 CHARACTERISTICS OF THE KNOWLEDGE REFINEMENT MODULE OF RAG-DDR
As shown in Figure 6 we explore the characteristics of the knowledge refinement module (VKR) optimized using different methods, including zero-shot (Vanilla RAG), the SET method (RA-DIT), and DDR (RAG-DDR).
For RA-DIT, we collect 30k pieces of data from the MARCO QA dataset to tune VKR module where each query has labeled positive and negative documents. We sample positive doc- ument and one negative document for each query to construct the SFT dataset containing 60k pieces of data and fine-tune VKR module. The number of the SFT dataset is consistent with the amount of the RAG-DDR training dataset. The SFT dataset consists of triples in the form of (query, positiveenegative documents, "YES"
As shown in Figure 6(a) we calculate the accuracy top-5 documents by Vanilla RAG, RA-DIT, and RAG-DDR. The retrieval accuracy evaluates whether or not the retained top-5 documents
20

---

Published as a conference paper at ICLR 2025
Table 10: The Effectiveness of Generation Module Optimized with Different Training Strategies.
Open-Domail QA
Mulii-Hop QA
Slot Filling
Dialogue
Method
NQ
TriviaQA
MARCO QA
HotpotQA
T-REx
WoW
MiniCPM-2.4B
Vanilla RAG
42.2
79.5
16.7
26.7
22.1
14.4
RA-DIT (Positive Label)
40,9
77.2
19.9
22.0
25.3
14.7
RA-DIT (Ground Truth)
41.8
78.6
19.6
25.2
15.7
RAG-DDR
46.8
81.7
28.3
32.2
17.0
Llama3-8B
Vanilla RAG
46.2
84,0
20.6
30.1
26.9
12.5
RA-DIT (Positive Label)
45.8
86.4
23.0
31.4
25.3
13.2
RA-DIT (Ground Truth)
46.2
87.4
20.3
34.9
41.7
14.8
RAG-DDR
50.2
87.8
25.2
36.9
36.2
14.8
Table 11: The Average Length of Responses Generated by Generation Module Optimized with Different Training Strategies.
Open-Domain QA
Multi-Hop QA
Slot Filling T-REx
Dialogue WoW
Method
NQ
TriviaQA MARCO QA
HotpotQA
MiniCPM-2.4B
Vanilla RAG
12.5
4.7
32.4
5.6
4.4
48.5
RA-DIT (Positive Label)
19.2
8.3
34.6
13.7
6.7
32.8
RA-DIT (Ground Truth)
7.6
4.8
13.8
8.9
5.0
17.2
RAG-DDR
27.4
16.9
47.8
27.2
11.2
56.6
Llama3-8B
Vanilla RAG
49.6
25.0
85.4
98.5
RA-DIT (Positive Label)
42.1
14.8
72.2
24.0
3.9
72.2
RA-DIT (Ground Truth)
6.9
8.7
14.5
23.6
4.0
16.7
RAG-DDR
56.0
33,9
81.2
51.6
35.2
101.5
tain the ground truth. If VKR module discards all the documents retrieved, the accuracy is 0. RAG- DDR outperforms Vanilla RAG and RA-DIT on all tasks. It indicates that the DDR method can make VKR module accurately retain documents that contain the necessary knowledge answer the query. As shown in Figure 6(b) we use the refined documents by different knowledge refinement module (VKR) augment the DDR trained generation module (VGen). DDR-RAG also outperforms other models, indicating that DDR can help better align data preferences between VGen and VKR modules.
A.7 THE GENERAL LLM ABILITY DDR OPTIMIZED GENERATION MODULE
In this experiment, further explore the characteristies of the generation module (VGen) optimized using different methods, zero-shot (LLM w/o RAG) and DDR (RAG-DDR).
As shown in Figure 7 we compare the general ability of the generation module several aspects: Mathematical (Liu et al. 2024a).. Disciplinary Knowledge (Hendrycks et al., 2020), World Knowl- edge (Kwiatkowski Reasoning (Clark et al. 2018). These tasks are commonly used as benchmarks to assess the model's inherent capabilities (Touvron et al. 2023 Hu et al. 2024).
al. 2019), Logical Reasoning (Suzgun al. (022). and Common Sense
As shown in Figure 7 DDR enables Llama3-8B to maintain its strong language understanding
and knowledge reasoning capabilities. The performance MiniCPM-2244 is shown in Figure (Tb) The evaluation results show that DDR significantly enhances the performance of the smaller pa- rameter models, MiniCPM-2244B particularly in mathematical and common sense reasoning tasks. It illustrates that DDR not only preserves the original capabilities of LLMs but also offers some potential for enhancing their performance.
A.8 THE IMPACT OF DIFFERENT TRAINING STRATEGIES ON THE GENERATION MODULE
In this experiment, we further investigate the impact of different training strategies the generation module Vam 's ability to utilize external knowledge and the average length of responses generated by Vae module. We compare four models in this experiment, including Vanilla RAG, RA-DIT
21

---

Published as a conference paper at ICLR 2025
(Positive Label), RA-DIT (Ground Truth) and RAG-DDR. For RA-DIT (Positive Label), employ the RA-DIT method to train the Vanilla RAG and regard the positive response DDR training data ground truth. For RA-DIT (Ground Truth), employ the RA-DIT method to train the Vanilla RAG with the annotated labels.
As shown in Table 10 RAG-DDR consistently outperforms RA-DIT (Positive Label) and RA-DIT (Ground Truth) across different datasets, highlighting the effectiveness of the DDR method to en- hance the ability of VGen module utilize external knowledge. contrast, RA-DIT (Positive Label) performs worse than RA-DIT (Ground Truth), demonstrating that relying solely on positive samples for supervised tinettunin insufficient for effectively training LLMs.
Furthermore, we analyze the average length of the responses generated by VGea module in Table I1 Compared to the RA-DIT (Ground Truth), RAG-DDR shows more similar generation length dis- tribution to the Vanilla RAG model, making the LLMs generate responses with more appropriate length to answer the question. Notably, the response length of RA-DIT (Positive Label) is longer than RA-DIT (Ground Truth). This observation further shows that the SFT training method may lead LLMs to overfitting the training signals, which affects the response length of LLMs.
22


Reviews:
[{"review_id": "efROfJ79Hy", "reviewer": "Reviewer_FTLt", "summary": "This paper focuses on improving the retrieval-augmented generation (RAG) system by fine-tuning the content filter and the LLM generator while keeping the retriever frozen. It claims that existing supervised fine-tuning (SFT) approaches fail to model data preferences for different modules in the RAG system (i.e., the filter and generator) and tend to overfit training signals. In this work, the authors introduce a training method called differentiable data rewards (DDR), which uses DPO loss to train both the filter and generator. Experiments on six knowledge-intensive tasks show superior performance compared to several off-the-shelf RAG baselines and a fine-tuned RAG baseline (though without the filter module to denoise the retrieved contents).", "questions": "Please address the questions in Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "=== Strengths ===\n\nS1. The motivation of this work is clear, and the paper is easy to follow.\n\nS2. The idea of introducing DPO loss to train the content filter and LLM generator in RAG is novel (although its significance is not quite clear given the current empirical study, see Weakness 1).\n\nS3. The main comparison between the proposed method and baseline methods, along with the ablation study, is evaluated on a wide range of knowledge-intensive tasks.", "weaknesses": "=== Weaknesses ===\n\nW1. Potentially problematic experimental setting. The main claimed contribution is proposing a DPO loss to train the content filter and LLM generator. However, in the experiments, the proposed method is primarily compared to off-the-shelf RAG methods without any training. The only fine-tuned baseline is RA-DIT, which only trains the LLM generator. Does RA-DIT also leverage the content filter module during evaluation? If not, it is unclear whether the improved evaluation performance is due to filtering or to the better optimization by DPO loss compared to traditional SFT loss. Furthermore, is “RAG w/ $V_{KR}$” the only RAG baseline that includes a content filter (i.e., knowledge refinement) module? The comparison in the main table and ablation study may be unfair and the results can be misleading if the proposed RAG-DDR can benefit from an additional filtering mechanism that other baselines lack of.\n\nW2. Missing comparisons and discussion with closely relevant RAG baselines on denoising retrieved content. Approaches for mitigating conflicts between external retrieved knowledge and the LLM’s internal parametric knowledge, as well as for denoising retrieved content, have been widely explored in RAG literature [1,2,3,4]. A more thorough discussion of these works is needed to better situate this work. For instance, [2] directly trains the LLM generator using SFT loss with noisy inputs to improve robustness to noise without any filtering/knowledge refinement, [3] employs self-synthetic rationales to denoise retrieved content and guide generation for better accuracy, which is also trained with SFT loss, and [4] introduces an inference-time algorithm to deal with irrelevant retrieved documents through a mechanism called isolation-then-aggregation. Without a detailed comparison with such related works, it is difficult to assess the empirical significance of this work, especially since some of these works are strong RAG methods trained with SFT loss (e.g., [2,3]) for specific denoising purposes.\n\n**References**\n\n[1]. AstuteRAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models. arXiv 2024.\n\n[2]. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. ICLR 2024.\n\n[3]. InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales. arXiv 2024.\n\n[4]. Certifiably Robust RAG against Retrieval Corruption. arXiv 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t8m7gkbmHY", "reviewer": "Reviewer_oqk9", "summary": "This paper introduces RAG-DDR, a novel reinforcement learning-based approach designed to improve retrieval-augmented generation (RAG) systems. The authors conceptualize the RAG framework as comprising two key modules: a knowledge refinement module, $V_{\\text{KR}}$​, and a generation module, $V_{\\text{Gen}}$. RAG-DDR computes partial reward for each module's response with a rollout process. Preference pairs are then constructed based on the rewards and then being used to optimize $V_{\\text{Gen}}$ and $V_{\\text{KR}}$ through DPO. Specifically, for $V_{\\text{Gen}}$ (with filtered documents), the response with highest score is considered positive and lowest score is considered negative. For $V_{\\text{KR}}$, the document that, when passed through $V_{\\text{Gen}}$, yields the highest response score is considered positive, while the document with the lowest score is negative.\n\nThe authors perform extensive experiments across various knowledge-intensive tasks to evaluate RAG-DDR, comparing it against the vanilla RAG and SFT-based methods, showing that RAG-DDR can generate more accurate response and has the ability to leverage LLM's internal knowledge while not being misled by the external noisy information. Ablation studies further demonstrate the effectiveness of optimizing both $V_{\\text{KR}}$ and $V_{\\text{Gen}}$ using the proposed DDR approach.", "questions": "1. What distinguishes the proposed DDR method from step-DPO? Is that the only difference comes from the base policy, i.e., in DDR the base policy at each step comes from different agents while for step-DPO the base policy at each step are shared?\n2. What is the impact of training RAG-DDR iteratively over multiple rounds? While results for 2-round training are reported in Table 1, a more detailed analysis of the outcomes would be beneficial.\n3. In section 5.3, the authors argue that \"SFT training method tends to cause LLMs to overfit the supervised data\" from the observation that RA-DIT's generated responses is significantly shorter than vanilla RAG and RAG-DDR's responses. While I generally agree with this statement, I wonder whether this phenomenon could be attributed to differences in the average response length within the training sets for RA-DIT and RAG-DDR. If so, what would occur if we trained RA-DIT using the positive samples from RAG-DDR? Would this adjustment also impact the model's accuracy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments together with detailed analysis. In particular, the result from Table 3 is interesting, as it shows RAG-DDR's ability to balance between using internal and external knowledge.\n2. The overall methodology is straightforward yet effective, leveraging a DPO-based approach to facilitate self-training.", "weaknesses": "1. Some aspects are not clear, see the question section.\n2. (Minor thing) Typo at line 850 \"identify the positive and positive responses for each query.\" -> \"identify the positive and negative responses for each query.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYmL6r8zBp", "reviewer": "Reviewer_QfBP", "summary": "This paper introduces Differentiable Data Rewards (DDR), a novel method for training Retrieval-Augmented Generation (RAG) systems. DDR collects rewards via a rollout method to optimize each agent while aligning data preferences between different agents within the system. The method addresses two key challenges in RAG systems: (1) training signal overfitting and (2) poor robustness against conflicting knowledge.", "questions": "1. Kindly refer to the Weakness section.\n\n2. Could the authors define what is meant by “data preferences between different RAG modules”? This concept appears multiple times in the paper, yet it remains somewhat unclear to me. Given that RAG primarily addresses knowledge-based tasks, could “preferences” here be more aligned with factual accuracy? I would appreciate a brief definition, ideally illustrated with a case study that demonstrates why this issue is important. It would be especially helpful if different base models were involved, as their internal knowledge may vary, potentially influencing their data preferences.\n\n3. In addition, regarding the rollout method:\n\n- What are the training speed and computational requirements? How does the method scale with larger datasets in terms of efficiency?\n- Could the DDR method be expanded beyond just two agents? A more in-depth discussion from the authors on this point would offer valuable insights for developing more complex RAG systems.\n\n4. An interesting finding in Appendix A.4 is that DDR enhances MiniCPM-2.4B’s mathematical capabilities. Could the authors provide a brief analysis explaining the possible reasons for this?\n\n5. Typo: Line 269, should be \"Llama3-8B-Instruct\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall, the presentation is good and the method is easy-to-follow.\n- The paper introduces the idea of DDR, which enables end-to-end optimization of RAG systems. The proposed method of aligning data preferences between knowledge refinement and generation agents alleviates the robustness issue when encountering external knowledge conflicts.\n- The experimental results show that RAG systems trained with DDR strategy demonstrate improvements in terms of knowledge usage and robustness against knowledge conflicts.", "weaknesses": "- Figure 1 is not clear enough. It could be better to illustrate the correspondence between the left and right parts, as well as the order of two modules in the right part (could use step 1, 2 as annotations). Additionally, it would be helpful to indicate how the reward signal propagates throughout the entire training process.\n\n \n- I realize that training large language models is resource-intensive, but the demonstrations in the paper could be more convincing by including:\n\n    - (If possible)Additional experimental results from larger-scale LLMs, such as Qwen2.5-14B or other larger models.\n    - (If possible)Additional evaluations using more different LLM combinations to compose the RAG system (It seems that the current setup consider Llama3-8B-Instruct as knowledge refinement module and MiniCPM-2.4B/Llama3-8B-Instruct as generator)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7wZfguXvUk", "reviewer": "Reviewer_1cKK", "summary": "The paper proposes the DDR framework for tuning agentic systems. The key idea is to sample multiple actions from an agent, and score them based on the downstream reward they fetch (computed via unrolling all subsequent agents in the chain). The actions are ordered by this score, and the DPO method is used to increase the likelihood of high scoring actions over low scoring actions.\n\nThe framework is applied to optimize a two agent RAG system consisting of a knowledge refinement agent (selecting relevant documents for RAG), and a generation agent that generates the response based on the selected document.\n\nThrough experiments and case studies, the authors show that the framework can perform better than several baselines, and can be better balance internal parametric knowledge with external information.", "questions": "Suggestion: I would encourage the authors to perform the ablation studies and related work comparisons suggested in the \"Weaknesses\" section.\n\nSome questions about the KR module:\n- The KR module generates binary Yes/No responses for the relevance of each document to the query. How is the binary output used to rank documents and identify the top-k documents?\n- Line 216 says:  \"document di that leads the agent system to achieve the highest evaluation reward is considered positive\". Does this mean a single document is selected and fed to the generator module? But equation 7 shows that k documents are fed. Am I missing something?\n- If it is a single document then when tuning the KR module do we consider the highest-rewards document as positive, and all others as negative? \n- Would it make sense to consider contrastive objectives like ensuring P(Yes | q, d_i) > P(Yes | q, d_j) whenever the reward fetched by d_i is higher than d_j?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses an important concern in RAG system of balancing use of external information versus internal parametric knowledge \nof the generation module. This is common problem for production RAG systems where retrieval is never perfect and often suffers from conflicting and/or missing information.\n\nThe idea of jointly tuning retrieval and generation modules is interesting and not studied as much in the literature (although some prior art does exist; see next section).\n\nThe results in Sections 5.3 and 5.4 are compelling and show a nice advancement over baseline methods in dealing with noisy retrieval in RAG systems.", "weaknesses": "The main weakness of the paper is inadequate comparison with related approaches on jointly tuning retriever and generator modules, which make it hard to appreciate the technical contribution of the paper. \n\nThere is prior work on supervising the retrieval module of RAG system based on performance of the generator (e.g., Wang et al. 2023, https://arxiv.org/pdf/2311.08377). In my eyes, the novelty of this paper is the use of DPO instead of a supervised tuning (which would just maximize likelihood of action fetching highest reward). Thus, if I understand correctly, the novelty of this work is the negative gradient induced by the contrastive DPO loss. However, this is not discussed in the paper. It would help to study the impact of the negative gradient in depth, via ablation studies. For instance, is the negative gradient responsible for the claim in Section 5.3: \"DDR can help learn more factual knowledge during training while also preventing the loss of previously memorized information\".\n\nThe paper highlights RAG-DDR's ability to effectively combine parametric knowledge with external knowledge. To assess this, it would be helpful to see a comparison with adaptive approaches that use different retrieval modules (including no retrieval) for different queries. For instance, see: Jeong et al., 2024: https://arxiv.org/pdf/2403.14403\n\nOther helpful ablations and baselines to experiment with:\n* Section 3.2 notes that RAG-DDR tunes the generation module first followed by the knowledge retriever (KR) module. It would be helpful to perform ablations where the KR module is tuned before the generation module (as in Wang et al,. 2023).\n\n* When evaluating vanilla LLM and vanilla RAG, it would help to experiment with large LLMs like Gemini-1.5-pro, GPT4o to understand if the inability to balance parametric knowledge vs. external knowledge is an artifact of smaller LLMs.\n\nThe paper introduces the framework of differentiable data rewards (DDR). However, the framework is only evaluated for simple two agent systems. Consequently, DDR does not come across as a fully developed contribution. Instead, the key contribution, at least to me, is the use of DPO in jointly trying retrieval and generation modules. I would encourage the authors to build on this aspect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bAiyAMYp4M", "reviewer": "Reviewer_LZe5", "summary": "This paper introduces RAG-DDR (Retrieval-Augmented Generation with Differentiable Data Rewards), a novel method for optimizing RAG systems using reinforcement learning. Rather than using traditional supervised fine-tuning approaches, RAG-DDR uses a reward-based learning system to optimize both the knowledge retrieval and generation components. The method specifically focuses on aligning data preferences between different modules in the RAG pipeline to improve overall system performance, particularly in handling conflicts between parametric and external knowledge.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach is novel to RAG optimization. By using differentiable rewards, it effectively addresses the knowledge conflict problem that often plagues RAG systems.", "weaknesses": "1, The computational cost is high. RAG-DDR requires significantly more computational resources than traditional methods, requiring about 5x more API calls than standard approaches. This could limit its practical applicability in resource-constrained settings.\n2, The paper also shows some limitations in complex reasoning tasks, particularly in multi-hop reasoning scenarios where the model can sometimes prematurely conclude questions are invalid. While the method shows strong performance on factual question-answering tasks, its effectiveness on more complex reasoning tasks could be improved.\n3, while the paper demonstrates strong empirical results, it could benefit from a more thorough theoretical analysis of why the method works so well, particularly in terms of how it achieves better alignment between different RAG components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)