RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards

## Abstract

Abstract Retrieval-Augmented Generation (RAG) has proven its effectiveness in mitigating hallucinations in Large Language Models (LLMs) by retrieving knowledge from external resources. To adapt LLMs for RAG pipelines, current approaches use instruction tuning to optimize LLMs, improving their ability to utilize retrieved knowledge. This supervised fine-tuning (SFT) approach focuses on equipping LLMs to handle diverse RAG tasks using different instructions. However, it trains RAG modules to overfit training signals and overlooks the varying data preferences among agents within the RAG system. In this paper, we propose a Differentiable Data Rewards (DDR) method, which end-to-end trains RAG systems by aligning data preferences between different RAG modules. DDR works by collecting the rewards to optimize each agent with a rollout method. This method prompts agents to sample some potential responses as perturbations, evaluates the impact of these perturbations on the whole RAG system, and subsequently optimizes the agent to produce outputs that improve the performance of the RAG system. Our experiments on various knowledge-intensive tasks demonstrate that DDR significantly outperforms the SFT method, particularly for LLMs with smaller-scale parameters that depend more on the retrieved knowledge. Additionally, DDR exhibits a stronger capability to align the data preference between RAG modules. The DDR method makes generation module more effective in extracting key information from documents and mitigating conflicts between parametric memory and external knowledge. All codes are available at https://github.com/OpenMatch/RAG-DDR .

## 1 Introduction

Large Language Models (LLMs) have demonstrated impressive capabilities in language understanding, reasoning, and planning capabilities across a wide range of natural language processing (NLP) tasks (Achiam et al., ; Touvron et al., ; Hu et al., ). However, LLMs usually produce incorrect responses due to hallucination (Ji et al., ; Xu et al., ).
To alleviate the problem, existing studies employ Retrieval Augmented Generation (RAG) (Lewis et al., ; Shi et al., ; Peng et al., ) to enhance the capability of LLMs and help LLMs access long-tailed knowledge and up-to-date knowledge from different data sources (Trivedi et al., ; He et al., ; Cai et al., ; Parvez et al., ). However, the conflict between retrieved knowledge and parametric memory usually misleads LLMs, challenging the effectiveness of RAG system (Li et al., ; Chen et al., ; Asai et al., ).

To ensure the effectiveness of RAG systems, existing research has focused on developing various agents to enhance the retrieval accuracy (Gao et al., ; Xu et al., ; Jiang et al., ; Xu et al., ). These approaches aim to refine retrieval results through query reformulation, reranking candidate documents, summarizing the retrieved documents or performing additional retrieval steps to find more relevant information (Yan et al., ; Trivedi et al., ; Asai et al., ; Yu et al., ). To optimize the RAG system, the methods independently optimize different RAG modules by using the EM method (Singh et al., ; Sachan et al., ) or build the instruct tuning dataset for Supervised Fine-Tuning (SFT) these LLM based RAG modules (Lin et al., ; Asai et al., ). However, these SFT-based methods usually train LLMs to overfit the training signals and face the catastrophic forgetting problem (Luo et al., ).

Current research further aims to optimize these RAG modules for aligning their data preferences and primarily focuses on optimizing a two-agent framework, which consists of a retriever and a generator. Typically, these systems train only the retriever to supply more accurate documents to satisfy the data preference of the generator (Shi et al., ; Yu et al., ). Aligning data preferences between the retriever and generator through refining retrieved knowledge is a straightforward approach to improving the effectiveness of RAG models. Nevertheless, the generator still faces the knowledge conflict, making LLMs not effectively utilize the retrieved knowledge during generation (Xie et al., ). Thus optimizing each RAG module by using the reward from the entire system during training, is essential for building a more tailored RAG system.

This paper introduces a Differentiable Data Rewards (DDR) method for end-to-end optimizing agents in the RAG system using the DPO (Rafailov et al., ) method. DDR uses a rollout method (Kocsis & Szepesvári, ) to collect the reward from the overall system for each agent and optimizes the agent according to the reward. Specifically, we follow Asai et al. () and build a typical RAG system to evaluate the effectiveness of our DDR model. It consists of a knowledge refinement module for selecting retrieved documents and a generation module for producing responses based on queries and refined knowledge. Then we conduct the RAG-DDR model by optimizing the two-agent based RAG system using DDR. Throughout the optimization process, we use the reward from the entire RAG system and iteratively optimize both the generation and knowledge refinement modules to align data preferences across both agents.

Our experiments on various Large Language Models (LLMs) demonstrate that Differentiable Data Rewards (DDR) outperforms all baseline models, achieving significant improvements over previous method (Lin et al., ) in a range of knowledge-intensive tasks. DDR can effectively retrofit LLMs for the RAG modeling and help generate higher-quality responses of an appropriate length. Our further analyses show that the effectiveness of our RAG-DDR model primarily derives from the generation module, which is optimized by the reward from the RAG system. Our DDR optimized generation module is more effective in capturing crucial information from retrieved documents and alleviating the knowledge conflict between external knowledge and parametric memory. Further analyses show that the effectiveness of DDR optimized RAG systems can be generalized even when additional noisy documents are incorporated during response generation.

## 2 Related Work

Retrieval-Augmented Generation (RAG) is widely used in various real-world applications, such as open-domain QA (Trivedi et al., ), language modeling (He et al., ), dialogue (Cai et al., ), and code generation (Parvez et al., ). RAG models retrieve documents from external corpus (Karpukhin et al., ; Xiong et al., ) and then augment the LLM’s generation by incorporating documents as the context for generation (Ram et al., ) or aggregating the output probabilities from the encoding pass of each retrieved documents (Shi et al., ). They help LLMs alleviate hallucinations and generate more accurate and trustworthy responses (Jiang et al., ; Xu et al., ; Luo et al., ; Hu et al., ; Kandpal et al., ). However, retrieved documents inevitably incorporate noisy information, limiting the effectiveness of RAG systems in generating accurate responses (Xu et al., ; ; Longpre et al., ; Liu et al., ).

Some studies have demonstrated that the noise from retrieved documents can mislead LLMs, sometimes resulting in degraded performance even on some knowledge-intensive tasks  (Foulds et al., ; Shuster et al., ; Xu et al., ). Such a phenomenon primarily derives from the knowledge conflict between parametric knowledge of LLMs and external knowledge (Jin et al., ; Longpre et al., ; Chen et al., ; Xie et al., ; Wu et al., ). Xie et al. () have demonstrated that LLMs are highly receptive to external evidence when external knowledge conflicts with the parametric memory (Xie et al., ). Thus, lots of RAG models focus on building modular RAG pipelines to improve the quality of retrieved documents (Gao et al., ). Most of them aim to conduct more accurate retrieval models by employing a retrieval evaluator to trigger different knowledge refinement actions (Yan et al., ), prompting LLMs to summarize the query-related knowledge from retrieved documents (Yu et al., ) or training LLMs to learn how to retrieve and utilize knowledge on-demand by self-reflection (Asai et al., ).

Optimizing RAG system is a crucial research direction to help generate more accurate responses. Previous work builds a RAG system based on pretrained language models and conducts an end-to-end training method (Singh et al., ; Sachan et al., ). They regard retrieval decisions as latent variables and then iteratively optimize the retriever and generator to fit the golden answers. Recent research primarily focuses on optimizing LLMs for RAG. INFO-RAG (Xu et al., ) focuses on enabling LLMs with the in-context denoising ability by designing an unsupervised pretraining method to teach LLMs to refine information from retrieved contexts. RA-DIT (Lin et al., ) builds a supervised training dataset and then optimizes the retriever and LLM by instruct tuning. However, these training methods focus on training LLMs to fit the training signals and face the issue of catastrophic forgetting during instruct tuning (Luo et al., ).

Reinforcement Learning (RL) algorithms (Schulman et al., ), such as Direct Preference Optimization (DPO) (Rafailov et al., ), are widely used to optimize LLMs for aligning with human preferences and enhancing the consistency of generated responses (Putta et al., ).
Agent Q integrates MCTS and DPO to allow agents to learn from both successful and unsuccessful trajectories, thereby improving their performance in complex reasoning tasks (Putta et al., ).
STEP-DPO further considers optimizing each inference step of a complex task as the fundamental unit for preference learning, which enhances the long-chain reasoning capabilities of LLMs (Lai et al., ). While these models primarily target the optimization of individual agents to improve response accuracy at each step, they do not focus on the effectiveness of data alignment within the multi-agent system. Instead of using SFT methods for RAG optimization (Lin et al., ), this paper focuses on using the DPO method to avoid overfitting the training signals and align data preferences across different agents, which is different from above RL-based optimization methods.

## 3 RAG Training with Differentiable Data Rewards

This section introduces the Differentiable Data Rewards (DDR) method. We first introduce the DDR method, which optimizes the agent system by aligning the data preferences between agents (Sec. 3.1). Then we utilize knowledge refinement and generation modules to build the RAG pipeline and utilize the DDR method to optimize agents in this system (Sec. 3.2).

### 3.1 Data Preference Learning with Differentiable Data Rewards

In a RAG system $\mathcal{V}=\{V_{1},\dots,V_{t},\dots,V_{T}\}$, agents exchange and communicate data. To optimize this system, we first forward-propagate data among agents and then evaluate the performance of the RAG system. Then we backward-propagate rewards to refine the data preferences of each agent.

Data Propagation. During communication, the $t$-th agent, $V_{t}$, acts both as a sender and a receiver. Agent $V_{t}$ receives data from agent $V_{t-1}$ and simultaneously passes data to agent $V_{t+1}$:

$$ $x\rightsquigarrow V_{1}\dots V_{t}\xrightarrow{y_{t}}V_{t+1}\dots V_{T-1} \xrightarrow{y_{T-1}}V_{T}\rightsquigarrow y_{T},$ (1) $$

where $V_{t}\xrightarrow{y_{t}}V_{t+1}$ denotes that the agent generates one response $y_{t}$ of the maximum prediction probability and sends it to the agent $V_{t+1}$. $x\rightsquigarrow$ and $\rightsquigarrow y_{T}$ represent sending the input $x$ to the agent system $\mathcal{V}$ and getting the output $y_{T}$ from $\mathcal{V}$. The performance of the agent system $\mathcal{V}$ can be evaluated by calculating the quality score $S(y_{T})$ of the final output $V_{T}$ of the agent system $\mathcal{V}$.

Differentiable Data Reward. Unlike conventional supervised fine-tuning approaches (Lin et al., ), DDR tries to optimize the agent $V_{t}$ to align the data preference $V_{t+1:T}$, making the agent system produce a better response with higher evaluation score $S(y_{T})$ of the whole RAG system.

To optimize the agent $V_{t}$, DDR aims to propagate the system reward to train the targeted agent. Specifically, we first instruct $V_{t}$ to sample multiple outputs $\tilde{y}_{t}$, which incorporate some perturbations into the agent system. Then we calculate the reward $r(x,\tilde{y}_{t})$ with a rollout process (Kocsis & Szepesvári, ). In detail, we regard the agents $\mathcal{V}_{t+1:T}$ as the evaluation model, feed $\tilde{y}_{t}$ to this subsystem $\mathcal{V}_{t+1:T}$ and calculate the evaluation score $S(y_{T})$ of the generated output $y_{T}$:

$$ $\tilde{y}_{t}\rightsquigarrow V_{t+1}\xrightarrow{y_{t}}V_{t+2}\dots V_{T-1} \xrightarrow{y_{T-1}}V_{T}\rightsquigarrow y_{T},\quad r(x,\tilde{y}_{t})=S(y_ {T}).$ (2) $$

Finally, we maximize the probability of generating $\tilde{y}_{t}^{+}$ over $\tilde{y}_{t}^{-}$, where $\tilde{y}_{t}^{+}$ wins higher reward than $\tilde{y}_{t}^{-}$ ($r(x,\tilde{y}_{t}^{+})>r(x,\tilde{y}_{t}^{-})$):

$$ $P(\tilde{y}_{t}^{+}>\tilde{y}_{t}^{-}|x)=\sigma(r(x,\tilde{y}_{t}^{+})-r(x, \tilde{y}_{t}^{-})),$ (3) $$

where $\sigma$ is the Sigmoid function. The parameters of $V_{t}$ can be trained using the DPO (Rafailov et al., ) training loss, aiding the DDR in identifying optimization directions by contrastively learning from the positive ($\tilde{y}_{t}^{+}$) and negative ($\tilde{y}_{t}^{-}$) outputs:

$$ $\mathcal{L}(V_{t};V_{t}^{\text{ref}})=-\mathbb{E}_{(x,\tilde{y}_{t}^{+},\tilde {y}_{t}^{-})\sim\mathcal{D}}[\log\sigma(\beta\log\frac{V_{t}(\tilde{y}_{t}^{+} \mid x)}{V_{t}^{\text{ref}}(\tilde{y}_{t}^{+}\mid x)}-\beta\log\frac{V_{t}( \tilde{y}_{t}^{-}\mid x)}{V_{t}^{\text{ref}}(\tilde{y}_{t}^{-}\mid x)})],$ (4) $$

where $\beta$ is a hyperparameter. $\mathcal{D}$ is the dataset containing the input $x$ and its corresponding preference data pairs $(\tilde{y}_{t}^{+},\tilde{y}_{t}^{-})$. $V_{t}^{\text{ref}}$ is the reference model, which is frozen during training.

Figure: Figure 1: The Illustration of End-to-End Retrieval-Augmented Generation (RAG) Training with Our Differentiable Data Reward (DDR) Method. During training, we iteratively optimize the Generation module ($V_{\text{Gen}}$) and Knowledge Refinement module ($V_{\text{KR}}$).
Refer to caption: extracted/5934510/image/model.png

### 3.2 Optimizing a Specific RAG System through DDR

Given a query $q$ and a set of retrieved documents $D=\{d_{1},\dots,d_{n}\}$, we build a RAG system by employing a knowledge refinement module ($V_{\text{KR}}$) to filter unrelated documents and a generation module ($V_{\text{Gen}}$) to produce a response. These two modules can be represented by a two-agent system:

$$ $\{q,D\}\rightsquigarrow V_{\text{KR}}\xrightarrow{\{q,\tilde{D}\}}V_{\text{Gen }}\rightsquigarrow y_{\text{Gen}},$ (5) $$

where $\tilde{D}\subseteq D$ and $V_{\text{KR}}$ produces the select actions to filter out the noise documents in $D$ to build $\tilde{D}$. $V_{\text{Gen}}$ generates the answer according to the query $q$ with filtered documents $\tilde{D}$. As shown in Figure 1, we iteratively tune different modules to conduct the RAG-DDR model, beginning with the generation module optimization and subsequently focusing on tuning the knowledge refinement module during training this RAG system. In the rest of this subsection, we will explain the details of how to optimize $V_{\text{KR}}$ and $V_{\text{Gen}}$ using DDR.

Knowledge Refinement Module. We follow Asai et al. () and build the knowledge refinement module $V_{\text{KR}}$ to estimate the relevance of each document $d_{i}$ to the query $q$ for refinement. We feed both query $q$ and document $d_{i}$ to the $V_{\text{KR}}$ and ask it to produce the action $y_{\text{KR}}^{i}\in\{\text{``YES''},\text{``NO''}\}$, which indicates whether $d_{i}$ is retained ($y_{\text{KR}}^{i}=\text{``YES''}$) or discarded ($y_{\text{KR}}^{i}=\text{``NO''}$):

$$ $y_{\text{KR}}^{i}=\text{LLM}(\text{Instruct}_{\text{KR}},d_{i}\oplus q),$ (6) $$

where $\oplus$ denotes the concatenation operation, and $\text{Instruct}_{\text{KR}}$ is a prompt designed for knowledge refinement. Then the refined document collection $\tilde{D}=\{d_{1},\dots,d_{k}\}$ is constructed, where $k<=n$. The document $d_{i}$ that leads the agent system to achieve the highest evaluation reward $r(x,y_{\text{KR}}^{i}=\text{``YES''})$ is considered positive, while the document $d_{j}$ that results in the lowest reward $r(x,y_{\text{KR}}^{j}=\text{``YES''})$ is regarded as negative. $V_{\text{KR}}$ is trained to maximize the probability $P(y_{\text{KR}}^{i}=\text{``YES''}>y_{\text{KR}}^{i}=\text{``NO''}|q,d_{i})$ for the positive document. As well as, we maximize the probability $P(y_{\text{KR}}^{j}=\text{``NO''}>y_{\text{KR}}^{j}=\text{``YES''}|q,d_{j})$ to filter out irrelevant documents.

Generation Module. After knowledge refinement, the query $q$ and filtered documents $\tilde{D}=\{d_{1},\dots,d_{k}\}$ are fed to the generation module $V_{\text{Gen}}$. The response $\tilde{y}_{\text{Gen}}$ is sampled from $V_{\text{Gen}}$:

$$ $\tilde{y}_{\text{Gen}}\sim\text{LLM}(\text{Instruct}_{\text{Gen}},d_{1}\oplus \dots\oplus d_{k}\oplus q),$ (7) $$

where $\text{Instruct}_{\text{Gen}}$ is a prompt for generating a tailored response. To reduce the misleading knowledge from the retrieved documents, we also sample responses using only the query as input:

$$ $\tilde{y}_{\text{Gen}}\sim\text{LLM}(\text{Instruct}_{\text{Gen}},q).$ (8) $$

The response that achieves the highest evaluation score $S(\tilde{y}_{\text{Gen}})$ is considered positive ($\tilde{y}_{\text{Gen}}^{+}$), while the lowest evaluation score is considered negative ($\tilde{y}_{\text{Gen}}^{-}$). The generation module $V_{\text{Gen}}$ is optimized to maximize the probability of generating the positive response $P(\tilde{y}_{\text{Gen}}^{+}>\tilde{y}_{\text{Gen}}^{-}|q,\tilde{D})$ to win a higher reward. By generating responses $\tilde{y}_{\text{Gen}}$ based on documents or the query alone, LLMs can learn to balance internal and external knowledge, alleviating the problems related to knowledge conflicts.

## 4 Experimental Methodology

This section first describes datasets, evaluation metrics, and baselines. Then we introduce the implementation details of our experiments. More experimental details are shown in Appendix A.1.

Dataset. In our experiments, we follow RA-DIT (Lin et al., ) and use the instruction tuning datasets for training and evaluating RAG models. For all datasets and all baselines, we use bge-large (Xiao et al., ) to retrieve documents from the MS MARCO 2.0 (Bajaj et al., ).

During the training of DDR, we collect ten datasets covering two tasks, open-domain QA and reasoning. Specifically, we randomly sample 32,805 samples for the training set and 2,000 samples for the development set in our experiments. Following previous work (Lin et al., ; Xu et al., ), we select the knowledge-intensive tasks for evaluation, including open-domain question answering, multi-hop question answering, slot filling, and dialogue tasks. The open-domain QA tasks consist of NQ (Kwiatkowski et al., ), MARCO QA (Bajaj et al., ) and TriviaQA (Joshi et al., ), which require models to retrieve factual knowledge to help answer the given question. For more complex tasks, such as multi-hop QA and dialogue, we use HotpotQA dataset (Yang et al., ) and Wikipedia of Wizard (WoW) (Dinan et al., ) for evaluation. Besides, we also employ T-REx (Elsahar et al., ) to measure one-hop fact look-up abilities of models.

Evaluation. Following Xu et al. (), we utilize Rouge-L and F1 as evaluation metrics for MARCO QA task and WoW task, respectively. For the rest tasks, we use Accuracy.

Baselines. In our experiments, we compare DDR with five baseline models, including zero-shot models and supervised finetuning models.

We first treat the LLM as a black box and conduct three baselines, including LLM w/o RAG, Vanilla RAG and REPLUG. For the LLM w/o RAG model, we directly feed the query to the LLM and ask it to produce the answer according to its memorized knowledge. To implement the vanilla RAG model, we follow previous work (Lin et al., ), use retrieved passages as context and leverage the in-context learning method to conduct the RAG modeling. REPLUG (Shi et al., ) is also compared, which ensembles output probabilities from different passage channels. Besides, RA-DIT (Lin et al., ) is also compared in our experiments, which optimizes the RAG system using the instruct-tuning method. In our experiments, we reimplement REPLUG and RA-DIT baselines and do not finetune the retriever during our reproduction process, as the retriever we used has already been trained with massive supervised data and is sufficiently strong.

Implementation Details. In our experiments, we employ Minicpm-2.4B-sft (Hu et al., ) and lama3-8B-Instruct (Touvron et al., ) as backbone models to construct the generation modules, and employ Llama3-8B-Instruct (Touvron et al., ) to build the knowledge refinement module. While training the RAG modules with DDR, we use automatic metrics such as Rouge-L and Accuracy to calculate the reward and set $\beta=0.1$. The learning rate is set to 5e-5, and each model is trained for one epoch. For the generation module, we feed 5 retrieved passages as external knowledge for augmenting the generation process. To optimize both knowledge refinement module and generation module, we use LoRA (Hu et al., ) for efficient training.

## 5 Evaluation Results

In this section, we first evaluate the performance of different RAG methods and then conduct ablation studies to show the effectiveness of different training strategies. Next, we examine the effectiveness of DDR training strategies on the generation module ($V_{\text{Gen}}$) and explore how it balances internal and external knowledge through DDR. Finally, we present several case studies.

### 5.1 Overall Performance

**Table 1: Overall Performance of Different RAG Models. The best and second best results are highlighted. In our experiments, we employ Llama3-8B as the knowledge refinement module and utilize LLMs of varying scales (Llama3-8B and MiniCPM-2.4B) as the generation module.**
| Method | Open-Domain QA | Multi-Hop QA | Slot Filling | Dialogue |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| NQ | TriviaQA | MARCO QA | HotpotQA | T-REx | WoW |  |
| MiniCPM-2.4B |  |  |  |  |  |  |
| LLM w/o RAG | 20.1 | 45.0 | 17.1 | 17.7 | 22.6 | 14.9 |
| Vanilla RAG () | 42.2 | 79.5 | 16.7 | 26.7 | 22.1 | 14.4 |
| REPLUG () | 39.4 | 77.0 | 19.4 | 24.7 | 26.7 | 15.7 |
| RA-DIT () | 41.8 | 78.6 | 19.6 | 26.1 | 25.2 | 15.7 |
| RAG-DDR (w/ 1-Round) | 47.0 | 82.7 | 28.1 | 32.5 | 32.6 | 17.2 |
| RAG-DDR (w/ 2-Round) | 47.6 | 83.6 | 29.8 | 33.2 | 32.1 | 18.2 |
| Llama3-8B |  |  |  |  |  |  |
| LLM w/o RAG | 35.4 | 78.4 | 17.0 | 27.7 | 16.0 | 14.6 |
| Vanilla RAG () | 46.2 | 84.0 | 20.6 | 30.1 | 26.9 | 12.5 |
| REPLUG () | 45.1 | 83.1 | 22.5 | 29.4 | 23.1 | 14.0 |
| RA-DIT () | 46.2 | 87.4 | 20.3 | 34.9 | 41.7 | 14.8 |
| RAG-DDR (w/ 1-Round) | 50.7 | 88.2 | 25.1 | 37.3 | 37.3 | 14.9 |
| RAG-DDR (w/ 2-Round) | 52.1 | 89.6 | 27.3 | 39.0 | 40.6 | 16.8 |

The performance of various RAG models is presented in Table 1. As shown in the evaluation results, RAG-DDR significantly outperforms these baseline models on all datasets. It achieves improvements of 7% compared to the Vanilla RAG model when using MiniCPM-2.4B and Llama3-8B to construct the generation module ($V_{\text{Gen}}$).

Compared with LLM w/o RAG, Vanilla RAG and REPLUG significantly enhance LLM performance on most knowledge-intensive tasks, indicating that external knowledge effectively improves the accuracy of generated responses. However, the performance of RAG models decreases on dialogue tasks, showing that LLMs can also be misled by the retrieved documents. Unlike these zero-shot methods, RA-DIT provides a more effective approach for guiding LLMs to filter out noise from retrieved content and identify accurate clues for answering questions. Nevertheless, RA-DIT still underperforms compared to Vanilla RAG on certain knowledge-intensive tasks, such as NQ and HotpotQA, showing that overfitting to golden answers is less effective for teaching LLMs to capture essential information for generating accurate responses. In contrast, RAG-DDR surpasses RA-DIT on almost all tasks, particularly with smaller LLMs (MiniCPM-2.4b), achieving a 5% improvement. This highlights the generalization capability of our DDR training method, enabling LLMs of varying scales to utilize external knowledge through in-context learning effectively.

**Table 2: Ablation Study. Both Vanilla RAG and RAG w/ $V_{\text{KR}}$ are evaluated in a zero-shot setting without any fine-tuning. We then use DDR to optimize the knowledge refinement module ($V_{\text{KR}}$), the generation module ($V_{\text{Gen}}$), and both modules, resulting in three models: RAG-DDR (Only $V_{\text{KR}}$), RAG-DDR (Only $V_{\text{Gen}}$) and RAG-DDR (All).**
| Method | Open-Domain QA | Multi-Hop QA | Slot Filling | Dialogue |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| NQ | TriviaQA | MARCO QA | HotpotQA | T-REx | WoW |  |
| MiniCPM-2.4B |  |  |  |  |  |  |
| Vanilla RAG | 42.1 | 78.0 | 16.6 | 24.9 | 22.0 | 15.1 |
| w/ $V_{\text{KR}}$ | 42.2 | 79.5 | 16.7 | 26.7 | 22.1 | 14.4 |
| RAG-DDR (Only $V_{\text{KR}}$) | 42.5 | 79.6 | 16.8 | 27.3 | 21.9 | 15.8 |
| RAG-DDR (Only $V_{\text{Gen}}$) | 46.8 | 81.7 | 28.3 | 31.2 | 32.2 | 17.0 |
| RAG-DDR (All) | 47.0 | 82.7 | 28.1 | 32.5 | 32.6 | 17.2 |
| Llama3-8B |  |  |  |  |  |  |
| Vanilla RAG | 45.4 | 83.2 | 20.8 | 28.5 | 26.6 | 12.3 |
| w/ $V_{\text{KR}}$ | 46.2 | 84.0 | 20.6 | 30.1 | 26.9 | 12.5 |
| RAG-DDR (Only $V_{\text{KR}}$) | 46.8 | 84.7 | 20.7 | 30.7 | 28.5 | 12.5 |
| RAG-DDR (Only $V_{\text{Gen}}$) | 50.2 | 87.8 | 25.2 | 36.9 | 36.2 | 14.8 |
| RAG-DDR (All) | 50.7 | 88.2 | 25.1 | 37.3 | 37.3 | 14.9 |

### 5.2 Ablation Studies

As shown in Table 2, we conduct ablation studies to explore the role of different RAG modules and evaluate different training strategies using DDR.

This experiment compares five models, utilizing MiniCPM-2.4b and Llama3-8b to construct the generation module. The Vanilla RAG model relies solely on the generation module ($V_{\text{Gen}}$) to produce answers based on the query and retrieved documents. RAG w/ $V_{\text{KR}}$ adds an additional knowledge refinement module ($V_{\text{KR}}$) to filter the retrieved documents and then feeds query and filtered documents to the generation module ($V_{\text{Gen}}$). RAG-DDR (Only $V_{\text{KR}}$) indicates that we tune the RAG w/ $V_{\text{KR}}$ model using DDR by only optimizing the knowledge refinement module ($V_{\text{KR}}$). RAG-DDR (Only $V_{\text{Gen}}$) only optimizes the generation module ($V_{\text{Gen}}$). RAG-DDR (All) optimizes both $V_{\text{KR}}$ and $V_{\text{Gen}}$.

Compared with the Vanilla RAG model, RAG w/ $V_{\text{KR}}$ improves the RAG performance on almost all evaluation tasks, demonstrating the effectiveness of the knowledge refinement module in improving the accuracy of LLM responses. In contrast, RAG-DDR (Only $V_{\text{Gen}}$) shows greater improvements over RAG w/ $V_{\text{KR}}$ than DDR (Only $V_{\text{KR}}$), indicating that the primary effectiveness of RAG-DDR comes from optimizing the generation module ($V_{\text{Gen}}$) through DDR. When we begin with the RAG-DDR (Only $V_{\text{Gen}}$) model and subsequently optimize the knowledge refinement module, the performance of RAG with $V_{\text{KR}}$ improves. It shows that filtering noise from retrieved documents using feedback from the generation module is effective, which is also observed in previous work (Yu et al., ; Izacard & Grave, ). However, the improvements from optimizing the knowledge refinement modules are limited, highlighting that enhancing the generation module’s ability to leverage external knowledge is more critical for the existing RAG system.

### 5.3 Characteristics of the Generation Module in RAG-DDR

In this experiment, we explore the characteristics of the generation module ($V_{\text{Gen}}$) by employing various training strategies, including zero-shot (Vanilla RAG), the SFT method (RA-DIT), and DDR (RAG-DDR). As illustrated in Figure 2, we present the performance of $V_{\text{Gen}}$ w/o RAG and $V_{\text{Gen}}$ w/ RAG. These experiments evaluate $V_{\text{Gen}}$’s ability to memorize knowledge and utilize external knowledge. Additionally, we report the average length of responses generated by $V_{\text{Gen}}$.

Figure: (a) $V_{\text{Gen}}$ w/o RAG.
Refer to caption: extracted/5934510/image/new_no_rag_leida.png

As shown in Figure 2(a), we compare the performance of the generation module that relies solely on internal knowledge of parametric memory. Compared to the Vanilla RAG model, RA-DIT demonstrates a decline in performance on the NQ and HotpotQA tasks. Such a phenomenon reveals that the model loses previously acquired knowledge while learning new information during SFT (Luo et al., ). In contrast, DDR not only outperforms the RA-DIT method but also achieves consistent improvements over the Vanilla RAG model across all evaluation tasks. This indicates that DDR can help learn more factual knowledge during training while also preventing the loss of previously memorized information through a reinforcement learning-based training approach. Then we feed retrieved documents to the generation module and show the generation performance in Figure 2(b). The evaluation results indicate that RA-DIT marginally outperforms the Vanilla RAG model, while RAG-DDR significantly improves generation accuracy by utilizing factual knowledge from the retrieved documents. Additional experiments showing the general capabilities of RAG-DDR are presented in Appendix A.4.

Finally, we show the average length of responses generated by $V_{\text{Gen}}$ in Figure 2(c). Compared to the Vanilla RAG model, the average length of responses generated by RA-DIT decreases significantly, indicating that the SFT training method tends to cause LLMs to overfit the supervised data. On the contrary, RAG-DDR shows a more similar length distribution with Vanilla RAG model, enabling the model to generate responses of a more appropriate length. It demonstrates that training LLMs to learn data preferences from generated responses can help align the output format of RAG models more closely with that of the original LLMs.

**Table 3: Experimental Results on Evaluating the Knowledge Usage Ability of the Generation Module ($V_{\text{Gen}}$) of Different RAG Models.**
| Method | Has-Answer | Miss-Answer | Internal Knowledge |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NQ | HotpotQA | T-REx | NQ | HotpotQA | T-REx | NQ | HotpotQA | T-REx |  |
| MiniCPM-2.4B |  |  |  |  |  |  |  |  |  |
| LLM w/o RAG | 27.6 | 26.7 | 36.6 | 2.6 | 11.7 | 4.1 | 100.0 | 100.0 | 100.0 |
| \hdashlineVanilla RAG | 59.1 | 51.7 | 36.8 | 1.1 | 9.1 | 2.5 | 71.1 | 70.1 | 60.9 |
| RA-DIT | 58.3 | 47.6 | 41.7 | 1.7 | 10.8 | 4.2 | 76.9 | 73.7 | 73.4 |
| RAG-DDR | 65.5 | 56.9 | 52.6 | 2.4 | 10.8 | 5.9 | 82.9 | 81.0 | 78.5 |
| Llama3-8B |  |  |  |  |  |  |  |  |  |
| LLM w/o RAG | 46.4 | 44.7 | 25.5 | 6.7 | 15.9 | 3.7 | 100.0 | 100.0 | 100.0 |
| \hdashlineVanilla RAG | 64.2 | 58.0 | 45.5 | 2.9 | 10.5 | 3.0 | 80.0 | 67.3 | 66.3 |
| RA-DIT | 64.1 | 59.7 | 65.3 | 3.4 | 17.9 | 10.8 | 81.0 | 79.2 | 77.9 |
| RAG-DDR | 69.5 | 64.3 | 59.9 | 4.1 | 18.0 | 5.4 | 88.4 | 82.0 | 79.2 |

Figure: (a) NQ.
Refer to caption: extracted/5934510/image/nq_noise.png

### 5.4 Effectiveness of RAG-DDR in Using External Knowledge

In this section, we investigate the capability of the generation module $V_{\text{Gen}}$ in the RAG model to leverage external knowledge for response generation. We first evaluate the ability of $V_{\text{Gen}}$ to balance the internal and external knowledge. Next, we evaluate the denoising ability by $V_{\text{Gen}}$ feeding additional unrelated documents as the external knowledge.

As shown Table 3, we first show the effectiveness of $V_{\text{Gen}}$ in balancing internal and external knowledge during producing responses. We compare three training strategies: zero-shot (Vanilla RAG), SFT (RA-DIT), and DDR (RAG-DDR). Our experiment establishes three testing scenarios to evaluate the effectiveness of different RAG models by categorizing the evaluation data into three distinct scenarios: Has-Answer, Miss-Answer, and Internal Knowledge. The Has-Answer scenario indicates that the retrieved documents contain the golden answer, which can help the generation module to answer the question accurately. The Miss-Answer scenario indicates that the retrieved documents do not contain the golden answer and fail to provide sufficient support for LLMs to generate accurate responses. Lastly, the Internal Knowledge scenario further evaluates the ability of LLMs in dealing with the conflict between internal and external knowledge. The test cases of the Internal Knowledge scenario indicate that LLMs can generate accurate answers using only parametric memory, whereas RAG models produce incorrect responses.

In the Has-Answer scenario, RAG-DDR and RA-DIT outperform the Vanilla RAG model on all datasets. This indicates that training the generation module enables it to capture essential knowledge facts, improving the accuracy of the generated responses. Compared with RA-DIT, RAG-DDR achieves consistent improvements over the Vanilla RAG model, which shows that DDR can better generalize the external knowledge usage ability to different knowledge-intensive tasks.
In the Miss-Answer scenario, all RAG models perform significantly worse compared to the Has-Answer scenario, showing that the retrieved knowledge fails to provide sufficient information for generating accurate responses. However, RAG-DDR and RA-DIT mitigate the performance drop caused by incorporating retrieved documents by fine-tuning the generation module, illustrating their ability to effectively leverage internal knowledge and avoid being misled by irrelevant information.
For the Internal Knowledge scenario, we evaluate the ability of RAG models in handling the conflict from external knowledge. The Vanilla RAG model decreases the generation accuracy more than 20% on all tasks, showing that the knowledge conflict can significantly affect the generation result. DDR exhibits strong effectiveness in mitigating the knowledge conflict in the Vanilla RAG model, resulting in a reduction of more than 10% in performance drop. This indicates that DDR effectively balances the utilization of internal and external knowledge, facilitating the robustness of the RAG model.

In the second experiment, we extend the Has-Answer setting and further investigate the denoising effectiveness of $V_{\text{Gen}}$ by adding different number of noise documents. As shown in Figure 3, we increase the noise by randomly replacing $n$ documents from the top-5 retrieved set with the last $n$ documents from the top-100 retrieved candidates, while ensuring that the ground truth document remains in the set. RA-DIT exhibits inconsistent performance and degrades in some RAG scenarios. In contrast, RAG-DDR consistently outperforms the Vanilla RAG model, maintaining a consistent improvement even as the number of noisy documents increases. It further confirms the effectiveness of our RAG-DDR approach in defencing noise.

**Table 4: Case Studies. These matching phrases are indicated in Red if the model answers correctly and in Blue if it answers incorrectly. We present the generated responses of different RAG models, which use MiniCPM-2.4B to build the generation module ($V_{\text{Gen}}$).**
| Case 1: When does Canada switch to daylight savings time<br>Document: Daylight Saving Time Canada. Daylight Saving Time in Canada. In most of Canada Daylight<br>Saving Time begins at 2:00 a.m. local time on the second Sunday in March. … Canada Daylight Saving<br>Time Start and End Dates 2020 March 8 November 1 2019 March 10 November … November 6 2010<br>March 14 November 7 2009 March 08 November 1 …<br>Vanilla LLMs: Canada does not switch to or from daylight savings time. Instead, the clocks are<br>automatically set forward by one hour during the spring.<br>Vanilla RAG: March 14      RA-DIT: March 8<br>RAG-DDR: Canada switches to Daylight Saving Time on the second Sunday of March and returns to<br>Standard Time on the first Sunday of November. |
| --- |
| Case 2: The manager who recruited David Beckham managed Manchester United during what timeframe<br>Document: Ferguson’s unprecedented achievements at Aberdeen led to managerial offers from some of the<br>most prestigious clubs in Europe over the years, and he signed with Manchester in November 1986. …<br>Ferguson retired at the end of the 2012–13 Premier League season but stayed on with Man U in a…<br>Vanilla LLMs: 1992–2003      Vanilla RAG: 1986–1993      RA-DIT: 1992 to 2013<br>RAG-DDR: The football manager who recruited David Beckham was Sir Alex Ferguson, and he<br>managed Manchester United from 1986 to 2013. |
| Case 3: Following success at the 1979 election whose party conference speeech included the lines<br>’you turn if you want to, the lady’s not for turning’?<br>Document: The lady’s not for turning - Wikipedia The lady’s not for turning From Wikipedia, the free<br>encyclopedia Jump to navigation Jump to search 1980 Margaret Thatcher speech … a phrase used by<br>Margaret Thatcher, then Prime Minister, in her speech to the Conservative Party Conference…<br>Vanilla LLMs: The speaker in question is Margaret Thatcher, who was the leader of the<br>Conservative Party and later became the Prime Minister of the United Kingdom.<br>Vanilla RAG: Conservative Party Conference RA-DIT: The Conservative Party<br>RAG-DDR: The 1979 Conservative Party conference speech by Margaret Thatcher included the lines<br>”you turn if you want to, the lady’s not for turning”. |

### 5.5 Case Studies

In Table 4, we present three randomly selected examples from the NQ, HotpotQA, and TrivialQA datasets to show the generated responses and evaluate the effectiveness of the RAG-DDR model.

For the first case, the query asks about the “daylight savings time of Canada” and the retrieved documents contain detailed information about when the daylight savings time begins each year in Canada. However, since the exact date of daylight savings time changes annually, the most accurate answer is “the second Sunday in March”. RAG-DDR shows its effectiveness in accurately answering the question, while both Vanilla RAG and RAG-DIT are misled by noisy information such as “14th March” and “10th March”, leading to incorrect responses. This demonstrates that RAG-DDR model has the ability to distinguish the most accurate knowledge from ambiguous or misleading information in the retrieved documents. In the second case, the model must integrate multiple pieces of knowledge from the provided documents to answer the question. While the Vanilla RAG model and RAG-DIT only correctly answer half of the questions using partial knowledge, RAG-DDR successfully identifies the correct start time and end time. This indicates that RAG-DDR has a stronger capacity to integrate factual knowledge from different document segments. As shown in the third case, Vanilla LLM can answer the question correctly only depending on the parametric memory. Nevertheless, both Vanilla RAG and RA-DIT are misled by the confusing information from these retrieved documents and generate the response “the Conservative Party Conference”, which is entirely unrelated to the given question. In contrast, RAG-DDR accurately follows the intent of the question for generating the response, demonstrating the ability of RAG-DDR to mitigate the negative influence of external knowledge.

## 6 Conclusion

This paper proposes Differentiable Data Rewards (DDR), a method aimed at end-to-end optimizing the Retrieval-Augmented Generation (RAG) model using the DPO method. DDR optimizes each agent by collecting the reward in a rollout way and aligns data preferences among these communicative agents. We build a two-agent RAG system and optimize it using DDR to implement the RAG-DDR model. Our experiments demonstrate that DDR helps the generation module produce responses of an appropriate length and avoids overfitting the training signals during SFT. Our further analyses reveal that the DDR optimized generation model can better capture key information from retrieved documents and mitigate the conflict between external knowledge and parametric memory.

## Appendix A Appendix

**Table 5: Data Statistics.**
| Split | Task | Dataset | Metric | Total |
| --- | --- | --- | --- | --- |
| Training | Open-Domain QA | Commonsense QA () | Accuracy | 4,200 |
| Math QA () | Accuracy | 4,200 |  |  |
| Web Questions () | Accuracy | 3,778 |  |  |
| Wiki QA () | Rouge-L | 1,040 |  |  |
| Yahoo! Answers QA | Rouge-L | 4,200 |  |  |
| MARCO QA () | Rouge-L | 4,200 |  |  |
| Reasoning | Algebra QA with Rationales () | Accuracy | 2,727 |  |
| Explanations for CommonsenseQ () | Accuracy | 4,200 |  |  |
| Grade School Math 8K () | Accuracy | 4,200 |  |  |
| StrategyQA () | Accuracy | 2,060 |  |  |
| Evaluation | Open-domain QA | Natural Questions () | Accuracy | 2,837 |
| TriviaQA () | Accuracy | 5,359 |  |  |
| MARCO QA () | Rouge-L | 3,000 |  |  |
| Multi-Hop QA | HotpotQA () | Accuracy | 5,600 |  |
| Slot Filling | T-REx () | Accuracy | 5,000 |  |
| Dialogue | Wizard of Wikipedia () | F1 | 3,000 |  |

### A.1 Additional Experimental Details

In this subsection, we first outline the process of constructing the training data. Then we show the prompt templates used in our experiments. In experiments, we keep the same training data, hyperparameters, and epochs to implement both RA-DIT and RAG-DDR for a fair comparison.

Data Preprocessing for DDR. The quantity of our training and evaluation data, along with the corresponding evaluation metrics, are presented in Table 5. Then we describe the details of data preprocessing during training both knowledge refinement ($V_{\text{KR}}$) and generation ($V_{\text{Gen}}$) modules using our DDR method.

Optimizing the knowledge refinement module is a rollout process, which individually feeds the top-100 retrieved documents to the generation module ($V_{\text{Gen}}$) with query to calculate the reward. Then, we apply the evaluation metrics shown in Table 5 to calculate the reward scores, identifying the documents that result in the highest scores as positive documents and those with the lowest scores as negative ones. Finally, we generate the triplet data $\{\text{query},\text{positive/negative documents},\text{``YES''/``NO''}\}$. To construct the dataset for training the generation module, we concatenate the refined documents with the query and then fed them into the generation module for sampling responses. We apply five different temperature settings (0.5, 0.6, 0.7, 0.8, 0.9) to sample responses and conduct the five-round sampling for each temperature. Afterward, we compute the reward score for each output using evaluation metrics to identify the positive and positive responses for each query. This process yields the triplet data $\{\text{query},\text{positive response},\text{negative response}\}$.

Figure: (a) MiniCPM-2.4B.
Refer to caption: extracted/5934510/image/generation_minicpm.png

Figure: Figure 5: Prompts Used for Knowledge Refinement.
Refer to caption: extracted/5934510/image/judgement_data.png

Prompt Templates. For RA-DIT, we use the same instruction tuning template as Lin et al. () and leverage top-5 retrieved documents as augmented knowledge during training. Then we describe the prompt templates used in RAG-DDR. As shown in Figure 4, we refer to the prompt designs of RA-DIT (Lin et al., ) and Self-RAG (Asai et al., ) to conduct tailored task prompts for different LLMs, helping LLMs generate better responses. In addition, we design separate prompts for LLMs w/ RAG and LLMs w/o RAG:{Background:{Documents}\n {Instruction}} and {Instruction}, where Documents indicates the external documents provided for LLMs and Instruction represents the task instructions. As illustrated in Figure 5, we design the prompts for the knowledge refinement tasks by referring to LangChain(^1^11[https://github.com/langchain-ai/langchain](https://github.com/langchain-ai/langchain)), enabling LLMs to correctly generate “YES” or “NO” to retain or discard retrieved documents.

**Table 6: Additional Ablation Study Results on RAG-DDR.**
| Method | Open-Domain QA | Multi-Hop QA | Slot Filling | Dialogue |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| NQ | TriviaQA | MARCO QA | HotpotQA | T-REx | WoW |  |
| MiniCPM-2.4B |  |  |  |  |  |  |
| Independent Tuning | 46.6 | 82.0 | 28.0 | 32.3 | 31.7 | 17.3 |
| RAG-DDR ($V_{\text{KR}}$ First) | 47.4 | 83.2 | 27.2 | 33.3 | 35.0 | 17.3 |
| RAG-DDR ($V_{\text{Gen}}$ First) | 47.0 | 82.7 | 28.1 | 32.5 | 32.6 | 17.2 |
| Llama3-8B |  |  |  |  |  |  |
| Independent Tuning | 49.9 | 88.3 | 25.1 | 37.4 | 37.0 | 14.9 |
| RAG-DDR ($V_{\text{KR}}$ First) | 50.4 | 88.6 | 25.6 | 36.9 | 36.7 | 14.9 |
| RAG-DDR ($V_{\text{Gen}}$ First) | 50.7 | 88.2 | 25.1 | 37.3 | 37.3 | 14.9 |

### A.2 Additional Ablation Studies on RAG-DDR

As shown in Table 6, we conduct additional ablation studies to explore the effectiveness of different training strategies: Independent Tuning, RAG-DDR ($V_{\text{KR}}$ First) and RAG-DDR ($V_{\text{Gen}}$ First). Independent Tuning indicates that the knowledge refinement module $V_{\text{KR}}$ and generation module $V_{\text{Gen}}$ are trained independently. RAG-DDR ($V_{\text{KR}}$ First) and RAG-DDR ($V_{\text{Gen}}$ First) are cascaded optimization models, indicating that we first train $V_{\text{KR}}$ or $V_{\text{Gen}}$ and subsequently optimize the other module by initializing the RAG model with the already optimized module.

Compared to Independent Tuning, the effectiveness of RAG-DDR is enhanced on all tasks. It shows that Independent Tuning results in misalignment of data preferences between the optimized modules, thereby impacting the overall performance of the RAG system. In contrast, the performance of RAG-DDR ($V_{\text{KR}}$ First) and RAG-DDR ($V_{\text{Gen}}$ First) is indistinguishable across different datasets and $V_{\text{Gen}}$, demonstrating that the DDR method is robust to the training orders.

Figure: (a) The Accuracy of Knowledge Refinement.
Refer to caption: extracted/5934510/image/refinement_acc.png

### A.3 Characteristics of The Knowledge Refinement Module of RAG-DDR

As shown in Figure 6, we explore the characteristics of the knowledge refinement module ($V_{\text{KR}}$) optimized using different methods, including Vanilla RAG, RA-DIT and RAG-DDR.

For RA-DIT, we collect 30k pieces of data from the MARCO QA dataset to tune $V_{\text{KR}}$ where each query has labeled positive and negative documents. We sample one positive document and one negative document for each query to construct the SFT dataset containing 60k pieces of data and fine-tune $V_{\text{KR}}$. The number of the SFT dataset is consistent with the amount of the RAG-DDR training dataset. The SFT dataset consists of triples in the form of $\{\text{query},\text{positive/negative documents},\text{``YES''/``NO''}\}$.

As shown in Figure 6(a), we calculate the accuracy of top-5 documents by Vanilla RAG, RA-DIT, and RAG-DDR. The retrieval accuracy evaluates whether or not the retained top-5 documents contain the ground truth. If $V_{\text{KR}}$ discards all the documents retrieved, the accuracy is 0. RAG-DDR outperforms Vanilla RAG and RA-DIT on all tasks. It indicates that the DDR method can make $V_{\text{KR}}$ accurately retain documents that contain the necessary knowledge to answer the query. As shown in Figure 6(b), we use the refined documents by different knowledge refinement module ($V_{\text{KR}}$) to augment the DDR trained generation module ($V_{\text{Gen}}$). DDR-RAG also outperforms other models, indicating that DDR can help better align data preferences between $V_{\text{Gen}}$ and $V_{\text{KR}}$.

Figure: (a) Llama3-8B.
Refer to caption: extracted/5934510/image/llama_leida.png

### A.4 The General LLM Ability of DDR Optimized Generation Module

In this experiment, we further explore the characteristics of the generation module ($V_{\text{Gen}}$) optimized using different methods, zero-shot (LLM w/o RAG) and DDR (RAG-DDR).

As shown in Figure 7, we compare the general ability of the generation module on several aspects: Mathematical (Liu et al., ), Disciplinary Knowledge (Hendrycks et al., ), World Knowledge (Kwiatkowski et al., ), Logical Reasoning (Suzgun et al., ), and Common Sense Reasoning (Clark et al., ). These tasks are commonly used as benchmarks to assess the model’s inherent capabilities (Touvron et al., ; Hu et al., ).

As shown in Figure 7(a), DDR enables Llama3-8B to maintain its strong language understanding and knowledge reasoning capabilities. The performance of MiniCPM-2.4B is shown in Figure 7(b). The evaluation results show that DDR significantly enhances the performance of the smaller parameter models, MiniCPM-2.4B, particularly in mathematical and common sense reasoning tasks. It illustrates that DDR not only preserves the original capabilities of LLMs but also offers some potential for enhancing their performance.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please address the questions in Weaknesses.
soundness: 4
strengths: === Strengths ===

S1. The motivation of this work is clear, and the paper is easy to follow.

S2. The idea of introducing DPO loss to train the content filter and LLM generator in RAG is novel (although its significance is not quite clear given the current empirical study, see Weakness 1).

S3. The main comparison between the proposed method and baseline methods, along with the ablation study, is evaluated on a wide range of knowledge-intensive tasks.
summary: This paper focuses on improving the retrieval-augmented generation (RAG) system by fine-tuning the content filter and the LLM generator while keeping the retriever frozen. It claims that existing supervised fine-tuning (SFT) approaches fail to model data preferences for different modules in the RAG system (i.e., the filter and generator) and tend to overfit training signals. In this work, the authors introduce a training method called differentiable data rewards (DDR), which uses DPO loss to train both the filter and generator. Experiments on six knowledge-intensive tasks show superior performance compared to several off-the-shelf RAG baselines and a fine-tuned RAG baseline (though without the filter module to denoise the retrieved contents).
weaknesses: === Weaknesses ===

W1. Potentially problematic experimental setting. The main claimed contribution is proposing a DPO loss to train the content filter and LLM generator. However, in the experiments, the proposed method is primarily compared to off-the-shelf RAG methods without any training. The only fine-tuned baseline is RA-DIT, which only trains the LLM generator. Does RA-DIT also leverage the content filter module during evaluation? If not, it is unclear whether the improved evaluation performance is due to filtering or to the better optimization by DPO loss compared to traditional SFT loss. Furthermore, is “RAG w/ $V_{KR}$” the only RAG baseline that includes a content filter (i.e., knowledge refinement) module? The comparison in the main table and ablation study may be unfair and the results can be misleading if the proposed RAG-DDR can benefit from an additional filtering mechanism that other baselines lack of.

W2. Missing comparisons and discussion with closely relevant RAG baselines on denoising retrieved content. Approaches for mitigating conflicts between external retrieved knowledge and the LLM’s internal parametric knowledge, as well as for denoising retrieved content, have been widely explored in RAG literature [1,2,3,4]. A more thorough discussion of these works is needed to better situate this work. For instance, [2] directly trains the LLM generator using SFT loss with noisy inputs to improve robustness to noise without any filtering/knowledge refinement, [3] employs self-synthetic rationales to denoise retrieved content and guide generation for better accuracy, which is also trained with SFT loss, and [4] introduces an inference-time algorithm to deal with irrelevant retrieved documents through a mechanism called isolation-then-aggregation. Without a detailed comparison with such related works, it is difficult to assess the empirical significance of this work, especially since some of these works are strong RAG methods trained with SFT loss (e.g., [2,3]) for specific denoising purposes.

**References**

[1]. AstuteRAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models. arXiv 2024.

[2]. Making Retrieval-Augmented Language Models Robust to Irrelevant Context. ICLR 2024.

[3]. InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales. arXiv 2024.

[4]. Certifiably Robust RAG against Retrieval Corruption. arXiv 2024.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What distinguishes the proposed DDR method from step-DPO? Is that the only difference comes from the base policy, i.e., in DDR the base policy at each step comes from different agents while for step-DPO the base policy at each step are shared?
2. What is the impact of training RAG-DDR iteratively over multiple rounds? While results for 2-round training are reported in Table 1, a more detailed analysis of the outcomes would be beneficial.
3. In section 5.3, the authors argue that "SFT training method tends to cause LLMs to overfit the supervised data" from the observation that RA-DIT's generated responses is significantly shorter than vanilla RAG and RAG-DDR's responses. While I generally agree with this statement, I wonder whether this phenomenon could be attributed to differences in the average response length within the training sets for RA-DIT and RAG-DDR. If so, what would occur if we trained RA-DIT using the positive samples from RAG-DDR? Would this adjustment also impact the model's accuracy?
soundness: 3
strengths: 1. Extensive experiments together with detailed analysis. In particular, the result from Table 3 is interesting, as it shows RAG-DDR's ability to balance between using internal and external knowledge.
2. The overall methodology is straightforward yet effective, leveraging a DPO-based approach to facilitate self-training.
summary: This paper introduces RAG-DDR, a novel reinforcement learning-based approach designed to improve retrieval-augmented generation (RAG) systems. The authors conceptualize the RAG framework as comprising two key modules: a knowledge refinement module, $V_{\text{KR}}$​, and a generation module, $V_{\text{Gen}}$. RAG-DDR computes partial reward for each module's response with a rollout process. Preference pairs are then constructed based on the rewards and then being used to optimize $V_{\text{Gen}}$ and $V_{\text{KR}}$ through DPO. Specifically, for $V_{\text{Gen}}$ (with filtered documents), the response with highest score is considered positive and lowest score is considered negative. For $V_{\text{KR}}$, the document that, when passed through $V_{\text{Gen}}$, yields the highest response score is considered positive, while the document with the lowest score is negative.

The authors perform extensive experiments across various knowledge-intensive tasks to evaluate RAG-DDR, comparing it against the vanilla RAG and SFT-based methods, showing that RAG-DDR can generate more accurate response and has the ability to leverage LLM's internal knowledge while not being misled by the external noisy information. Ablation studies further demonstrate the effectiveness of optimizing both $V_{\text{KR}}$ and $V_{\text{Gen}}$ using the proposed DDR approach.
weaknesses: 1. Some aspects are not clear, see the question section.
2. (Minor thing) Typo at line 850 "identify the positive and positive responses for each query." -> "identify the positive and negative responses for each query."

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Kindly refer to the Weakness section.

2. Could the authors define what is meant by “data preferences between different RAG modules”? This concept appears multiple times in the paper, yet it remains somewhat unclear to me. Given that RAG primarily addresses knowledge-based tasks, could “preferences” here be more aligned with factual accuracy? I would appreciate a brief definition, ideally illustrated with a case study that demonstrates why this issue is important. It would be especially helpful if different base models were involved, as their internal knowledge may vary, potentially influencing their data preferences.

3. In addition, regarding the rollout method:

- What are the training speed and computational requirements? How does the method scale with larger datasets in terms of efficiency?
- Could the DDR method be expanded beyond just two agents? A more in-depth discussion from the authors on this point would offer valuable insights for developing more complex RAG systems.

4. An interesting finding in Appendix A.4 is that DDR enhances MiniCPM-2.4B’s mathematical capabilities. Could the authors provide a brief analysis explaining the possible reasons for this?

5. Typo: Line 269, should be "Llama3-8B-Instruct".
soundness: 3
strengths: - Overall, the presentation is good and the method is easy-to-follow.
- The paper introduces the idea of DDR, which enables end-to-end optimization of RAG systems. The proposed method of aligning data preferences between knowledge refinement and generation agents alleviates the robustness issue when encountering external knowledge conflicts.
- The experimental results show that RAG systems trained with DDR strategy demonstrate improvements in terms of knowledge usage and robustness against knowledge conflicts.
summary: This paper introduces Differentiable Data Rewards (DDR), a novel method for training Retrieval-Augmented Generation (RAG) systems. DDR collects rewards via a rollout method to optimize each agent while aligning data preferences between different agents within the system. The method addresses two key challenges in RAG systems: (1) training signal overfitting and (2) poor robustness against conflicting knowledge.
weaknesses: - Figure 1 is not clear enough. It could be better to illustrate the correspondence between the left and right parts, as well as the order of two modules in the right part (could use step 1, 2 as annotations). Additionally, it would be helpful to indicate how the reward signal propagates throughout the entire training process.

 
- I realize that training large language models is resource-intensive, but the demonstrations in the paper could be more convincing by including:

    - (If possible)Additional experimental results from larger-scale LLMs, such as Qwen2.5-14B or other larger models.
    - (If possible)Additional evaluations using more different LLM combinations to compose the RAG system (It seems that the current setup consider Llama3-8B-Instruct as knowledge refinement module and MiniCPM-2.4B/Llama3-8B-Instruct as generator)

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Suggestion: I would encourage the authors to perform the ablation studies and related work comparisons suggested in the "Weaknesses" section.

Some questions about the KR module:
- The KR module generates binary Yes/No responses for the relevance of each document to the query. How is the binary output used to rank documents and identify the top-k documents?
- Line 216 says:  "document di that leads the agent system to achieve the highest evaluation reward is considered positive". Does this mean a single document is selected and fed to the generator module? But equation 7 shows that k documents are fed. Am I missing something?
- If it is a single document then when tuning the KR module do we consider the highest-rewards document as positive, and all others as negative? 
- Would it make sense to consider contrastive objectives like ensuring P(Yes | q, d_i) > P(Yes | q, d_j) whenever the reward fetched by d_i is higher than d_j?
soundness: 3
strengths: The paper addresses an important concern in RAG system of balancing use of external information versus internal parametric knowledge 
of the generation module. This is common problem for production RAG systems where retrieval is never perfect and often suffers from conflicting and/or missing information.

The idea of jointly tuning retrieval and generation modules is interesting and not studied as much in the literature (although some prior art does exist; see next section).

The results in Sections 5.3 and 5.4 are compelling and show a nice advancement over baseline methods in dealing with noisy retrieval in RAG systems.
summary: The paper proposes the DDR framework for tuning agentic systems. The key idea is to sample multiple actions from an agent, and score them based on the downstream reward they fetch (computed via unrolling all subsequent agents in the chain). The actions are ordered by this score, and the DPO method is used to increase the likelihood of high scoring actions over low scoring actions.

The framework is applied to optimize a two agent RAG system consisting of a knowledge refinement agent (selecting relevant documents for RAG), and a generation agent that generates the response based on the selected document.

Through experiments and case studies, the authors show that the framework can perform better than several baselines, and can be better balance internal parametric knowledge with external information.
weaknesses: The main weakness of the paper is inadequate comparison with related approaches on jointly tuning retriever and generator modules, which make it hard to appreciate the technical contribution of the paper. 

There is prior work on supervising the retrieval module of RAG system based on performance of the generator (e.g., Wang et al. 2023, https://arxiv.org/pdf/2311.08377). In my eyes, the novelty of this paper is the use of DPO instead of a supervised tuning (which would just maximize likelihood of action fetching highest reward). Thus, if I understand correctly, the novelty of this work is the negative gradient induced by the contrastive DPO loss. However, this is not discussed in the paper. It would help to study the impact of the negative gradient in depth, via ablation studies. For instance, is the negative gradient responsible for the claim in Section 5.3: "DDR can help learn more factual knowledge during training while also preventing the loss of previously memorized information".

The paper highlights RAG-DDR's ability to effectively combine parametric knowledge with external knowledge. To assess this, it would be helpful to see a comparison with adaptive approaches that use different retrieval modules (including no retrieval) for different queries. For instance, see: Jeong et al., 2024: https://arxiv.org/pdf/2403.14403

Other helpful ablations and baselines to experiment with:
* Section 3.2 notes that RAG-DDR tunes the generation module first followed by the knowledge retriever (KR) module. It would be helpful to perform ablations where the KR module is tuned before the generation module (as in Wang et al,. 2023).

* When evaluating vanilla LLM and vanilla RAG, it would help to experiment with large LLMs like Gemini-1.5-pro, GPT4o to understand if the inability to balance parametric knowledge vs. external knowledge is an artifact of smaller LLMs.

The paper introduces the framework of differentiable data rewards (DDR). However, the framework is only evaluated for simple two agent systems. Consequently, DDR does not come across as a fully developed contribution. Instead, the key contribution, at least to me, is the use of DPO in jointly trying retrieval and generation modules. I would encourage the authors to build on this aspect.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: NA
soundness: 3
strengths: The approach is novel to RAG optimization. By using differentiable rewards, it effectively addresses the knowledge conflict problem that often plagues RAG systems.
summary: This paper introduces RAG-DDR (Retrieval-Augmented Generation with Differentiable Data Rewards), a novel method for optimizing RAG systems using reinforcement learning. Rather than using traditional supervised fine-tuning approaches, RAG-DDR uses a reward-based learning system to optimize both the knowledge retrieval and generation components. The method specifically focuses on aligning data preferences between different modules in the RAG pipeline to improve overall system performance, particularly in handling conflicts between parametric and external knowledge.
weaknesses: 1, The computational cost is high. RAG-DDR requires significantly more computational resources than traditional methods, requiring about 5x more API calls than standard approaches. This could limit its practical applicability in resource-constrained settings.
2, The paper also shows some limitations in complex reasoning tasks, particularly in multi-hop reasoning scenarios where the model can sometimes prematurely conclude questions are invalid. While the method shows strong performance on factual question-answering tasks, its effectiveness on more complex reasoning tasks could be improved.
3, while the paper demonstrates strong empirical results, it could benefit from a more thorough theoretical analysis of why the method works so well, particularly in terms of how it achieves better alignment between different RAG components.

Meta-review: This paper introduces a new method for end-to-end training of Retrieval-Augmented Generation (RAG) systems using a technique with DPO loss. The authors train both a generator and a filter component, allowing the system to effectively identify and utilize relevant knowledge for generating accurate answers. Extensive experiments demonstrate the superiority of this approach compared to existing alternatives.

Strengths:
- The paper is well-written and the empirical results are comprehensive and convincing.
- The authors address a significant limitation in current RAG systems.

Weaknesses:
- The paper's positioning within the rapidly evolving landscape of end-to-end RAG tuning research is not entirely clear.
- The computational complexity of the training process could be a potential drawback.

Despite some weaknesses, the novel methodology, compelling empirical results, and the paper's focus on an important problem in RAG systems lead me to recommend acceptance.

Decision: Accept (Poster)