GainRAG: Preference Alignment in Retrieval-Augmented Generation through Gain Signal Synthesis
Authors: Yi Jiang, Sendong Zhao, Jianbo Li, Haochun Wang, Bing Qin
Accepted at: acl25

GainRAG: Preference Alignment in Retrieval-Augmented Generation through Gain Signal Synthesis

Yi Jiang, Sendong Zhao*, Jianbo Li, Haochun Wang, Bing Qin
Research Center for Social Computing and Interactive Robotics,
Harbin Institute of Technology, China
{yjiang,sdzhaoj,jbli,hcwang,qinb}@ir.hit.edu.cn

Abstract

The Retrieval-Augmented Generation (RAG) framework introduces a retrieval module to dynamically inject retrieved information into the input context of large language models (LLMs), and has demonstrated significant success in various NLP tasks. However, the current study points out that there is a preference gap between retrievers and LLMs in the RAG framework, which limit the further improvement of system performance. Some highly relevant passages may interfere with LLM reasoning because they contain complex or contradictory information; while some indirectly related or even inaccurate content may help LLM generate more accurate answers by providing suggestive information or logical clues. To solve this, we propose GainRAG, a novel approach that aligns the retriever’s and LLM’s preferences by defining a new metric, “gain”, which measure how well an input passage contributes to correct outputs. Specifically, we propose a method to estimate these gain signals and train a middleware that aligns the preferences of the retriever and the LLM using only limited data. In addition, we introduce a pseudo-passage strategy to mitigate degradation. The experimental results on 6 datasets verify the effectiveness of GainRAG1.

1 Introduction

Large Language Models (LLMs) (Achiam et al., 2023; Touvron et al., 2023) perform well in processing natural language tasks, but their knowledge is fixed in model parameters and is difficult to update dynamically over time (Ji et al., 2023; He et al., 2022). To tackle this issue, the Retrieval Augmented Generation (RAG) framework adds a retrieval module that brings in relevant external knowledge and integrates it into the input context of the LLMs. This approach has shown impressive results across various natural language processing tasks (Gao et al., 2023; Lewis et al., 2020). Previous work is devoted to solving two problems, namely, retrieving more relevant information in retrieval and effectively utilizing context to generate the correct answer in generation. However, they fail to address the preference gap between the retriever and the LLMs. Recent studies have highlighted that while retrieved passages may be relevant, they are not necessarily preferred for generation. In other words, only passages that align with the LLM’s preferences can provide meaningful gain and enhance generation performance.

Specifically, existing retrievers are usually designed based on human-defined relevance criteria, such as whether a passage directly contains the answer to a question (Ke et al., 2024). However, this approach does not fully align with the way LLMs process information. Some highly relevant passages may actually disrupt LLMs reasoning by introducing complexity or contradictions, while some seemingly unrelated or even partially incorrect content can help by offering useful hints or

<table>
  <tr>
    <th colspan="2">Passages w/ Answers: Correct vs. Incorrect Outputs</th>
    <th>recall—incorrect</th>
    <th>recall—correct</th>
  </tr>
  <tr>
    <td>47.84%</td>
    <td>52.16%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>2WikiMultiHopQA</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>38.63%</td>
    <td>61.37%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <th colspan="2">Correct Outputs: Passages w/ vs. w/o Answers</th>
    <th>none—correct</th>
    <th>recall—correct</th>
  </tr>
  <tr>
    <td>89.14%</td>
    <td>10.86%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>2WikiMultiHopQA</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>73.37%</td>
    <td>26.63%</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 1: We analyze the preference gap between retrieved passages and LLMs on 2 datasets: HotpotQA and 2Wiki2MultiHopQA. The top shows the proportion of correct and incorrect generations when the retrieved passage contains the gold answer. The bottom shows the proportion of whether the passage used contains the golden answer when the LLM response is correct.

1*Corresponding author
1The source code is publicly available at https://github.com/liunian-Jay/GainRAG
Figure 2: Illustration of the GainRAG framework. The GainRAG workflow, preference signal synthesis, and selector distillation fine-tuning are shown respectively.

logical cues (Dong et al., 2024; Cuconasu et al., 2024). Therefore, retrieval should shift its focus from traditional “relevance” to “gain”—prioritizing information that helps LLMs generate more accurate results.

To examine and validate this preference gap, we retrieve 100 relevant passages for each sample on two multi-hop question-answering datasets. Each passage is used to enhance the sample query for evaluation. As shown at the top of Fig. 1, we find that even when the retrieved passage contains the correct answer, nearly half of the samples still generate incorrect responses, indicating that while these passages are relevant, they are not particularly beneficial for generation. As shown at the bottom of Fig. 1, in most cases where the generation is correct, the passages used do not directly contain the answer. Instead, some passages that indirectly provide answers or clues may be less relevant but more beneficial, as they align better with the LLM’s preferences.

Existing work aligns the retriever with the LLM’s preferences mainly by fine-tuning the retriever or training both together. For example, Replug (Shi et al., 2023b) aligns the retriever with LLM preferences by training the retriever directly, while RA-DIT (Lin et al., 2023) uses dual training. However, this approach requires a large amount of high-quality data and is difficult to implement in real-world industrial settings. There are also some training middleware to align language model preferences, such as BGM (Ke et al., 2024) and DPA-RAG (Dong et al., 2024), but their perception and measurement of LLM preferences are coarse, capturing only basic patterns without a detailed understanding of nuanced differences.

To address the above challenges, we introduce a middleware between the retriever and the LLM to solve the problem of inconsistent preferences between the two, that is, the most profitable passages can be selected from the large number of passages retrieved by the retriever. Specifically, we introduce a method to quantify LLM’s preference based on based on perplexity and contrastive decoding. This enables passage gain calculation and mitigates the LLM’s overconfidence bias. Secondly, we use this to synthesize a small number of samples to distill the preference perception ability to the selector. Finally, we introduce a pseudo-passage strategy to prevent situations where all retrieved passages are not profitable. Together with the selector, it mitigates degradation and enables efficient integration of internal and external knowledge.

In general, our contributions can be summarized as follows:

• We analyze the preference gap between retrievers and LLMs, quantifying the LLM’s preference for passages by defining “gain” and introducing GainRAG to address this gap.

• We provide a selector and introduce a pseudo-passage strategy that work together to not only avoid degeneration and triviality, but also
achieve efficient integration of internal and external knowledge.

• We train with very few samples and validate on 6 datasets. The results show excellent performance and generalization of GainRAG.

2 Related Works

2.1 Retrieval-augmented Generation
In recent years, in order to solve the problems of outdated knowledge in the model and hallucination of large language models, retrieval-augmented generation has been introduced (Fan et al., 2024; Gao et al., 2023), and many efforts have been made in two aspects: “how to retrieve more relevant information” including retriever fine-tuning (Nian et al., 2024) and query optimization (Ma et al., 2023; Wang et al., 2023) and “how to better use the retrieved information to generate answers” including special fine-tuning (Wang et al., 2024; Zhang et al., 2024) and decoding strategies (Shi et al., 2023a).

2.2 Retriever-LLM Preference Alignment
The misalignment between the preferences of the retriever and LLM results in retrieved information being “relevant” but not always “useful”. There are many existing works that have made efforts to solve this problem. Replug (Shi et al., 2023b) and Altas (Izcard et al., 2022) use LLM to supervise and guide the fine-tuning of the retriever. RA-DIT (Lin et al., 2023) uses dual training allows the language model to guide the training of the retriever and the language model to adapt to the retrieved information. However, these methods usually require a large amount of data, which makes the resources expensive and huge. DPA-RAG (Dong et al., 2024) synthesizes data and trains the reranker and LLM to align the preferences of the two. BGM (Ke et al., 2024) trains an intermediate to complete the rearrangement and selection of information from the coarsely ranked retriever. These methods simply do not clearly define and quantify this preference, which may lead to deviations in alignment. Different from them, we propose a method to quantify the preference, so that a small amount of data can be synthesized through LLM to fine-tune the selector and achieve alignment between the two.

2.3 Bridge Between Retriever and LLM
Many existing works achieve RAG optimization by providing a bridge between the retriever and LLM. Rerankers (Glass et al., 2022) such as BGE-reranker (Xiao et al., 2024) are the most common middleware, which can find more accurate passages from the information recalled by the retriever. Compressors such as RECOMP (Xu et al., 2024) and rewriters (Ma et al., 2023; Wang et al., 2023) are also commonly used bridges, which compress the retrieved passages to achieve denoising and solve the problem of context overload. Similar to them, GainRAG we proposed introduces a preference selector to select the optimal passage, which achieves score perception and avoids excessively long context.

3 Methodology
In this section, we introduce GainRAG (Fig. 2), starting with the synthesis and motivation of the gain signal, followed by selector training, and concluding with the GainRAG workflow.

3.1 Preliminaries
Before discussing GainRAG, we first provide a formal definition of the RAG problem. Given a query \( q \) and a corpus of documents \( \mathcal{D} \), RAG systems typically follow a retrieval-then-reading framework. In this approach, the retriever selects relevant passages \( C = \{c_1, c_2, ..., c_k\} \subset \mathcal{D} \) from the entire corpus \( \mathcal{D} \), and the generator model (LLM) utilizes these passages \( C \) to generate the answer \( \hat{a} \). This process can be represented as:

\[
C = \mathcal{R}(q, \mathcal{D}, k),
\]
\[
\hat{a} = \mathcal{G}(\mathcal{P}(q, C)),
\]

where \( \mathcal{R} \) denotes the retrieval function that selects \( k \) relevant passages, \( \mathcal{P} \) represents the prompt template that combines \( q \) and \( C \), and \( \mathcal{G} \) is the generator, i.e., the LLM, which generates the final answer \( \hat{a} \).

3.2 Gain signal to quantify preference
To make the passage used meet the preferences of the LLM, we need to quantify the benefit of a passage for the LLM to answer a question. To achieve this goal, we introduce contrastive decoding (Li et al., 2022) and calculate the perplexity (Li et al., 2023) after contrastive decoding.

Perplexity In instruction tuning, the model is trained to maximize the likelihood of a response given the instruction. Thus, perplexity (PPL) serves as an indicator of difficulty (Li et al., 2023, 2024). Specifically, the PPL of a given sample \((q, a)\) is
defined as
\[
\mathrm{PPL}(a \mid q) = \exp \left( - \frac{1}{N} \sum_{j=1}^N \log p(a_j \mid q, a_1, \ldots, a_{j-1}) \right).
\]
(2)

Similarly, we can also use perplexity to measure how difficult it is for the LLM to generate the correct answer \( a \) for question \( q \) given \( c \) in RAG.
\[
\mathrm{PPL}(a \mid q, c) = \exp \left( - \frac{1}{N} \sum_{j=1}^N \log p(a_j \mid q, c, a_1, \ldots, a_{j-1}) \right).
\]
(3)

**Contrastive Perplexity** The indicator \( \mathrm{PPL}(a \mid q, c) \) indicates the difficulty of the LLM to generate answers through query \( q \) and context \( c \), but it cannot distinguish whether the answer generation is driven by context \( c \) or the LLM’s internal knowledge. Therefore, directly using PPL for passage screening is biased. Inspired by CAD (Shi et al., 2023a), we introduce contrastive perplexity, which calculates perplexity using contrastive decoded logits (Li et al., 2022) to measure the gain of context \( c \) for answering \( a \). This mitigates bias from the model’s internal knowledge, enabling a more accurate quantification of the gain from context \( c \) to query \( q \). Given our modeling of the internal prior \( p(a_t \mid q, a_{<t}) \), the probability distribution of the LLM output is adjusted as:
\[
a_t \sim \tilde{p}(a_t \mid c, q, a_{<t})
\]
\[
\propto p(a_t \mid c, q, a_{<t}) \left( \frac{p(a_t \mid c, q, a_{<t})}{p(a_t \mid q, a_{<t})} \right)^{\alpha},
\]
where \( p \) is the original probability distribution from the LLM, \( \tilde{p} \) is the distribution adjusted via contrastive decoding, \( a_{<t} \) is the generated sequence, and \( \alpha \) is a hyperparameter controlling the degree of adjustment. After rearranging the formula,
\[
a_t \sim \operatorname{softmax} \left[ (1 + \alpha) \operatorname{logit}_\theta(a_t \mid c, q, a_{<t}) \right.
\]
\[
\left. - \alpha \operatorname{logit}_\theta(a_t \mid q, a_{<t}) \right].
\]
(4)

Therefore, the gain of passage \( c \) on the correct answer \( a \) generated by the LLM for question \( q \) can be quantified as \( \mathcal{M}(c, a \mid q) \), formally,
\[
\mathcal{M}(c, a \mid q) = \exp \left( - \frac{1}{N} \sum_{j=1}^N \log \tilde{p}(a_j \mid q, c, a_1, \ldots, a_{j-1}) \right).
\]
(5)

**3.3 Selector between LLM and Retriever**

To effectively align the retriever’s output with with the preferences of the LLM, we introduce a middleware, the selector, to identify the most beneficial passages. This selector leverages the context-gain-aware approach to refine passage selection, ensuring that the information fed into the LLM maximally enhances its performance.

The selector is formulated as a gain estimation problem, where gain estimates are distilled into a learnable function \( f(q, c; \theta) \to \hat{v} \), where \( \theta \) is the trainable parameter of the model. We employ the BGE pre-trained model as the foundation for the selector, defining its function mapping as:
\[
\hat{V} = [f(q, c_0), \ldots, f(q, c_k)] ,
\]
(6)

where \( \hat{V} \) denotes the predicted gain values for the retrieved passages \( C \) with respect to query \( q \). This formulation enables the selector to effectively rank passages, ensuring that the most relevant and gainful content is utilized by the LLM.

**3.4 Pseudo-passage Strategy**

While increasing the number of retrieved passages raises the likelihood of finding useful information, passage selection can still become degenerate, that is, all retrieved passages do not provide any useful information, or augmenting with each retrieved passage may be worse than LLM direct response.

To address this challenge, we introduce the pseudo-passage strategy. Concretely, before selecting any external passages, we generate a pseudo-passage \( c_0 \) by prompting the LLM with the query. Formally, we define:
\[
c_0 = \mathcal{G}(\mathcal{P}_0(q)),
\]
(7)

where \( \mathcal{P}_0 \) refers to the prompt template used to generate the pseudo-passage. This pseudo-passage \( c_0 \) is then added to the selector’s candidate list alongside the truly retrieved passages.

This strategy reduces over-reliance on potentially unhelpful retrieved passages and ensures that the selection of passages is always gain-oriented. Consequently, the pseudo-passage strategy not only mitigates degradation but also promotes the collaborative and efficient integration of internal and external knowledge of the LLM, ultimately leading to more robust performance.

**3.5 Training of the LLM Preference Selector**

As shown in Fig. 2, we train a middleware, i.e., selector, to align the preferences of the LLM by selecting the most beneficial passage.

**Data Construction** Starting with a set of QA pairs, for each \( \{q, a\} \) pair, we first retrieve \( k \) relevant passages \( C = \{c_1, \ldots, c_k\} \), and then
calculate the gain to construct the training set \( \{ (q_i, a_i, c_i^j, v_i^j) \mid v_i^j = M(c_i^j, a_i \mid q_i) \} \). Additionally, to mitigate degradation, we generate and add a pseudo-passage \( c_0 \) to the set. Algorithm 1 summarizes this process.

Training Loss To make the selector aware of the relative gains of different \( c \) on the same \( q \), we refer to Shi et al. (2023b); Lin et al. (2023) and use distillation loss. Note that due to the long-tail distribution of \( V \) and the large values at the tail, the label \( V \) we actually use is a simple transformation, that is, \( v = -\log(v + 1) \). The distillation loss is calculated as follows,
\[
P = \operatorname{softmax}(V), \quad V = [v_1, \ldots, v_k],
\]
\[
Q = \operatorname{softmax}(\hat{V}), \quad \hat{V} = [\hat{v}_1, \ldots, \hat{v}_k],
\]
\[
\mathcal{L} = \mathrm{KL}(P \parallel Q) = \sum_i P_i \log \frac{P_i}{Q_i}.
\] (8)

3.6 GainRAG Inference Workflow
After obtaining the selector, it acts as a middleware to align preferences between the retriever and the LLM.

Specifically, when a query \( q \) comes, we first prompt the LLM to generate the internal information about this query and use the retriever to retrieve several relevant passages. Formally,
\[
c_0 = \mathcal{G}(q),
\]
\[
[c_1, \ldots, c_k] = \mathcal{R}(q, \mathcal{D}, k),
\] (9) (10)

After getting all the passages, we use the selector to predict the gain of these passages for \( q \) and select the passage with the highest gain. Formally,
\[
c^* = \arg \max_{c \in \{c_0, c_1, \ldots, c_k\}} f(q, c)
\] (11)

Finally, we use the selected optimal passage for enhanced generation to obtain the predicted answer. In terms of formula,
\[
\hat{a} = \mathcal{G}(q, c^*).
\] (12)

Algorithm 2 summarizes this process.

4 Experiments
In this section, we report our experimental details and results, and provide an experimental analysis of GainRAG.

4.1 Implementation Details
Training Data We randomly selecte 20k samples from the HotpotQA training set and about 4k from the WebQuestions training set. For each sample, we gather 21 relevant passages: 20 retrieved using the most common retriever Contriever (Izacard et al., 2021) and 1 generated internally. We then apply Algorithm 1 to add relevant passages and filter out samples where the passage with the highest gain is incorrectly generated, resulting in about 10k samples. The decoding \( \alpha \) is set to 0.5 according to CAD (Shi et al., 2023a).

Training Details We use LLama3-8b to generate preference values and BGE-reranker-base (Xiao et al., 2024) for the selector’s initial weights, training for 2 epochs. All experiments are conducted on a single A100 with 80G memory.

Inference Details During inference, we use Contriever (Izacard et al., 2021) as the retriever and

Algorithm 1 Gain Signal Construction
Input: Original Dataset \( D_o = \{(q, a), \ldots\} \), Corpus \( \mathcal{D} \)
Output: Enriched dataset \( D_v = \{(q, c, v), \ldots\} \)
1: Initialize \( D_v \leftarrow \emptyset \) \( \triangleright \) Initialize
2: for \( (q, a) \in D_o \) do
3: \hspace{1em} \( c_0 \leftarrow \mathcal{G}(\mathcal{P}_0(q)) \) \( \triangleright \) Generate pseudo-passage
4: \hspace{1em} \([c_1, \ldots, c_k] \leftarrow \mathcal{R}(q, \mathcal{D}, k)\) \( \triangleright \) Retrieve relevant passages
5: \hspace{1em} for \( c \in \{c_i \mid i = 0 \ldots k\} \) do
6: \hspace{2em} Compute \( v = M(c, a \mid q) \)
7: \hspace{2em} Add \( (q, c, v) \) to \( D_v \)
8: \hspace{1em} end for
9: end for
10: return \( D_v \) \( \triangleright \) Return the enriched dataset

Algorithm 2 GainRAG Inference Workflow
Input: Query \( q \), Corpus \( \mathcal{D} \)
Output: Answer \( \hat{a} \)
1: \( c_0 \leftarrow \mathcal{G}(\mathcal{P}_0(q)) \) \( \triangleright \) Generate pseudo-passage
2: \([c_1, \ldots, c_k] \leftarrow \mathcal{R}(q, \mathcal{D}, k)\)
3: \( \hat{V} \leftarrow [] \) \( \triangleright \) Initialize score list
4: for \( i \leftarrow 0 \) to \( k \) do \( \triangleright \) Iterate through all passages
5: \hspace{1em} \( \hat{V}[i] \leftarrow f(q, c_i) \) \( \triangleright \) Compute score for \( c_i \)
6: end for
7: \( i^* \leftarrow \arg \max \hat{V} \)
8: \( c^* \leftarrow c_{i^*} \) \( \triangleright \) Select the best passage \( c^* \)
9: \( \hat{a} \leftarrow \mathcal{G}(q, c^*) \) \( \triangleright \) Predict the answer
10: return \( \hat{a} \) \( \triangleright \) Return the final answer
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">2WikiMultiHopQA</th>
    <th colspan="3">WebQuestions</th>
  </tr>
  <tr>
    <th>EM</th><th>F1</th><th>Avg</th>
    <th>EM</th><th>F1</th><th>Avg</th>
    <th>EM</th><th>F1</th><th>Avg</th>
  </tr>
  <tr>
    <td colspan="9">w/o retrieval</td>
  </tr>
  <tr>
    <td>Naive</td><td>22.40</td><td>22.44</td><td>22.42</td><td>26.80</td><td>20.44</td><td>23.62</td><td>44.39</td><td>35.90</td><td>40.14</td>
  </tr>
  <tr>
    <td>GenRead</td><td>31.00</td><td>30.50</td><td>30.75</td><td>30.60</td><td>25.24</td><td>27.92</td><td>47.69</td><td>31.42</td><td>39.55</td>
  </tr>
  <tr>
    <td colspan="9">w/ retrieval</td>
  </tr>
  <tr>
    <td>Standard RAG</td><td>31.80</td><td>33.23</td><td>32.51</td><td>23.40</td><td>21.81</td><td>22.61</td><td>35.04</td><td>33.26</td><td>34.15</td>
  </tr>
  <tr>
    <td>Self-RAG</td><td>30.60</td><td>18.83</td><td>24.71</td><td>34.00</td><td>17.33</td><td>25.67</td><td>42.18</td><td>23.14</td><td>32.66</td>
  </tr>
  <tr>
    <td>Rerank</td><td>35.80</td><td>37.45</td><td>36.62</td><td>24.20</td><td>22.94</td><td>23.57</td><td>37.50</td><td>35.55</td><td>36.52</td>
  </tr>
  <tr>
    <td>GainRAG</td><td><b>39.60</b></td><td><b>41.99</b></td><td><b>40.79</b></td><td><u>31.40</u></td><td><u>28.92</u></td><td><u>30.16</u></td><td><u>42.51</u></td><td><u>39.17</u></td><td><u>40.84</u></td>
  </tr>
</table>

Table 1: EM/F1/Avg(EM,F1) of different methods experimented on datasets HotpotQA, 2WikiMultiHopQA, WebQuestions. The best and second best scores are highlighted in bold and underlined, respectively.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">SQuAD</th>
    <th colspan="3">NaturalQA</th>
    <th colspan="3">TriviaQA</th>
  </tr>
  <tr>
    <th>EM</th><th>F1</th><th>Avg</th>
    <th>EM</th><th>F1</th><th>Avg</th>
    <th>EM</th><th>F1</th><th>Avg</th>
  </tr>
  <tr>
    <td colspan="9">w/o retrieval</td>
  </tr>
  <tr>
    <td>Naive</td><td>18.50</td><td>21.57</td><td>20.03</td><td>31.25</td><td>29.02</td><td>30.13</td><td>60.20</td><td>59.96</td><td>60.08</td>
  </tr>
  <tr>
    <td>GenRead</td><td>21.13</td><td>20.90</td><td>21.01</td><td>38.48</td><td>32.77</td><td>35.62</td><td>64.15</td><td>58.94</td><td>61.55</td>
  </tr>
  <tr>
    <td colspan="9">w/ retrieval</td>
  </tr>
  <tr>
    <td>Standard RAG</td><td>29.53</td><td>32.46</td><td>30.99</td><td>38.14</td><td>36.82</td><td>37.48</td><td>62.16</td><td>61.87</td><td>62.02</td>
  </tr>
  <tr>
    <td>Self-RAG</td><td>27.69</td><td>14.78</td><td>21.23</td><td>35.60</td><td>39.78</td><td>37.69</td><td>61.65</td><td>35.21</td><td>48.43</td>
  </tr>
  <tr>
    <td>Rerank</td><td>29.36</td><td>31.84</td><td>30.60</td><td>30.86</td><td>30.60</td><td>30.73</td><td>65.55</td><td>65.09</td><td>65.32</td>
  </tr>
  <tr>
    <td>GainRAG</td><td><b>34.65</b></td><td><b>37.55</b></td><td><b>36.10</b></td><td><b>41.97</b></td><td><b>41.27</b></td><td><b>41.62</b></td><td><b>67.29</b></td><td><b>66.73</b></td><td><b>67.01</b></td>
  </tr>
</table>

Table 2: EM/F1/Avg(EM,F1) of different methods experimented on datasets SQuAD, NaturalQA, TriviaQA. The best and second best scores are highlighted in bold and underlined, respectively.

LLama3-8B as the generator. We set the initial retrieval setting k to 100 because 100 has a high coverage, as shown in Appendix B analysis experiment. For all datasets, we use 21M English Wikipedia (Karpukhin et al., 2020) dump as the source passages for the retrieval. Prompts for the experiments can be found in Appendix D

<table>
  <tr>
    <th>Task Type</th>
    <th>Datasets</th>
    <th># Samples</th>
  </tr>
  <tr>
    <td rowspan="2">Multi-HopQA</td>
    <td>2WikiMultiHopQA</td>
    <td>500</td>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>500</td>
  </tr>
  <tr>
    <td rowspan="4">OpenQA</td>
    <td>WebQuestions</td>
    <td>2032</td>
  </tr>
  <tr>
    <td>NaturalQA</td>
    <td>3610</td>
  </tr>
  <tr>
    <td>SQuAD</td>
    <td>10570</td>
  </tr>
  <tr>
    <td>TriviaQA</td>
    <td>11313</td>
  </tr>
</table>

Table 3: Description of tasks and evaluation datasets.

4.2 Datasets and Evaluation Metrics

Eval Datasets To verify the effectiveness and generalization of GainRAG, we use the open domain question answering datasets WebQuestion (Berant et al., 2013), NaturalQA (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) and SQuAD (Rajpurkar, 2016), as well as the complex multi-hop question answering datasets HotpotQA (Ho et al., 2020a) and 2WikiMultiHopQA (Ho et al., 2020b). The statistics are shown in Table 3. Its detailed description can be found in Appendix A.

Evaluation Metrics We calculate exact match (EM) and F1 scores. Following Asai et al. (2023); Mallen et al. (2022), we apply a non-strict EM metric, which considers a model’s generation correct if it includes the golden answer, rather than requiring an exact match. F1 measures the overlap between the predicted and golden answers. Note that in our study, longer responses tend to increase EM scores due to higher matching probabilities, but often lower F1 scores due to irrelevant content. Therefore, the average of both metrics may be a more balanced evaluation.

4.3 Baselines

We selected several of the most common methods for comparison. 1) StandardRAG, which is the most classic “retrieve-then-read” paradigm. 2) GenRead (Yu et al., 2022): Its retriever can be seen as itself since it uses self-generated context to answer questions. It has almost no preference for misalignment, but there may be insufficient information. 3) Self-RAG (Asai et al., 2023): Through adaptive retrieval and self-criticism, it alleviates
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">HotpotQA</th>
    <th colspan="3">2WikiMultiHopQA</th>
    <th colspan="3">WebQuestions</th>
    <th colspan="3">NaturalQA</th>
  </tr>
  <tr>
    <th>EM</th><th>F1</th><th>Avg</th>
    <th>EM</th><th>F1</th><th>Avg</th>
    <th>EM</th><th>F1</th><th>Avg</th>
    <th>EM</th><th>F1</th><th>Avg</th>
  </tr>
  <tr>
    <td>Standard RAG</td>
    <td>31.80</td><td>33.23</td><td>32.51</td>
    <td>23.40</td><td>21.81</td><td>22.61</td>
    <td>35.04</td><td>33.26</td><td>34.15</td>
    <td>38.14</td><td>36.82</td><td>37.48</td>
  </tr>
  <tr>
    <td>w/o all</td>
    <td>35.80</td><td>37.45</td><td>36.62</td>
    <td>24.20</td><td>22.94</td><td>23.57</td>
    <td>37.50</td><td>35.55</td><td>36.52</td>
    <td>30.86</td><td>30.60</td><td>30.73</td>
  </tr>
  <tr>
    <td>w/o pseudo</td>
    <td>37.80</td><td>40.65</td><td>39.23</td>
    <td>27.20</td><td>24.88</td><td>26.04</td>
    <td>41.24</td><td>38.97</td><td>40.11</td>
    <td>41.25</td><td>40.94</td><td>41.09</td>
  </tr>
  <tr>
    <td>w/o distillation</td>
    <td>34.20</td><td>35.85</td><td>35.02</td>
    <td>29.60</td><td>26.68</td><td>28.14</td>
    <td>43.21</td><td>36.66</td><td>39.94</td>
    <td>32.41</td><td>31.39</td><td>31.90</td>
  </tr>
  <tr>
    <td>GainRAG</td>
    <td><b>39.60</b></td><td><b>41.99</b></td><td><b>40.79</b></td>
    <td><b>31.40</b></td><td><b>28.92</b></td><td><b>30.16</b></td>
    <td><b>42.51</b></td><td><b>39.17</b></td><td><b>40.84</b></td>
    <td><b>41.97</b></td><td><b>41.27</b></td><td><b>41.62</b></td>
  </tr>
</table>

Table 4: Ablation studies, including: w/o all (removing all modules i.e., the ordinary reranker), w/o pseudo (removing the strategy for generating pseudo-passage, w/o distillation (removing the distillation fine-tuning.)

the preference misalignment problem to a certain extent. 4) Rerank (Glass et al., 2022; Xiao et al., 2024): It is a supplement to the classic RAG. It adds middleware between the retriever and the LLM. Following the “retrieve-rerank-read”, we use the BGE-Reranker-base model. For fairness, Rerank has the same settings as ours. StandardRAG and Self-RAG also only use top-1. The rest of the settings follow the settings of their original papers.

4.4 Main Results

Experimental results are presented in Table 1 and Table 2, and we can get the following analysis:

1) Our method achieves state-of-the-art performance on almost all datasets. Despite using only a small subset of HotpotQA and WebQuestions for data synthesis, it generalizes well across datasets, demonstrating the robustness of GainRAG.

2) In WebQuestions, RAG methods generally underperform compared to those without external knowledge, suggesting that retrieval is not always beneficial. In cases of preference misalignment, retrieved passages can even be harmful. However, our approach still achieves the best average performance.

3) The reranking method outperforms other baselines, proving that middleware integration is both simple and effective. By leveraging a small amount of data to mitigate preference misalignment, our method significantly surpasses standard reranker.

4.5 Ablation Study

In order to verify the effectiveness of each module, we conducted ablation experiments on several datasets. The results, shown in Table 4, confirm that every module plays a crucial and irreplaceable role. The key findings are:

1) Without distillation fine-tuning, performance drops significantly, highlighting the importance of preference alignment. However, due to the existence of the pseudo-passage strategy, the performance can still be significantly improved over the common methods.

2) When pseudo passage strategy is absent, performance drops on some datasets significantly but not all. It shows that it plays a significant role when there is a degenerate solution, that is, when the correct preferred passage cannot be retrieved.

3) Pseudo passages and preference perception fine-tuning are complementary and essential. Together, they prevent degenerate solutions and improve passage selection, aligning preferences between the retriever and the LLM.

![Illustration of gain. Changes in recall of the gold answer and downstream performance after using GainRAG.](page_1012_1012_388_388.png)

Figure 3: Illustration of gain. Changes in recall of the gold answer and downstream performance after using GainRAG.

4.6 Effect of Preference Selection

In order to explore why GainRAG improves the response performance of downstream LLMs, we removed the pseudo-passage strategy and calculated the changes in Recall@1 and downstream generation metrics.

As shown in Fig. 3, we find that there are three general cases. 1) Our selector sometimes significantly improves the Recall@1, which further improves downstream performance. 2) Our selector does not significantly improve the Recall@1, but the downstream performance are significantly im-
proved. 3) Our selector reduces the Recall@1, but the downstream performance are significantly improved.

Case 1 is intuitive, while Cases 2 and 3 demonstrate that our selector’s impact goes beyond just relevance, highlighting the benefits of selection based on gain. This further confirms the effectiveness of our approach.

![Illustration of the pseudo-passages generated for each dataset to avoid degenerate solutions.](page_374_682_695_180.png)

Figure 4: Illustration of the pseudo-passages generated for each dataset to avoid degenerate solutions.

4.7 What is the Effect of Pseudo-passage?
To examine the role of pseudo-passages in mitigating performance degradation, we analyze their usage across different datasets. Specifically, we count the overall use of pseudo-passages, as shown in the right part of Fig. 4. In addition, we replace these cases with non-pseudo-passage with the largest gain and performed a Win-Tie-Lose comparison, as shown in the left part of Fig. 4.

In many cases, internally generated passages are selected, with internal knowledge used in 50% of the cases on 2WikiMultiHopQA. This is consistent with the comparative experiment, which shows that GenRead significantly improves performance on 2WikiMultiHopQA, highlighting the value of LLM-generated passages for this dataset. Additionally, the Win-Tie-Lose comparison reveals that the number of winning cases after replacing pseudo-passages with the highest-gain passages far outweighs the losing cases, further demonstrating the effectiveness of pseudo-passages in alleviating degradation.

4.8 Synthetic Signal Analysis
To explore the effect of contrastive decoding debiasing, we use the ordinary PPL synthetic signal to fine-tune the selector under the same settings. Its performance and changes are shown in Table 5.

We find that contrastive debiasing is crucial, as removing it weakens preference perception for individual passages. This is because this strategy enhances the model’s perception of gain rather than just relevance, thereby alleviating over-reliance on LLM internal knowledge.

<table>
  <tr>
    <th>Datasets</th>
    <th>EM / F1 / Avg(EM,F1)</th>
  </tr>
  <tr>
    <td>HotpotQA</td>
    <td>38.2 (± 1.40) / 41.38 (± 0.61) / 39.79 (± 1.00)</td>
  </tr>
  <tr>
    <td>2WikiMQA</td>
    <td>29.4 (± 2.00) / 27.12 (± 1.80) / 28.26 (± 1.90)</td>
  </tr>
</table>

Table 5: Performance degradation after removing contrastive decoding

4.9 In-Depth Comparison with the Reranker
To explore the impact of the number of selector choices on performance and compare it with the reranker, we set the selector to the interval [1,5] and observe the performance changes, as shown in Fig. 5. The results reveal the following:

1) Increasing K significantly boosts the recall rate, as expected, since more passages increase the likelihood of including the gold answer.

2) For our selector, selecting the top passage is usually sufficient. Even for general rerankers, increasing K does not improve downstream generation performance, and longer contexts even add overhead and may be harmful.

3) While the recall rate increases, downstream generation performance remains largely unchanged, highlighting the scientificity and rationality of selection. This further supports the observation that simply including the gold answer does not guarantee correct generation.

![As the number of passages increases, the changes in recall and downstream generation performance. The left part is the change of our selector, the right side is the BGE-reranker, and the upper and lower parts are recall and Avg(EM, F1) respectively.](page_1012_1342_388_388.png)

Figure 5: As the number of passages increases, the changes in recall and downstream generation performance. The left part is the change of our selector, the right side is the BGE-reranker, and the upper and lower parts are recall and Avg(EM, F1) respectively.
5 Conclusion

This work analyzes the preference gap between retrievers and LLMs and proposes GainRAG to address this misalignment. We define and quantify preferences, then fine-tune a selector with signals from a small number of samples. By adding a selector and using a pseudo-passage strategy to prevent degradation, GainRAG effectively integrates internal and external knowledge of LLMs, achieving superior performance.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China [62206079]; and the Key R&D Program of Heilongjiang Province [2023ZX01A11]. We also appreciate the support from China Mobile Group Heilongjiang Co., Ltd. @ on our research, the research is jointly completed by both parties.

Limitations

GainRAG selects passages with gain by calculating the gain score. However, this selection may not be the optimal solution. Whether there are some combinations of passages that make the gain stronger remains to be verified. And we only used a very small amount of training data to show the effect. In the future, large-scale data training experiments are still needed to verify whether it will get better performance. In addition, for the signal generation of large-scale data, whether a small model can be used as a generator when generating signals to accelerate the experiment is also a need for further experimental verification.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544.

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719–729.

Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, and Ji-Rong Wen. 2024. Understand what llm needs: Dual preference alignment for retrieval-augmented generation. arXiv preprint arXiv:2406.18676.

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Naik, Pengshan Cai, and Alfio Gliozzo. 2022. Re2g: Retrieve, rerank, generate. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2701–2715.

Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00303.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020a. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020b. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118.

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 1(2):4.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea
Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.

Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.

Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.

Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. 2024. Bridging the preference gap between retrievers and llms. arXiv preprint arXiv:2401.06954.

Jaehyun Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Summarizing retrievals using answer candidates for open-domain qa of llms. arXiv preprint arXiv:2404.13081.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Prabhat, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474.

Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. 2024. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. arXiv preprint arXiv:2402.00530.

Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. 2023. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032.

Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097.

Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Loemli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint arXiv:2310.01352.

Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting in retrieval-augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5303–5315.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 7.

Jinming Nian, Zhiyuan Peng, Qifan Wang, and Yi Fang. 2024. W-rag: Weakly supervised dense retrieval in rag for open-domain question answering. arXiv preprint arXiv:2408.08444.

P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen-tau Yih. 2023a. Trusting your evidence: Hallucinate less with context-aware decoding. arXiv preprint arXiv:2305.14739.

Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023b. Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Harsh Trivedi, Niranjana Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509.

Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414–9423.

Yuhao Wang, Ruiyang Ren, Junyi Li, Wayne Xin Zhao, Jing Liu, and Ji-Rong Wen. 2024. Rear: A relevance-aware retrieval-augmented framework for open-domain question answering. arXiv preprint arXiv:2402.17497.

Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muenninghoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024. Recomp: Improving retrieval-augmented lms with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations.

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2022. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209.10063.

Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. 2024. Raft: Adapting language model to domain specific rag. arXiv preprint arXiv:2403.10131.

A Dataset

Here, we introduce in detail the datasets we used, which are seven datasets on four tasks.

2WikiMultiHopQA (Ho et al., 2020b) and HotpotQA (Ho et al., 2020a): Both datasets are multi-hop question answering datasets based on Wikipedia. Considering the limitation of experimental cost, we used the sub-sampling set published by Trivedi et al. (2022); Kim et al. (2024), which is obtained by extracting 500 questions from the validation set of each dataset.

WebQuestions (Berant et al., 2013): Constructed from questions posed by the Google Suggest API, where the answers are specific entities listed in Freebase.

NaturalQA (Kwiatkowski et al., 2019): A dataset designed to support comprehensive QA systems. It consists of questions from real Google search queries. The corresponding answers are text spans from Wikipedia articles, carefully identified by human annotators.

SQuAD (Rajpurkar, 2016): It is a dataset for evaluating reading comprehension, created by annotators who generate questions based on the documents they read. It is widely used for training and testing open-domain QA systems.

TriviaQA (Joshi et al., 2017): A compilation of trivia questions paired with answers, both originally pulled from online sources.

B Coverage Study

To analyze the performance of the original retriever, we conducted experiments on four datasets. Specifically, we used the retriever to retrieve [1, 5, 10, 20, 50, 100] paragraphs respectively and calculated the recall, EM coverage, and F1 coverage. For EM coverage, we used each paragraph to enhance the query separately, and as long as there is one correct response, it is considered to be covered. For EM coverage, we used each paragraph to enhance the query separately, took the response with the largest F1 value, and calculated the average of the overall dataset.

As shown in Fig. 6, Fig. 7 and Fig. 8, as K increases, the recall and coverage will steadily increase. When retrieving 100, the coverage is large enough and far exceeds that of the current state-of-the-art RAG method.

![Illustration of the change in recall as the number of retrievals K increases](page_1012_1042_377_246.png)
Figure 6: Illustration of the change in recall as the number of retrievals K increases

![Illustration of the change in EM coverage as the number of retrievals K increases](page_1012_1342_377_246.png)
Figure 7: Illustration of the change in EM coverage as the number of retrievals K increases

![Illustration of the change in F1 coverage as the number of retrievals K increases](page_1012_1642_377_246.png)
Figure 8: Illustration of the change in F1 coverage as the number of retrievals K increases
C  Training Details

We use LLama3-8b as our generator for generating preference values and BGE-ranker-base (Xiao et al., 2024) for initializing the selector. We train selection for 2 epochs. During fine-tuning, we set train-group-size to 16 and batch-size to 8. In addition, during training, we randomly select 16 out of 21 passages to ensure generalization. The rest of the settings follow the official fine-tuning script (Xiao et al., 2024). Regarding the training data, we randomly selected 20,000 samples from the Hotpot training set and all 3,778 samples from the WebQuestion training set. After filtering, we finally obtained 14,084 training data. All experiments are conducted on a single A100 with 80G memory.

D  Prompt Templates

All the prompt templates used by our proposed GainRAG are shown in Table 6.

<table>
  <tr>
    <th>Task</th>
    <th>Task Instruction</th>
  </tr>
  <tr>
    <td>Generation</td>
    <td>{passage} \n ### Instruction: \n Answer the question below concisely in a few words. \n\n ### Input: \n {query}</td>
  </tr>
  <tr>
    <td>Pseudo-Passage</td>
    <td>Please provide background for the question below in 100 words. Do not respond with anything other than background. If you do not know or are unsure, please generate “N/A” directly. Question: {query}</td>
  </tr>
</table>

Table 6: Full list of instructions used during zero-shot evaluations and pseudo-passage generation. Where query and passage are the paragraph to be used and the question to be answered.

Reviewer paper summaries: The paper proposed a new metric with which we can learn an adaptor to mitigate preference differences between a retrieval and an LLM. The proposed metric is reasonable and the performance improvement is significant.

The paper proposes GainRAG, a preference alignment method for RAG that leverages contrastive distillation and pseudo-passages. Experimental results on various QA datasets show that GainRAG outperforms Self-RAG and other existing methods.
Summary of strengths: The new metric, if it can be fully verified to be effective on alignment the preference differences between a retrieval and an LLM, would be extremely useful to a lot of RAG applications.

1. The idea of combining contrastive distillation with the use of pseudo-passages is both interesting and novel.
2. Experimental results demonstrate that the proposed GainRAG outperforms existing methods such as Self-RAG.
3. The presentation is relatively clear, and the content is well-described.
Summary of weaknesses: The biggest weakness of this paper is that the new metric is not fully verified. All datasets which the authors used are QA datasets. Some prior papers like “Bridging the Preference Gap between Retrievers and LLMs” used different kinds of datasets. Their finding was that the adaptors trained for different tasks cannot be re-used across tasks. So it will be very interesting to see if this new metric can help to make the adaptors universally useful across different kinds of RAG tasks.

1. While alignment of RAG for LLMs has been actively studied recently, the paper lacks comprehensive comparisons with other recent alignment methods. For instance, including DPA-RAG and BGM as additional baselines would strengthen the empirical evaluation.
2. The proposed method fine-tunes the reranker (selector) using contrastive perplexity and pseudo-passages. However, it is unclear whether fine-tuning the reranker is necessary, since passage selection could potentially be done directly using contrastive perplexity without additional training.
3. The use of contrastive perplexity should be more thoroughly evaluated against other likelihood-based reranking metrics. Is contrastive perplexity truly novel? A comparison with related or alternative scoring methods would help clarify its contribution.
Comments, suggestions and typos: Paper looks good to me.

Please see weaknesses
Overall assessment scores: 4

3
Metareview score: 3.5
Metareview: This paper introduces GainRAG, a method to bridge the gap between retriever outputs and LLM preferences in RAG systems. It quantifies a passage’s gain, i.e., how beneficial a passage is for producing accurate answers, using contrastive perplexity and decoding. GainRAG then fine-tunes a selector to choose the optimal passage, including generating a pseudo-passage when needed, to mitigate performance degradation. Experimental results across six QA datasets indicate that this approach improves generation outcomes over standard RAG methods.
Summary of reasons to publish: This paper tackles a recognized challenge in RAG by introducing a new metric for quantifying the gain of retrieved passages and aligning them with the preferences of large language models. It develops a framework that integrates contrastive perplexity, selector fine-tuning, and a pseudo-passage strategy, with experiments demonstrating its robustness across QA datasets. The paper offers a novel approach and solid empirical evidence.