Unsupervised Information Refinement Training of Large Language Models for Retrieval-Augmented Generation (2024)

Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating additional information from retrieval. However, studies have shown that LLMs still face challenges in effectively using the retrieved information, even ignoring it or being misled by it. The key reason is that the training of LLMs does not clearly make LLMs learn how to utilize input retrieved texts with varied quality. In this paper, we propose a novel perspective that considers the role of LLMs in RAG as “Information Refiner”, which means that regardless of correctness, completeness, or usefulness of retrieved texts, LLMs can consistently integrate knowledge within the retrieved texts and model parameters to generate the texts that are more concise, accurate, and complete than the retrieved texts. To this end, we propose an information refinement training method named INFO-RAG that optimizes LLMs for RAG in an unsupervised manner. INFO-RAG is low-cost and general across various tasks. Extensive experiments on zero-shot prediction of 11 datasets in diverse tasks including Question Answering, Slot-Filling, Language Modeling, Dialogue, and Code Generation show that INFO-RAG improves the performance of LLaMA2 by an average of 9.39% relative points. INFO-RAG also shows advantages in in-context learning and robustness of RAG.

heading: 1 Introduction
text: Retrieval-augmented generation (RAG) is a popular framework in modern NLP systems that equips neural with retrieved information for text generation like open-domain question answering, dialogue (Lewis et al., 2020; Guu et al., 2020) etc. Recently, RAG has been applied to large language models (LLMs) to provide additional knowledge
∗Work done during the Tencent Rhino-bird Research Elite Program at WeChat.
†Corresponding authors.
and mitigate issues such as hallucination (Peng et al., 2023; Shi et al., 2023; Ren et al., 2023).
Despite the improved performance of retrieval models, the internet continues to be inundated with fake news, rumors, and fragmented, noisy information, posing challenges for retrieval models to reliably identify and shield against such content (Sun et al., 2022; Thakur et al., 2021). Consequently, not all retrieved texts are beneficial, necessitating that LLMs determine how to judiciously utilize them. However, pre-training tasks do not explicitly enable LLMs to learn how to utilize the retrieved texts with varied quality for generation. For a question and its retrieved texts as input sequence, RAG aims to minimize the negative log-likelihood (NLL) of sub-sequence (question and generated answer) by referring to the retrieved texts. However, mainstream pre-training for LLMs with decoder-only architecture is language modeling based on the prefix (Radford et al., 2018; Touvron et al., 2023a), the training objective aims to minimize the negative log-likelihood (NLL) of the entire input sequence (retrieved texts, question, and generated answer) (Mikolov et al., 2012). This gap causes
133
LLMs to only regard the input retrieved texts as a part of the prefix for language modeling rather than additional reference, which leads to the following problems. Firstly, for the long and complex retrieved texts, LLMs struggle to extract the correct answers (Deng et al., 2023) accurately. Secondly, in situations where the retrieved texts cannot address the task, LLMs lack the capability to integrate the knowledge within model parameters with the retrieved texts to generate improved texts. Thirdly, LLMs are susceptible to incorrect and noisy information in retrieved texts, posing a risk of being misled (Chen et al., 2023; Yoran et al., 2023).
To solve above problems, some previous methods explore strategies for how or when to perform retrieval for LLMs by prompt techniques (Press et al., 2023; Khattab et al., 2022; Xu et al., 2023; Asai et al., 2023). However, prompt cannot materially change the ability of LLMs to utilize retrieved texts because model parameters are not updated for this ability. Some methods fine-tune LLMs on the constructed RAG data for a specific task such as QA (Yoran et al., 2023; Yu et al., 2023). However, under the trend that LLMs are regarded as foundation models for various tasks in zero-shot setting, fine-tuning LLMs only on a few tasks make LLMs limited to the RAG of training tasks and lose their generalizability. Because catastrophic forgetting still exists in supervised fine-tuning of LLMs (Luo et al., 2023). Although constructing data for a large number of tasks can alleviate this, it is hard to design the data in various RAG tasks and requires high data annotation costs. Our paper aims to fundamentally improve the ability of LLMs to utilize retrieved texts while preserving the generalizability of LLMs for various RAG tasks in zero-shot setting, which is orthogonal to prompt techniques and can be combined with them to get better performance.
In this paper, considering that LLMs have a certain ability to use their own knowledge to examine information (Dhuliawala et al., 2023), we introduce a novel perspective to reassess the role of LLMs in RAG. Specifically, we propose considering LLMs as “Information Refiner”. The key idea behind this is to continue training the pre-trained LLMs with an Information Refinement objective that regardless of the correctness, completeness, or usefulness of the input retrieved texts, LLMs can consistently integrate knowledge within the retrieved texts and model parameters to generate the texts that are more concise, accurate, and complete than the retrieved texts (Figure 1). We term this
process “Positive Information Gain”. This enables LLMs to extract correct information from complex texts as well as resist and rectify retrieved erroneous information and noise, thereby improving the information bottleneck of the RAG and allowing the knowledge capacity of RAG to approximate the combined knowledge of IR and LLMs.
We make the information refinement training work in a completely unsupervised manner, such that it is easy to obtain large-scale training data and maintain the generalizability of the trained LLMs that can be used in various RAG tasks in zero-shot setting. Specifically, we propose an unsupervised training method named INFO-RAG. INFO-RAG classifies the retrieved texts into three scenarios (shown in Figure 1) and proposes the unsupervised training task for each scenario. For the first scenario that all knowledge for the question is already in the retrieved texts, LLMs need to accurately extract relevant knowledge from complex retrieved texts and generate more concise texts. For the second scenario that retrieved texts are incomplete or incorrect for the question, LLMs need to combine the knowledge within model parameters to verify the retrieved texts, correct the wrong knowledge, and complete the missing knowledge. For the third scenario that retrieved texts are relevant but do not have any answer, LLMs need to find the knowledge within model parameters based on relevant context to generate correct answers. We mix the above three tasks to train INFO-RAG unsupervisedly.
Main contributions of this paper are as follows: (1) We introduce a novel perspective to reassess the role of LLMs in the RAG system that considers LLMs as “Information Refiner” that can produce positive information gain in RAG scenarios. (2) We propose an unsupervised training method named INFO-RAG that enables LLMs to perform information refinement in RAG. INFO-RAG is low-cost and general for various RAG tasks. (3) Extensive experiments show INFO-RAG enhances the zero-shot RAG of LLaMA2 across Question Answering, Slot-Filling, Language Modeling, Dialog, and Code Generation. INFO-RAG also shows advantages in in-context learning and robustness of RAG. Code is released at https: //github.com/xsc1234/INFO-RAG/.

heading: 2 Related Work
text: Retrieval Augmented Generation Retrieval augmented generation (RAG) aims to provide addi-
tional knowledge for language models by retrieving information from external databases (Lewis et al., 2020; Guu et al., 2020; Borgeaud et al., 2022; Izacard et al., 2022). RAG makes the text generated by LLM more accurate and credible, and is widely used in Open-domain QA (Karpukhin et al., 2020; Trivedi et al., 2022a), dialogue (Cai et al., 2018, 2019) and Code Generation (Parvez et al., 2021). Recently, RAG has also been widely applied in LLMs (Peng et al., 2023; Shi et al., 2023; Ren et al., 2023). The form of RAG in LLMs is using the retrieved texts as contexts (Ram et al., 2023).
Some studies have noted that noise in retrieved texts will interfere with the performance of the language model or even mislead it (Xu et al., 2023; Wang et al., 2023; Chen et al., 2023; Xu et al., 2024). These works try to solve this problem from the interactive framework between IR and LM, while our work points out a more essential view. That is, previous studies on RAG do not define the role of LLMs in RAG clearly. Our paper introduces a novel perspective to reassess the role of LLMs in RAG that considers LLMs as “Information Refiner”.
Unsupervised Learning of RAG Unsupervised learning of RAG can be divided into the training of retrievers and language models. As for retrievers, REALM (Guu et al., 2020) proposes using masked language modeling to pre-train a knowledge retriever. REPLUG (Shi et al., 2023) trains the retriever according to the feedback from black-box LM. As for language models, RETRO (Borgeaud et al., 2022) improves language models by retrieving tokens. Atlas proposes pretext tasks to jointly train the retriever and language model. However, these two methods focus on the model of encoderdecoder architecture, which is inconsistent with the current mainstream LLMs based on decoder-only.
Previous unsupervised training methods do not consider the specific role that language models should play in RAG. In this paper, we focus on training language model as an “Information Refiner” that can further improve the information bottleneck of RAG and be robust to retrieved texts.

heading: 3 Our INFO-RAG
text: This section introduces our INFO-RAG, an unsupervised training method to enable LLMs to perform information refinement in RAG. Firstly, we summarize the retrieved texts in RAG into three scenarios and define the positive information gain
for each scenario. Secondly, we construct sample pairs in which the output has information gain compared to the input for these three scenarios and design three training tasks. Thirdly, we train LLMs under our designed tasks on the unsupervised samples. Unsupervised training makes INFO-RAG low-cost and general for RAG in various tasks.

heading: 3.1 Positive Information Gain in RAG
text: In this paper, we introduce a novel perspective to reassess the role of LLMs in RAG that LLMs should be the “Information Refiner” that can produce “Positive Information Gain” in the information flow of RAG. This section details the scenarios of retrieved texts and defines specific information gain LLMs should produce in each scenario.
Scenario 1. The first scenario is that all knowledge for the question is already in the retrieved texts. Even if the correct knowledge already exists in the retrieved texts, complex and lengthy retrieved texts are not conducive for users to directly obtain the knowledge. Therefore, the positive information gain in this scenario means that LLMs extract correct knowledge as much as possible while removing irrelevant information, thereby generating more direct and concise texts for users.
Scenario 2. The second scenario is that although the retrieved texts contain some usable knowledge, they still contain some incomplete or incorrect knowledge. This scenario is very common, especially with the current proliferation of fake news, misinformation, and fragmented knowledge on the Internet. There has been study proving that noise and erroneous knowledge in retrieved texts greatly mislead the generation of LLMs (Xu et al., 2023). The positive information gain in this scenario is that LLMs can exploit the knowledge within their parameters to verify the knowledge in the retrieved texts. Utilize accurate knowledge, rectify incorrect knowledge, and complete missing knowledge
Scenario 3. The third scenario is that the retrieved texts do not have any answer that can used to solve the question. This scenario means that the question is very difficult or the target knowledge is very long-tail for information retrieval systems. Even in this case, the retrieval model’s ability to model semantics allows it to provide texts that are semantically related to the question (Karpukhin et al., 2020). Therefore, the positive information gain in this scenario is that LLMs can stimulate the knowledge within their parameters based on semantically relevant context to solve the question.
Target
Wikipedia Document
Intercept k consecutive sentences
Simulated Retrieved TextsInitial Sentence Set
…
…
𝑠𝑠1 𝑠𝑠2
𝑠𝑠𝑘𝑘
…
𝑠𝑠𝑙𝑙 ……
𝑠𝑠𝑙𝑙𝑡𝑡 …
𝑠𝑠𝑙𝑙 𝑝𝑝 …Prefix
Target
(a) Data Collection (b) Data Construction (c) Unsupervised multi-task learning
…
Prefix … …
…
𝑠𝑠1 𝑠𝑠2
𝑠𝑠𝑘𝑘
…
𝑠𝑠𝑙𝑙 ……
Randomly masking or replacement on informative tokens
… …
…
𝑠𝑠1 𝑠𝑠2
𝑠𝑠𝑘𝑘
…Eliminate 𝑠𝑠𝑙𝑙
… … 𝑠𝑠1 𝑠𝑠2
𝑠𝑠𝑘𝑘
…
𝑠𝑠𝑙𝑙 ……
Keep
𝑠𝑠𝑙𝑙 𝑝𝑝 … LLM 𝑠𝑠𝑙𝑙𝑡𝑡 …
𝑠𝑠𝑙𝑙 𝑝𝑝 … LLM 𝑠𝑠𝑙𝑙𝑡𝑡 …
𝑠𝑠𝑙𝑙 𝑝𝑝 … LLM 𝑠𝑠𝑙𝑙𝑡𝑡 …𝑆𝑆
𝑆𝑆𝑆
𝑆𝑆 − {𝑠𝑠𝑙𝑙}
𝑝𝑝 𝑠𝑠𝑙𝑙𝑡𝑡 = 𝑝𝑝𝜃𝜃([𝑆𝑆; 𝑠𝑠𝑙𝑙 𝑝𝑝])
𝑝𝑝 𝑠𝑠𝑙𝑙𝑡𝑡 = 𝑝𝑝𝜃𝜃([𝑆𝑆𝑆; 𝑠𝑠𝑙𝑙 𝑝𝑝])
𝑝𝑝 𝑠𝑠𝑙𝑙𝑡𝑡 = 𝑝𝑝𝜃𝜃([𝑆𝑆 − {𝑠𝑠𝑙𝑙}; 𝑠𝑠𝑙𝑙 𝑝𝑝])
Select and Copy
Contextual Stimulation
Correct and Complete
Scenario 1
Scenario 2
Scenario 3
…
…
𝑠𝑠1 𝑠𝑠2
𝑠𝑠𝑘𝑘
…
𝑠𝑠𝑙𝑙 ……𝑆𝑆
…
…
𝑠𝑠1 𝑠𝑠2
𝑠𝑠𝑘𝑘
…
𝑠𝑠𝑙𝑙 ……
𝑆𝑆
…
…
𝑠𝑠1 𝑠𝑠2
𝑠𝑠𝑘𝑘
…
𝑠𝑠𝑙𝑙 ……
𝑆𝑆
… 𝓡𝓡(𝑠𝑠𝑙𝑙
𝑝𝑝)𝑆𝑆
Figure 2: Overview of our INFO-RAG. Each sample is only processed for a single scenario to avoid data leakage.

heading: 3.2 Unsupervised Learning
text: This section introduces unsupervised learning in INFO-RAG. We construct the input-output pairs that satisfy the information gain in the above three scenarios on Wikipedia. We continue to train pretrained LLMs on the constructed data to perform information refinement in the form of next token prediction in prefix language modeling, which is general for various tasks. Pipeline is in Figure 2.

heading: 3.2.1 Data Collection
text: The data construction is performed on English Wikipedia. Specifically, for each document d in Wikipedia, we intercept k consecutive sentences from d and get the sentence set S = [s1, s2, ..., sk]. Our method randomly selects sl from S and uses it as the object for language modeling. The first 13 to 2 3 of the tokens of sl are randomly intercepted as the prefix (spl ) and the other tokens of sl are used as the prediction target (stl). We also perform the process (Section 3.2.2) on sentence set S so that it can be used to simulate the retrieved texts R(spl ) for prefix spl in three scenarios for conditioning the generation of stl . Then, we can get an unsupervised training sample for prefix language modeling that predicts stl given the prefix s p l and the retrieved texts R(spl ). This can be formulated as:
p(stl) = pθ([R(spl ); s p l ]), (1)
θ are parameters of LLMs, [R(spl ); s p l ] is the concatenation of R(spl ) and s p l by a special token.

heading: 3.2.2 Data Construction and Training Tasks
text: This section details our data construction and training tasks for three scenarios in Section 3.1.
For Scenario 1 that needs LLMs to extract the correct knowledge from the complex texts, we
propose the training task named Select and Copy. Specifically, given the sentence set S for a sample, Select and Copy directly uses all sentences in S as retrieved texts for conditioning LLMs to predict stl for the given prefix spl . This can be formulated as:
p(stl) = pθ([S; s p l ]). (2)
In Select and Copy, sl (both s p l and s t l) has been contained in the retrieved texts S, this needs LLMs to select the texts matching the prefix spl from the complex retrieved texts S and directly copy the target stl for generation. The information gain between stl and input retrieved texts S is that s t l is more concise to be used as the postfix for spl . For Scenario 2 that needs LLMs to verify the knowledge in the retrieved texts, utilize accurate knowledge, rectify incorrect knowledge, and complete missing knowledge. We propose the training task named Correct and Complete. Given a sentence set S, firstly, this task uses the stability of word distribution between layers to get informative tokens. The intention for this is that the more unstable the word distribution of the token among the topmost layers is, the more it indicates that the token is an informative token. We follow (Chuang et al., 2023) to achieve this. Specifically, for each sentence si in S, our method obtains the next word distribution of the a-th token s[a]i given prefix s <a i of si in each layer of LLM as:
dj(s [a] i |s<ai ) = softmax(WH [a] j ), (3)
in which j indicates the j-th layer of LLMs, H[a]j ∈ Rh is the hidden states for token s[a]i in the j-th layer, W ∈ Rh×v is the vocabulary head that maps the hidden states H[a]j to the word distribution with
vocabulary size v. Then, for the LLM with N layers, our method uses Jensen-Shannon Divergence (JSD) to measure the differences in word distribution between layers and gets the word distribution stability of token s[a]i as:
O [a] i = max
j∈J JSD(dN (s
[a] i |s<ai )||dj(s [a] i |s<ai )),
in which J is the set of candidate layers (0-th to N 2 -th layers), dN (s [a] i |s<ai ) is the word distribution of the last layer. This design aims to find the layer with the largest word distribution difference between the last layer and use the JSD of the two as the word distribution stability of the token s [a] i (Chuang et al., 2023). For each token of si, we obtain its word distribution stability in parallel and get the set of word distribution stability for si as:
Oi = {O[0]i , O [1] i , ..., O [n] i }. (4)
We choose the tokens corresponding to the top 50% of the elements in Oi as informative tokens within the sentence si. Subsequently, we apply a specific percentage (30%) of random masking and replacement to these tokens. For the randomly selected token, we replace it with [MASK] with a 50% probability to simulate the incomplete knowledge, and randomly replace it with another token with a 40% probability to simulate the incorrect knowledge, while keeping it unchanged with a 10% probability to simulate the correct knowledge. We do the above pipeline for each sentence in the set S and get the processed set S′. RAG in Correct and Complete can be formulated as:
p(stl) = pθ([S ′; spl ]). (5)
In Correct and Complete, the broken sl with noise is already in S′. The information gain in this task requires LLM to extract, correct, and complete the knowledge in sl from S′ to generate stl .
For Scenario 3 that needs LLMs to find answers from their knowledge based on relevant texts in context. We propose the training task named Contextual Stimulation. Contextual Stimulation eliminates sl (both s p l and s t l) from the set S and uses the remaining sentences as retrieved tests for generation, which can be formulated as:
p(stl) = pθ([S − {sl}; spl ]). (6) In Contextual Stimulation, each sentence in retrieved texts S − {sl} is semantically relevant to spl but cannot help LLMs to directly generate s t l . LLMs need to be stimulated by relevant information to generate stl based on their own knowledge.

heading: 3.2.3 Training Strategy
text: After the data construction for three training tasks, we mix them for multi-task training. Specifically, we use LoRA (Hu et al., 2021) to train the pretrained LLMs on the mixed dataset of three tasks. Three tasks are trained alternately in batches. Since Select and Copy is relatively simple for LLMs, it only accounts for 20% of the batches, while Correct and Complete and Contextual Stimulation each account for 40% of the batches. Using LoRA not only reduces training costs but also makes our method plug-and-play. The trained LoRA parameters are loaded when LLMs need to perform RAG and unloaded when RAG is not needed.

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation Metrics
text: To demonstrate the generality of our unsupervised training method, we evaluate the performance of INFO-RAG on eleven datasets across seven tasks. Open-domain Question Answering Open-domain QA is a typical knowledge-intensive task that can directly evaluate the knowledge of LLMs. We use Natural Questions (Kwiatkowski et al., 2019) (NQ) and WebQuestions (Berant et al., 2013) (WebQ) as the datasets. We use cover Exact Match (EM) to determine whether the ground truth exactly appears in the output and the accuracy is used as the evaluation metric, following (Schick et al., 2023) Soft Filling Soft filling requires LLMs to output the object entities for the input subject entity and relation. We use two knowledge-intensive datasets including Zero Shot RE (Levy et al., 2017) (ZS) and T-REx (Elsahar et al., 2018). We use the same evaluation metric as Open-domain QA. Long-Form Question Answering Compared with open-domain QA, LFQA is the QA task whose ground truth answer is a relatively long text. We use ELI5 (Fan et al., 2019), a knowledge-intensive dataset for LFQA. We use ROUGE-L as the evaluation metric (Petroni et al., 2020). Dialogue Dialogue in our experiment focuses on the factual knowledge. We use Wizard of Wikipedia (Dinan et al., 2018) (WoW), a knowledge-powered dialogue dataset whose conversation is grounded with knowledge. We use F1 as the evaluation metric (Petroni et al., 2020). Language Modeling We use WikiText-103 (Merity, 2016), a popular dataset for language modeling. We use ROUGE-L as the evaluation metric.
Multi-Hop Question Answering Multi-hop QA measures the ability of LLMs to perform combined reasoning on multiple knowledge. We use HotpotQA (Yang et al., 2018) and Musique (Trivedi et al., 2022b) for this task. We use the same evaluation metric as Open-domain QA. Code Generation Code generation aims to generate the code for the given natural language. We use Java and Python in CodeXGLUE (Iyer et al., 2018) for this task. We use CodeBLEU (Ren et al., 2020) as the evaluation metric.

heading: 4.2 Experimental Settings
text: LLMs in our paper include LLaMA-2-7B, 13B and their chat version (Touvron et al., 2023b). We use LoRA to fine-tune these pre-trained LLMs on four A100 GPUs with the learning rate of 1e-5, per-gpu batch size of 4 (for 7B) and 2 (for 13B) for 5K steps. As for the training data, we intercept 15 consecutive sentences for each example.
As for the retrieval model and retrieval database, for Open-domain QA, Soft Filling and Language Modeling, we use ColBERTv2 (Santhanam et al., 2022), a late-interaction model with excellent generalization ability as the retriever, and use Wikipedia consisting of 21,015,324 passages (Karpukhin et al., 2020) as retrieval database. For Code Generation, we SCODE-R (Parvez et al., 2021) as code retriever and use deduplicated source codes in CodeSearchNET (Husain et al., 2019) as retrievel database. For all the above tasks, we give Top-5 retrieved passages to each example. For LFQA, Dialog, and Multi-Hop QA, we use the list of contextual passages provided in the datasets as the retrieved list (distractor setting). In each experiment, all baselines and our method share the same retrieved documents.

heading: 4.3 Experimental Results
text: Main Results (Zero-Shot Setting) Experimental results in Table 1 show the improvement (the average is 9.39%) of our method on the utilization of retrieved knowledge from four aspects.
(1) Short and Direct Knowledge. Our method can significantly improve the RAG performance of LLaMA on ODQA and Slot-Filling tasks. The answer in ODQA and Slot-Filling is short and direct, it can directly reflect the ability of LLMs to utilize the knowledge in retrieved texts.
(2) Reasoning on Multiple Knowledge. Our INFORAG has advantages in cross-passage reasoning on multiple knowledge of retrieval lists. Questions in both HotpotQA and Musique are complex and need multiple knowledge from different passages. These questions not only require LLMs to extract correct knowledge from the retrieved passage list but also to combine the knowledge of different passages in the list for reasoning to give the final answer.
(3) Long and Complex Knowledge. Our INFORAG can improve the RAG performance of LLaMA on LFQA, Dialogue and Language Modeling. These tasks require LLaMA to output long and complex texts grounded with intensive knowledge.
(4) Code Knowledge. Our INFO-RAG can also improve the RAG performance of LLaMA on Code Generation. This further demonstrates the crosstask generality of INFO-RAG. Our method is only trained on natural language but can also show advantages in programming language tasks, which demonstrates that INFO-RAG successfully enables LLMs to learn how to exploit the retrieved information rather than just fitting the data. Unsupervised and prefix language modeling training paradigms make INFO-RAG general in various tasks.
Results on In-context Learning for RAG Besides, our INFO-RAG allows further improvement cooperating with in-context learning (ICL). ICL (Brown et al., 2020) works by prepending a few examples of the target task before the query, which helps LLMs understand the task. However, ICL may not always help in the RAG setting, mainly due to the confusion between the retrieved texts of the query and the few-shot examples. As shown in Table 3, LLaMA-2 cannot further improve the RAG performance from ICL, even sometimes hurt by the few-shot examples while INFORAG can further improve RAG by ICL. This is mainly because INFO-RAG enables LLaMA to understand the task form of RAG, thereby better learning the general task pattern from ICL examples. In this experiment, we construct the ICL example consisting of a query, a relevant passage, and an answer. For a fair comparison, we need to ensure that the performance of our method and the baseline are close in non-ICL setting. Therefore, we select queries for which the baseline gives the same answer as our method (both correct or both incorrect) and evaluate the ICL performance on these queries.

heading: Enhancing Previous SOTA in Open-Retrieval
text: Setting We further show that our INFO-RAG can cooperate well with the recent prompting techniques that perform multi-step reasoning to combine with retrieval to solve questions (Xu et al., 2023; Khattab et al., 2022; Press et al., 2023; Yao et al., 2022). To make a fair comparison, we follow SearChain (Xu et al., 2023) that runs on Multi-Hop QA and Slot-Filling in open-retrieval setting that retrieves passages from the full Wikipedia in each reasoning step. SearChain and other baselines use LLaMA-2-13B-chat as the backbone. Then, we
perform SearChain based on LLaMA-2-13B-chat trained by INFO-RAG to show the enhancement to SearChain by INFO-RAG. Results in Table 4 show that INFO-RAG can make SearChain achieve better performance. This provides additional support that our unsupervised INFO training fundamentally improves the RAG performance of LLMs.

heading: 4.4 Analysis

heading: Fine-grained Analysis for Three Scenarios As
text: shown in Table 2, our INFO-RAG is effective in all three RAG scenarios and shows better robustness to incorrect, incomplete, and noisy retrieved texts. We propose corresponding unsupervised training tasks for the three scenarios of RAG. This section introduces the fine-grained analysis for each sce-
nario. For Scenario 1, we use cover EM to select those samples that already contain the correct answers in the retrieval list. For Scenario 2, we randomly replace the correct answers in the retrieved texts with another phrase with the same properties. For Scenario 3, we use cover EM to select those samples that retrieved texts do not contain any correct answers. We count the accuracy of LLaMA on samples of these three scenarios respectively. Questions in the third scenario are more difficult than in the second scenario because retrieval models cannot find anything to solve them. Table 2 indicates that our method shows advantages in each scenario and is more robust regardless of whether the retrieved texts contain the correct answer.
Ablation Study We conduct ablation study to explore the effects of the following factors. (1) Additional Training on Wikipedia. We study whether our improvement is from helping the model to achieve information refinement, or simply because of additional training on Wikipedia. To this end, we train LLaMA-2 on Wikipedia with standard language modeling objective, by setting the same hyperparameters as our INFO-RAG. The results in Table 5 show that this baseline leads to no improvement over the backbone LLaMA-2, confirming the effectiveness of our training method rather than additional training on Wikipedia. (2) Training tasks. We perform three training tasks proposed in INFO-RAG separately on original data and data constructed for each scenario to explore their effects respectively. Table 6 shows that both S2 and S3 have gains in their scenarios. Although
S1 has negative effects when performed alone, it can achieve the best results when trained together with S2 and S3. This is mainly because S1 alone is so simple that causes LLM to overfit the data. Adding S2 and S3 allows LLM to learn the task paradigm of information refinement, making LLM better extract the correct answer for Scenario 1.
Robustness to Retrieval Results Table 7 shows INFO-RAG is more robust to changes in retrieval results including the ratio and position of positive passages and number of retrieved passages. More details can be found in Section A of Appendix. Avoid Catastrophic Forgetting Experiment on MMLU (Hendrycks et al., 2020) without RAG shows that INFO-RAG performs very close to the original LLaMA-2 (7B: 45.0 vs. 45.3; 13B: 54.3 vs. 54.8), which indicates that INFO-RAG enhances RAG while avoiding catastrophic forgetting. More details can be found in Section A.6 of Appendix.

heading: 5 Conclusion
text: This paper proposes a novel perspective to reassess the role of LLMs in RAG that considers LLMs as “Information Refiner”. This means that regardless of the correctness, completeness, or usefulness of the retrieved texts, LLMs can consistently integrate knowledge within model parameters and the retrieved texts to generate texts that are more concise, accurate, and complete. To achieve it, we propose an information refinement training method named
INFO-RAG in an unsupervised manner, which is low-cost and general across various tasks. Extensive experiments across 11 datasets of 7 tasks in zero-shot setting show that INFO-RAG improves the performance of LLMs for RAG. INFO-RAG also shows advantages in ICL and robustness of RAG and can be combined with the SOTA RAG framework to further improve its performance.

heading: Limitations
text: This paper aims to enable LLMs to perform information refinement in RAG by unsupervised training, so as to accurately extract correct information and avoid the interference of incorrect information. The main limitation of this paper is that due to the lack of computing resources, we only conduct experiments on models with 7B and 13B parameter sizes. In the future, we consider using more computing resources to explore the performance of models with larger parameter sizes.

heading: Ethics Statement
text: After careful consideration, we believe that our paper does not introduce additional ethical concerns. We declare that our work complies with the ACL Ethics Policy.

heading: Acknowledgements
text: This work was supported by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103704), the National Natural Science Foundation of China (NSFC) under Grants No. 62276248 and U21B2046, and the Youth Innovation Promotion Association CAS under Grants No. 2023111.

heading: A More Analysis

heading: A.1 Robustness to ratio of Positive Passages
text: Our INFO-RAG improves the robustness of RAG performance to retrieval performance. The performance of the retriever greatly affects the performance of LLM in RAG (Chen et al., 2023). We explore this in this section. Specifically, we simulate changes in retrieval performance by varying the ratio of positive and negative passages in the retrieved list and report the RAG performance with different ratios. Table 8 shows INFO-RAG performs better when the ratio is low and the performance is more stable than baseline when the ratio changes from 100% to 0% (Max ∆). The model in this experiment is LLaMA-2-13B-chat.

heading: A.2 Robustness to Positive Passage Position
text: Experimental results in Table 9 show that our INFO-RAG consistently outperforms the baseline (LLaMA-2) regardless of where the positive passage (passage contains the correct answers) appears in the retrieved list. Specifically, we mix positive and negative passages in a ratio of 1:9 to simulate the retrieved passage list, vary the position of the positive passage in the retrieved list from 0 to 9, and evaluate the corresponding RAG performance respectively. The model in this experiment is LLaMA-2-13B-chat. Experimental results show that our INFO-RAG not only outperforms the baseline at every position but also achieves more stable performance varying with the position (Max ∆).

heading: A.3 Robustness to Number of Retrieved Passages
text: Experimental results in Table 10 show that our INFO-RAG consistently outperforms the baseline with the different number of retrieved passages (from 1 to 10) and is robust to the change of the number. In this experiment, we use LLaMA-213B-chat as the base model, change the number of retrieved passages from 1 to 10, and evaluate the corresponding performance.

heading: A.4 Ablation Study on Masking Strategy
text: In general, Table 13 and 12 show our masking strategy in Scenario 3 is more effective than simple and straightforward masking. Specifically, our method is more significantly effective in the scenarios that correct answers are randomly replaced with other phrases (replace) and retrieval cannot find any answers (no answer).

heading: A.5 Works with Different Retriever
text: We evaluate our method and baseline (LLaMA213B-chat) with BM25 as the retriever, the experimental results shown in Table 11 indicate that our method still performs better than baseline when the retriever as BM25.

heading: A.6 Performance on MMLU
text: Experimental results on MMLU benchmark in the setting without RAG shown in Table 14 show that our INFO-RAG significantly improves the performance of LLMs in RAG, while still maintaining its versatility and avoiding catastrophic forgetting. MMLU is a benchmark that measures massive multitask language understanding ability of LLMs. It covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem-solving ability (Hendrycks et al., 2020). Experiments show that our INFO-RAG performs very close to the original LLaMA-2 on MMLU, which shows that our INFO-RAG does not damage the basic language understanding ability of LLMs. This is mainly because the prefix language model-
ing training paradigm of our method is consistent with the pre-training task of LLMs. The difference is that in the training of prefix language modeling, our method learns to perform information refinement that utilizes the retrieved texts for the next token prediction.
Humanities STEM Social-Sciences Other Average

heading: Future Work
text: In the future, we consider using more computing resources to explore the performance of models with larger parameter sizes