Blinded by Generated Contexts: How Language Models Merge Generated and Retrieved Contexts When Knowledge Conflicts? (2024)

Abstract: While auxiliary information has become a key to enhancing Large Language Models (LLMs), relatively little is known about how LLMs merge these contexts, specifically contexts generated by LLMs and those retrieved from external sources. To investigate this, we formulate a systematic framework to identify whether LLMs’ responses are attributed to either generated or retrieved contexts. To easily trace the origin of the response, we construct datasets with conflicting contexts, i.e., each question is paired with both generated and retrieved contexts, yet only one of them contains the correct answer. Our experiments reveal a significant bias in several LLMs (GPT-4/3.5 and Llama2) to favor generated contexts, even when they provide incorrect information. We further identify two key factors contributing to this bias: i) contexts generated by LLMs typically show greater similarity to the questions, increasing their likelihood of being selected; ii) the segmentation process used in retrieved contexts disrupts their completeness, thereby hindering their full utilization in LLMs. Our analysis enhances the understanding of how LLMs merge diverse contexts, offers valuable insights for advancing current LLM augmentation methods, and highlights the risk of generated misinformation for retrieval-augmented LLMs1.

heading: 1 Introduction
text: Recent advancements in augmenting Large Language Models (LLMs) with auxiliary information have significantly revolutionized their efficacy in knowledge-intensive tasks (Chang et al., 2023; Ram et al., 2023). This auxiliary information can originate from contexts generated by LLMs or retrieved from external sources. For the former, Liu et al. (2022); Sun et al. (2023); Yu et al. (2022)
†Corresponding author. 1Code released at https://github.com/Tan-Hexiang
/RetrieveOrGenerated.
instruct LLMs to initially generate a background context tailored to the given question, which then serves as the basis for producing the final answer. In contrast, retrieval-augmented approaches (Lewis et al., 2020; Ram et al., 2023) include relevant passages retrieved from external corpora, such as Wikipedia, as context, thereby notably enhancing LLMs’ capability to address challenges like knowledge updates (Jang et al., 2022) and long-tail knowledge (Kandpal et al., 2023).
Recent hybrid methods have attempted to integrate these two types of contexts to further improve performance in tasks like Question Answering (QA) (Yu et al., 2022; Mallen et al., 2023; Zhang et al., 2023). However, we observe an abnormal phenomenon: in certain cases, models relying solely on retrieval contexts succeeded, whereas hybrid approaches failed, as depicted in Figure 1. This observation implies that LLMs may struggle to effectively integrate diverse types of contexts, overlooking correct information in retrieved contexts. To uncover the reasoning behind this, this study investigates the underlying mechanisms by which LLMs process the two types of contexts, especially when they contain conflicting information.
To facilitate the investigation, we proposed a systematic framework to dissect the process by which
6207
LLMs merge generated and retrieved contexts. We curated tailored context-conflicting (CC) datasets in which each question is accompanied by a pair of generated and retrieved contexts. These contexts are deliberately designed to be inconsistent, with only one containing the correct answer to its corresponding question. These datasets provide a solid foundation for determining whether LLMs utilize retrieved or generated context to produce responses in QA tasks.
In this paper, we conducted a series of controlled experiments using our uniquely designed datasets to empirically study this question, focusing on several state-of-the-art closed (GPT-3.5/4) and open (Llama2-7b/13b) LLMs. Surprisingly, our findings reveal a pronounced bias in LLMs to favor generated contexts across various LLMs (§4.2), even when the generated contexts offer incorrect information while the retrieval contexts hold the correct answers. Further analysis shows that the bias is prevalent across various retrieval models (§4.3). These findings highlight a critical challenge for existing LLMs in effectively merging contexts from diverse sources, especially in light of the increasing prevalence of LLM-generated content on the internet, which may contain potential misinformation (Pan et al., 2023; Chen and Shu, 2023).
Through extensive empirical analyses, we reveal several factors contributing to this bias: (i) confirmation bias (Xie et al., 2023) is not a key factor (§5.1): LLMs maintain a significant preference for generated contexts even when they are inconsistent with LLMs’ parametric knowledge. (ii) text similarity is a significant factor (§5.2): compared to retrieved contexts, generated contexts typically exhibit a higher degree of similarity to the questions, even when they contain incorrect information. The samples with a larger similarity gap between generated and retrieved contexts exhibit a more pronounced bias. (iii) semantic completeness matters (§5.3): LLMs tend to favor contexts with semantic integrity. The segmentation process used in retrieved contexts may disrupt their completeness, thereby hindering their full utilization in LLMs. This finding emphasizes the need for optimizing passage segmentation in current retrieval systems.
In summary, we explore the challenges LLMs face when utilizing conflicting contexts and make the following contributions: • We uncover a critical bias in existing LLMs,
where they heavily rely on generated contexts regardless of correctness, indicating an insufficient
use of diverse information sources.
• To facilitate controlled experiments, we develop a specialized framework for constructing tailored datasets and excluding confounding factors, e.g. input order, and context length.
• Our extensive analyses have identified two key factors, i.e., text similarity and semantic completeness, in the context utilization of LLMs.

heading: 2 Background & Study Formulation
text: In this section, we briefly review three categories of LLMs augmented with auxiliary information for QA tasks: retrieval-augmented, generationaugmented, and hybrid approaches. Additionally, we introduce the framework of our investigation.

heading: 2.1 Background
text: Figure 2 presents high-level abstract frameworks for three typical types of QA systems, each centered around an LLM as the reader component, and potentially incorporating additional components like a retriever, generator, or a blend of both, tailored to the specific methodology. Retrieval-Augmented Approach. As shown in Figure 2a, for a given question q in a set of questions Q, these approaches (Guu et al., 2020; Lewis et al., 2020; Ram et al., 2023; Gao et al., 2023) initially use a retrieval model γ to select the top k relevant documents Dγk=γk(q,C)={d γ 1 , . . . , d γ k} from a corpus C={d1, . . . , d|C|}. Then, a reader (often LLM) ϕ employs these documents Dγk to generate an answer aγϕ, expressed as a γ ϕ = ϕ(q,D γ k). Generation-Augmented Approach. In contrast, as illustrated in Figure 2b, these works (Yu et al., 2022; Sun et al., 2023; Liu et al., 2022) involve an LLM as a generator ϱ to produce k tailored background contexts Dϱk=ϱk(q)={d ϱ 1, . . . , d ϱ k} for a give question q, thereby enhancing the utilization of the LLM’s internal knowledge. These LLMgenerated contexts Dϱk form the input for reader ϕ to produce the final answer: aϱϕ = ϕ(q,D ϱ k). Hybrid Approach, as depicted in Figure 2c, combines retrieved and generated contexts to enhance performance (Yu et al., 2022; Abdallah and Jatowt, 2023), as aϕ = ϕ(q,D γ k , D ϱ k). These hybrid approaches face a significant challenge: conflicts between diverse sources can impede the effectiveness of information integration (Zhang et al., 2023). Knowledge Conflicts within Contexts. These studies mainly focus on conflicts within a single type of input contexts, either only retrieved (Chen et al.,
2022) or generated (Xie et al., 2023), leaving underexplored how LLMs resolve conflicts between diverse contexts.

heading: 2.2 Answer Tracing Task
text: Departing from previous research, our study investigates the mechanisms by which LLMs merge contexts from diverse sources in hybrid approaches. As illustrated in Figure 3, we design a task to ascertain whether an answer aϕ originates from generated contexts Dϱk or retrieved contexts D γ k . For a more controlled and simpler analysis, we limit the context to a single instance from each source, i.e., k=1 and aϕ=ϕ(q, d γ 1 , d ϱ 1). Then, by comparing the answer aϕ with the answers derived from the retrieved context aγϕ and the generated context aϱϕ, we can determine its source, thereby analyzing the merging mechanism of LLMs.
We specifically focus on non-tunable LLMs, i.e. in zero-shot settings, to reflect prevalent real-world use cases like ChatGPT. This direction is motivated by the high cost and limited accessibility of finetuning, which makes the direct use of non-tunable LLMs popular. Additionally, given the extensive use of LLMs, any bias or issue in their merging mechanisms could lead to serious consequences.

heading: 3 Experimental Setup
text: To facilitate our investigation into how LLMs merge generated and retrieved contexts, this sec-
tion elaborates on the construction of our contextconflicting datasets and the evaluation metric.

heading: 3.1 Context-Conflicting Datasets
text: As depicted in Figure 4, in our dataset Dcc, each sample x is a quintet (q, dγ1 , d ϱ 1, a γ ϕ, a ϱ ϕ), where d γ 1 is the context returned by retriever γ for question q, dϱ1 represents the context generated by LLM ϱ, aγϕ and a ϱ ϕ are the candidate answers provided by the reader ϕ, each based solely on the respective contexts dγ1 and d ϱ 1. To guarantee that our dataset is suitable for controlled experiments aimed at investigating the merging mechanisms of LLMs, it should adhere to specific criteria: • Traceability: aγϕ and a ϱ ϕ should be supported by
their corresponding contexts, dγ1 and d ϱ 1.
• Exclusivity: Only one of the contexts, dγ1 or d ϱ 1,
provides the correct answer, i.e., either aγϕ or a ϱ ϕ
matches the gold answer of question q. Such constraints establish a solid basis to identify which context, generated or retrieved, is selected by LLMs to produce answers in hybrid approaches.
We utilize the dev and test sets of two opendomain QA benchmark datasets with golden answers, i.e., NaturalQuestions (NQ) (Kwiatkowski et al., 2019) and TriviaQA (TQA) (Joshi et al., 2017), to assemble our experimental datasets. The overall pipeline for dataset construction is depicted in Figure 4, with detailed steps outlined as follows: Context Preparation. Step 1 in Figure 4 illustrates the process of preparing contexts for each question. For retrieved contexts, it is obtained from the top-1 ranked passage from Wikipedia using Contriever (Izacard et al., 2021), a powerful off-the-shelf retrieval model that is extensively employed in various retrieval-augmented generation systems (Shi et al., 2023; Ram et al., 2023).
For generated context, we follow the GenRead (Yu et al., 2022), instructing the generator, e.g.,
LLM like GPT-4, to generate a background document based on the question. All LLMs in this paper, unless otherwise noted, have a temperature setting of zero to ensure result reproducibility. However, this method often yields contexts much longer (>250 words) than the retrieved contexts (typically truncated to ∼100 words (Karpukhin et al., 2020; Izacard et al., 2021)). The discrepancy in length could potentially affect the merging mechanisms of LLMs (Xie et al., 2023). To exclude this disturbance, we regulate the length of the generated context by incorporating length constraint in the prompt, resulting in an average length discrepancy below 3%. All subsequent experiments, unless otherwise specified, employ this method to eliminate the impact of length variations. More details can be found in Appendix A.1. Sample Filtering for Traceability. With each question paired with a single context (either generated or retrieved) established in the initial stage, the reader generates the corresponding candidate answer, as shown in Step 2 of Figure 4. To unravel the mechanisms of LLMs in context merging, it is essential to ensure the traceability, i.e., the output answer is derived from the input context, rather than the intrinsic parametric knowledge of the LLMs. To achieve this, we only keep samples in which both the generated and retrieved contexts exactly include their respective generated answers, exemplified by aγϕ∈d γ 1 , where ∈ denotes dγ1 contains the answer string aγϕ. This practice is grounded in the findings of Chen et al. (2022); Xie et al. (2023), which demonstrate that in the presence of external context, LLMs mostly rely on external context rather than their intrinsic parametric knowledge. Despite these efforts, the complete elimination of the influence from parametric knowledge remains challenging. Therefore, we separately examine its impact in §5.1, where we demonstrate that it has a
negligible effect on our conclusions. Building Context-Conflicting Dataset. Having obtained answers for each type of context, we are now positioned to construct our context-conflicting (CC) datasets, as depicted in Step 3 of Figure 4. Initially, We employ the exact match metric (Yu et al., 2022) to evaluate the correctness of candidate answers derived from contexts, considering an answer correct if its normalized form matches any of the golden answers.
Subsequently, the context-conflicting datasets are composed of samples for which only one of the two types of contexts, either generated or retrieved, yields the correct answer, thereby ensuring the exclusivity. Notably, each dataset comprises two distinct subsets: AIG, consisting of samples with correct answers only in the generated context; and AIR comprising samples with correct answers only in the retrieved context.

heading: 3.2 Statistics of Datasets
text: For each reader-generator pair, we respectively construct context-conflicting datasets from test and dev sets of NQ and TQA: NQ-CC (NQ-AIG + NQAIR) and TQA-CC (TQA-AIG + TQA-AIR).
We initially adopt a typical and simple setting in which an LLM serves as both the generator and reader. Table 1 provides statistics for the constructed subsets corresponding to various LLMs, including GPT-4 (gpt-4-0613), GPT-3.5 (gpt-3.5turbo-0613), Llama2-7b/13b (Llama2-7b/13b-chat
(Touvron et al., 2023)). The statistics show that the context-conflicting subsets form a substantial part of the datasets, underscoring the need to investigate how LLMs integrate these distinct contexts. Notably, GPT-4 has fewer conflicting instances than other LLMs, because of its higher efficacy in answering questions using either solely retrieved or generated contexts.
Section 4.2 also explores a more complex scenario in which the generator and reader are distinct LLMs, with the statistics shown in Appendix A.2.

heading: 3.3 Evaluation Metric
text: Besides datasets, we also develop metrics to study how LLMs merge generated and retrieved contexts in hybrid approaches. Specifically, the selection of LLMs towards either generated or retrieved context can be measured by the proportion of answers that exactly match the answer produced solely by the corresponding context, denoted as ρgen=avg(em(aϕ, a ϱ ϕ)), ρret = avg(em(aϕ, a γ ϕ)) where em(a, b) returns 1 if a exactly match b, and 0 otherwise. The proportion of instances where aϕ does not match either a ϱ ϕ or a γ ϕ is negligible to the conclusion in this work, as demonstrated in Table 17. To facilitate a simple and efficient experiment, we define a synthesized metric as follows:
DiffGR = ρgen − ρret ρgen + ρret
(1)
The metric DiffGR, ranging from [−1, 1], quantifies the extent of LLMs’ tendency to rely on generated contexts over retrieved contexts. Using AIR as an example, where all correct answers come from retrieved contexts, an ideal DiffGR is −1, i.e., LLMs should always rely on retrieved contexts.

heading: 4 How LLMs Merge Contexts?
text: This section conducts experiments on the constructed datasets to investigate the merging mechanism of the LLMs in hybrid approaches. We first consider a typical setting where the generator and reader share a single LLM, to explore how LLMs merge retrieved and self-generated contexts (§4.1). Then, we extend our experiments to include more flexible combinations of generator and reader (§4.2), and various retrieval models (§4.3), to investigate their effects.

heading: 4.1 LLMs Prefer Self-Generated Contexts
text: Our preliminary experiments, in which a single LLM serves both as generator and reader, are designed to explore how LLMs integrate information
from retrieved and self-generated contexts. The LLMs under evaluation are tasked with answering questions using both types of contexts on their corresponding CC datasets. In all experiments, we employ a randomized input sequence of contexts to mitigate the influence of order, which is further discussed in Appendix B.3.
We begin our analysis by examining LLMs’ QA performance on CC datasets to reveal how well can LLMs utilize both types of contexts. Table 2 presents the Exact Match scores (Yu et al., 2022) across various LLMs. Surprisingly, LLMs demonstrate significantly low performance (≤ 22.16%) on AIR subsets, despite the fact that the retrieved context alone consistently yields the correct answer on these subsets. In contrast, LLMs exhibit strong performance on AIG subsets (most near 90%). Overall, all LLMs exhibit a significant performance gap between AIR and AIG datasets, with a pronounced decline in performance when the correct answers come from retrieved contexts.
To further reveal LLMs’ behavior underlying the QA performance, we trace the source contexts of LLMs’ answers using the proposed DiffGR metric. An ideal LLM should always rely on retrieved contexts on AIR subsets (DiffGR = −1), and always rely on generated contexts on AIG subsets (DiffGR = 1). Contrary to expectations, Figure 5 illustrates that LLMs fail to identify the correct information and consistently tend to rely on generated contexts on both AIG and AIR subsets. This result indicates a pronounced bias in LLMs to favor generated contexts, even when they provide
incorrect information. This bias leads to the insufficient utilization of retrieved contexts mentioned above and highlights a critical challenge for existing LLMs in effectively merging generated and retrieved contexts. As the bias on AIR subsets has a more direct impact on the performance, the following experiments and analysis will focus on the biases on these subsets to conserve space. Results on the AIG subsets can be found in Appendix B.1.

heading: 4.2 LLMs Broadly Prefer Generated Contexts
text: The above experiments reveal the bias in LLMs to favor the self-generated context. A question emerges: Do LLMs also prefer contexts generated by other LLMs? To investigate this question, this section extends the experiments to more flexible combinations of generators and readers. This setting is also of practical significance, as recent works have explored the decoupling of generators and readers to achieve modularization of knowledge (Luo et al., 2023; Feng et al., 2023).
We construct context-conflicting datasets for each (generator, reader) pair respectively. The statistics of these datasets are shown in Appendix A.2. Based on these datasets, we then compute DiffGR metric to examine biases across various (generator, reader) pairs, as shown in Figure 6, and observe two notable insights: (i) LLMs are also biased towards contexts generated by other LLMs. This suggests that such bias in LLMs is widespread and not limited to self-generated contexts. (ii) LLMs usually exhibit a stronger bias to contexts generated by themselves. The sole exception is Llama2-7b, which shows the strongest bias when paired with Llama2-13b as its generator. This phenomenon likely results from their highly similar model structures and training processes (Touvron et al., 2023).

heading: 4.3 Consistent Preferences Across Retrievers
text: The above experiments have demonstrated the bias in LLMs against contexts retrieved by Contriever, a dense retrieval model. This section further explores whether such bias remains consistent across different retrieval models. To this end, we incorporate BM25 as a representative of sparse retrieval techniques, along with human-labeled golden passages, to mimic an ideal retrieval model (Gold). For each retrieval model, we respectively construct AIR context-conflicting datasets based on the NQ dev sets from Karpukhin et al. (2020), which include the golden passages. As illustrated in Table 3, the results indicate that LLMs consistently display a pronounced bias in favor of generated contexts regardless of the retrieval model used.

heading: 5 Why LLMs Prefer Generated Contexts
text: In this section, we investigate the causes of the observed bias from several perspectives: the effect of parametric knowledge in §5.1, context similarity to the question in §5.2, and context completeness in §5.3. This section primarily presents the result for Llama2-13b and GPT-3.5, while the results for the other LLMs are provided in Appendix B.

heading: 5.1 Effect of Parametric Knowledge
text: Recently, Xie et al. (2023) demonstrated that LLMs exhibit a bias towards contexts consistent with their parametric knowledge (or memory), a phenomenon termed confirmation bias. This section explores whether the observed bias arises from the potential consistency between generated contexts and parametric knowledge. To investigate this, we select subsets where the answers from “retrieved context”, “generated context”, and “parametric knowledge” were all different from one another.
To achieve the subset, we first establish the
LLM’s parametric knowledge about a question using a closed-book QA task, following Xie et al. (2023), which can be expressed as allmϕ = ϕ(q). Then, we select the cases that satisfy allmϕ ̸= a γ ϕ ̸= aϱϕ from the AIR datasets constructed in §3. Table 4 shows the number of samples after filtering.
The filtered datasets not only exclude the effect of confirmation bias but also help us identify whether LLMs use parametric knowledge or contexts to answer the question. Table 5 illustrates the proportion of LLMs choosing the answer provided by generated contexts, retrieved contexts, or parametric knowledge. We observe two key insights: (i) The proportion of choosing LLMs’ parametric knowledge is very small. This result is consistent with the findings of several previous works (Xie et al., 2023; Chen et al., 2022), which found that LLMs mostly rely on the context even when it conflicts with the parametric knowledge. (ii) LLMs still show a significant preference for generated contexts when excluding the influence of parametric knowledge. This suggests that the confirmation bias is not the primary cause of the observed bias.
To ensure rigorous analysis, we use the filtered data in Table 4 for subsequent experiments to eliminate any potential distractions.

heading: 5.2 Effect of Text Similarity
text: The text similarity between a context and a question can reflect the degree of their relevance. To investigate the potential effect of the similarity, we employ Jaccard similarity and BERTScore (Zhang et al., 2020) to analyze the contexts on the constructed context-conflicting datasets with the reader and generator sharing a single LLM. Figure 7 shows that generated contexts exhibit a significantly higher similarity to the question on AIR
subsets, despite the fact that generated contexts are incorrect on these subsets. This similarity discrepancy between generated and retrieved contexts persists whether assessed by term-based overlap (average 0.37 vs. 0.18 on TQA-AIR) or semantic similarity (0.90 vs. 0.86).
To further clarify the influence of the observed similarity discrepancies, we rank the samples according to the similarity gap ∆sim between generated and retrieved contexts.
∆sim = sim(q, dϱ)− sim(q, dγ) sim(q, dϱ) + sim(q, dγ)
Here, sim(q, dϱ) is the similarity between generated context and question, and sim(q, dγ) is for retrieved context. Then, we divide the dataset into n (n = 5 here2) slices with an equal number of samples. Ensuring that each slice contains an equal number of samples helps to avoid fluctuations caused by sample size variations.
2Similar results and observations are found with other n.
Figure 8 illustrates the relationship between the average ∆sim within each slice and the corresponding DiffGR3. From the results, we observe a general trend that LLMs exhibit an increased bias to generated contexts on slices with a larger average similarity gap, which indicates that text similarity is a significant factor in the preference for generated contexts. These findings suggest that generated contexts should be applied with greater caution to mitigate the influence of highly relevant but misleading information.
To facilitate understanding why the similarity affects LLMs’ preference, we include some examples in Table 16. From these cases, we observe that contexts with higher similarity often support candidate answers more straightforwardly, for instance, by mirroring the phrasing used in the questions. Conversely, the contexts with low similarity introduce more challenges, often necessitating an understanding of synonyms and even some inferences.

heading: 5.3 Effect of Context Completeness
text: In all the above experiments, there is a key difference between generated and retrieved contexts that may affect the context preference: semantic and sentence completeness. Concretely, current retrieval systems typically employ fixed-length truncation to divide a complete article into multiple passages, which serve as the fundamental units for retrieval tasks (Karpukhin et al., 2020; Wang et al., 2019; Zhu et al., 2021). This truncation often results in retrieved contexts with incomplete semantic meaning, as well as sentences that are cut off at beginnings or endings. In contrast, generated contexts in the above experiments are naturally produced by LLMs (Nature), resulting in enhanced semantic and sentence completeness.
To investigate the potential effects of completeness on the observed bias, we conduct controlled experiments that vary the semantic and sentence
3We also tried to manipulate the context similarity to investigate its impact, but we found it challenging to instruct LLMs to generate contexts with low similarity to the question while still providing an answer to it.
completeness of generated contexts4 using the following methods: (a) Truncation (Trunc.) eliminates the length constraints from the generation prompt of §3, allowing LLMs to generate extended contexts. These generated contexts are then truncated to match the length of retrieved contexts, thereby simulating both semantic and sentence incompleteness of retrieved contexts. (b) Sentence Truncation (S-Trunc.): Based on the method (a), we truncate generated contexts only at the end of a sentence to preserve the sentence completeness, while simulating the semantic incompleteness.
To eliminate the interference of similarity factors, we select questions whose three types of generated contexts have nearly equivalent similarity, with BERTScore differences less than 0.055. Table 6 demonstrates that three types of generated contexts also have similar average lengths after filtering. This means that both the influences of similarity and length are mitigated, thereby highlighting the principal disparities in semantic and sentence completeness.
We evaluate LLMs’ preference between generated versus retrieved context, varying the completeness of generated context, following the same pipeline in §4.1. Table 7 presents the DiffGR with different semantic and sentence completeness in generated contexts. A comparison between “Trunc.” and “S-Trunc.” reveals that sentence completeness has a very slight impact on LLMs’ preference for generated contexts. In contrast, comparing “Nature” and “S-Trunc.”, we find a significant increase in bias towards generated contexts that are semantically more complete. These findings indicate that LLMs tend to favor contexts with enhanced semantic completeness, underscoring the necessity to investigate improved passage segmentation methods that maintain semantic completeness for current retrieval-augmented LMs.
4We also tried to vary the completeness of retrieved contexts but found it challenging to isolate it from confounding factors like length. This aspect is left for future work.
5We also tried several other thresholds for similarity differences, e.g. 0.005, all yielding same conclusions.

heading: 6 Related Work

heading: 6.1 Generation-Augmented Approaches
text: Generation-augmented methods prompt LLMs to generate intermediate contexts for the final response, thereby leveraging their extensive parametric knowledge acquired during the pre-training phase on vast text corpora (Roberts et al., 2020; Petroni et al., 2019). These generated contexts may encompass various types of knowledge, such as background knowledge (Sun et al., 2023; Yu et al., 2022), commonsense knowledge (Liu et al., 2022), domain-specific knowledge (Feng et al., 2023; Luo et al., 2023), and chain-of-thought reasoning processes (Wei et al., 2022; Kojima et al., 2022). Despite their effectiveness, the LLM-generated knowledge may contain hallucinations (Chen et al., 2023; Ji et al., 2023) due to LLMs’ outdated memory (De Cao et al., 2021) and limited memory for longtail knowledge (Kandpal et al., 2023). This inaccuracy information could potentially mislead current retrieval model (Dai et al., 2023) and open-domain question answering systems (Pan et al., 2023).

heading: 6.2 Retrieval-Augmented Approaches
text: Retrieval-augmented methods (Guu et al., 2020; Lewis et al., 2020; Ram et al., 2023; Gao et al., 2023) enhance LLMs by integrating relevant documents from external corpora, addressing limitations like the need for knowledge updates (Jang et al., 2022) and long-tail knowledge (Kandpal et al., 2023). Early methods (Guu et al., 2020; Lewis et al., 2020; Izacard et al., 2022) focused on joint training of LLMs and retrievers. Recent studies (Ram et al., 2023; Shi et al., 2023) append relevant documents directly to the input while keeping LLMs static. Despite their effectiveness, these methods face challenges with irrelevant retrievals and incomplete knowledge coverage (Yu et al., 2023; Mallen et al., 2023). These noisy retrieval results can misguide LLMs (Mallen et al., 2023; Yoran et al., 2023; Ren et al., 2023).

heading: 6.3 Hybrid Approaches and Knowledge
text: Conflicts
Recent works investigate merging retrieved and generated contexts to leverage both sources of knowledge (Abdallah and Jatowt, 2023; Yu et al., 2022; Zhang et al., 2023) and achieve improved performance over those relying solely on a single information source. However, conflicts between diverse sources can impede the effectiveness of
information integration (Zhang et al., 2023) Current research on knowledge conflicts primarily focuses on two aspects: context-memory conflict and inter-context conflict (Xu et al., 2024). Regarding context-memory conflict, Xie et al. (2023); Chen et al. (2022) find that LLMs are highly receptive to the input contexts rather than their internal memory. Concerning inter-context conflict, Chen et al. (2022) demonstrates that LLMs tend to rely on a few most relevant retrieved contexts. Additionally, Xie et al. (2023) reveals that LLMs favor contexts consistent with their parametric knowledge when confronted with both supporting and opposing contexts. In contrast to these studies which are limited to conflicts within a single type of context, our work further considers conflicts between generated and retrieved contexts and reveals a bias in LLMs.

heading: 7 Conclusion and Future Work
text: In this study, we propose a framework to investigate the underlying mechanisms by which LLMs merge retrieved and generated contexts. Our results reveal a pronounced bias towards generated contexts in several LLMs (GPT 3.5/4 and Llama27b/13b). We further identify two key factors that may contribute to this bias: higher similarity between generated contexts and questions, and the semantic incompleteness of retrieved contexts. Our insights highlight the critical need for advanced integration methods that can validate and leverage information from both sources, moving beyond the current overreliance on generated contexts.
While this study primarily focuses on integrating generated and retrieved contexts, the observed bias also highlights a critical risk of retrieval-augmented LLMs. With LLM-generated content increasingly prevalent on the internet, retrieval results may include generated contexts (Chen and Shu, 2023). As this scenario becomes more common, the observed bias of LLMs toward generated contexts implies their susceptibility to misinformation and malicious attacks generated by LLMs. This raises concerns about the security of retrieval-augmented systems, a critical problem that is gaining attention in recent works (Pan et al., 2023; Zou et al., 2024). Addressing the challenges posed by the widespread presence of generated content, such as the detection of such content, represents a promising direction for future research.

heading: Limitations
text: Our work has the following limitations: • This study is confined to open-domain question
answering, a representative knowledge-intensive task. The behavior of LLMs across a broader spectrum of natural language processing tasks remains to be further explored.
• This work does not propose specific solutions to effectively mitigate the observed bias, as we focus on revealing the phenomena and analyzing the causes.
• To create a controlled environment conducive to analysis, we utilize a single instance for each context type. LLMs face increasingly intricate conflict scenarios when handling multiple contexts from each type. These conflicts emerge not only between retrieved and internally generated contexts but also among the various contexts originating from the same source (Chen et al., 2022; Xie et al., 2023).

heading: Ethics Statement
text: Data All data used in this study are publicly available and do not pose any privacy concerns.
AI Writing Assistance In our study, we only employed ChatGPT to polish our textual expressions rather than to generate new ideas or suggestions.

heading: Acknowledgements
text: This work was supported by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103704), the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0680202), the Innovation Funding of ICT, CAS (E361120), the National Natural Science Foundation of China (No.62172393), and Major Public Welfare Project of Henan Province (No.201300311200).

heading: A Detailed Datasets Statistics

heading: A.1 Length Control for Generated Contexts
text: In our proposed framework, we regulate the length of generated contexts by incorporating length constraints in the prompt:
Generate a background context from Wikipedia to answer the given question {#question}. Keep the length of the document around n words.
We observed that GPT 4 effectively controls the output length, whereas other models struggle with this aspect. To address this issue in the latter, we employ multiple values of n and select the one that best matches the retrieved context.
As a result, Table 8 illustrates the average length of different contexts. Figure 9 shows the length distribution of retrieved contexts and contexts generated by various LLMs. The length distribution of retrieved contexts is more concentrated as they consist of text limited to precisely 100 words, along with their titles (Karpukhin et al., 2020). The variation in the length of different retrieved contexts is solely due to the differences in title lengths.

heading: A.2 Dataset Size
text: Table 9 presents the data size of context-conflicting datasets corresponding to various generator-reader pairs. The statistics indicate that conflicting data comprise a substantial proportion across all combinations of generators and readers.

heading: B Additional Results

heading: B.1 More Results on AIG Datasets
text: Figure 10 shows the DiffGR with different (reader, generator) pairs on their corresponding NQ-AIG datasets. It can be observed that LLMs show a strong tendency to rely on generated contexts across various (reader, generator) pairs.

heading: B.2 More Results about Effect of Parametric Knowledge
text: Table 10 illustrates the size of the filtered AIR datasets with the constraint “allmϕ ̸= a γ ϕ ̸= a ϱ ϕ”.
GPT-4 corresponds to fewer samples because it has better parametric knowledge, i.e., allmϕ is more likely to be correct. Table 11 illustrates the proportion of LLMs choosing the answer provided by generated contexts, retrieved contexts, or parametric knowledge.

heading: B.3 Effect of Context Order
text: In the above experiments, retrieved and generated contexts are presented in random order. Previous studies (Xie et al., 2023; Liu et al., 2023; Lu et al., 2022) have found that the model may be sensitive to the order of the input contexts. In their experiments, the input context was either all retrieved (Liu et al., 2023) or all generated (Xie et al., 2023). We conducted experiments to investigate whether the context order impacts the preference for the generated context. The generated and retrieved contexts are concatenated with three different orders: generated-first, retrieved-first, and random. To control the cost of API, this section conducts experiments on the context-conflicting datasets from only the test sets of NQ and TQA. We compute the DiffGR with different context orders respectively.
As shown in Table 12, across all context orders, LLMs consistently show a strong tendency to favor generated contexts. When the retrieved context is positioned first, there is a slight reduction in DiffGR. This reduction may result from the LLMs’ preference for generated contexts being partially offset by their bias towards the top context (Liu et al., 2023; Xie et al., 2023).

heading: B.4 More Results about Effect of Context Completeness
text: (a) Length Distribution. In §5.3, we employ three methods, “Nature”, “Trunc.” and “S-Trunc.”, to vary the completeness of generated contexts, while controlling the length at the same time. Figure 11 illustrates the length distribution for generated contexts corresponding to these methods. From the results, we can observe that the contexts generated by original GenRead (Yu et al., 2022) are significantly longer compared to the retrieved contexts. (b) Similarity and Length Control. Table 13 presents the data quantity after filtering out samples where three types of generated contexts exhibit significant differences in similarity. Table 14 illustrates the average similarity and completeness of three types of generated contexts. “Nature”, “Trunc.” and “S-Trunc.” result in contexts with Comparable average length and similarity, with preliminary differences in completeness.

heading: B.4.1 Cases about Completeness
text: Table 15 provides some examples to facilitate the understanding of completeness. From the cases, we observe that retrieved contexts and “Trunc.” often contain incomplete sentences. Additionally, compared to “S-Trunc”, “Nature” typically exhibits greater semantic completeness. Specifically, “Nature” often encompasses a full logical structure of an article, including an introduction, discussion, and conclusion, whereas “S-Trunc” may terminate abruptly.

heading: B.5 More Results about Effect of Text Similarity
text: (a) Similarity Metric. We employ Jaccard similarity to assess the term-based overlap, and BERTScore (Zhang et al., 2020) for evaluating the semantic similarity between contexts and questions. To mitigate the effect of length discrepancies between contexts and questions, we calculate the similarity at the sentence level and then aggregate them to derive the overall context-question similarity. In this work, we adopt a maximum aggregation strategy due to the single-hop nature of the NQ and TQA datasets, where the majority of questions
can be answered using a small subset of sentences. We also try the average aggregation strategy and observe similar results.
Figure 12 illustrates the distribution of similarity when employing maximum and average aggregation methods. It is observable that the generated contexts exhibit a markedly higher degree of similarity regardless of the aggregation method used. Furthermore, this disparity in similarity is more pronounced with maximum aggregation, as contexts typically contain sentences that are irrelevant, which dilute the similarity scores when an average aggregation is applied. (b) Similarity Distribution. Figure 13 and 14 show the similarity distribution of retrieved and generated contexts across various generators. All LLM-generated contexts exhibit a higher similarity over retrieved contexts. (c) Effect of Similarity. Figure 15 demonstrates a general trend that on slices with a smaller average similarity gap, LLMs exhibit a reduced preference for generated context. (d) Cases about Similarity. Table 16 shows examples that contain contexts with different similarities to the question. The contexts with high similarity typically directly support answering by repeating the phrasing in the question. Conversely, the contexts with low similarity introduce more challenges, often necessitating an understanding of synonyms and even some inferences. These observations indicate that text similarity can partly reflect the relevance between a question and a context, as well as the difficulty the LLM encounters in identifying potential answers.

heading: B.6 The Effectiveness of Exactly Matching
text: Table 17 demonstrates that the proportion of “Others” is significantly lower relative to the disparities between “Gen” and “Ret”, its impact on the conclusions of this paper is negligible.
TQA-AIR Example TQA-AIG Example
Question Between 1959 and 1967 which city was the capital of Pakistan (Islamabad was being built)? Who is the most successful UK solo artist in the USA?
answers in the context in pink, and the correct answers in the context in green.

heading: Future Work
text: Addressing the challenges posed by the widespread presence of generated content, such as the detection of such content, represents a promising direction for future research