Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents

Authors: evgenii  kortukov
alexander  rubinstein
elisa  nguyen
seong joon oh
mary  quant
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 11:26:14

Abstract: Retrieval-augmented generation (RAG) mitigates many problems of fully parametric language models, such as temporal degradation, hallucinations, and lack of grounding. In RAG, the model’s knowledge can be updated from documents provided in context. This leads to cases of conflict between the model’s parametric knowledge and the contextual information, where the model may not always update its knowledge. Previous work studied context-memory knowledge conflicts by creating synthetic documents that contradict the model’s correct parametric answers. We present a framework for studying such knowledge conflicts in a realistic setup. We update incorrect parametric knowledge using real conflicting documents. This reflects how knowledge conflicts arise in practice. In this realistic scenario, we find that knowledge updates fail less often than previously reported. In cases where the models still fail to update their answers, we find a parametric bias: the incorrect parametric answer appearing in context makes the knowledge update likelier to fail. These results suggest that the factual parametric knowledge of LLMs can negatively influence their reading abilities and behaviors. Our code is available at https://github.com/kortukov/realistic_knowledge_conflicts/.

Main paper: 
 introduction: 
 Retrieval-augmented generation systems (RAG) combine a generative language model with a non-parametric datastore. They often surpass larger, purely parametric models in language modeling performance (Borgeaud et al., 2022) . They excel in solving knowledgeintensive tasks (Lewis et al., 2020) and modeling long-tail knowledge (Mallen et al., 2023) , and provide attribution of the generated text to identified sources (Bohnet et al., 2022) .A particularly attractive feature of RAG is the ability to update a system's world knowledge without the need for costly retraining. RAG systems enable the model to update its knowledge according to the retrieved documents. Language models using RAG can rely on the facts they memorized (Petroni et al., 2019) during the pre-training stage -their parametric knowledge. Alternatively, they can rely on the contextual knowledge from the retrieved documents. If the two sources contradict each other, we speak of a knowledge conflict (Longpre et al., 2021) , also referred to as context-memory conflict (Xu et al., 2024) . A knowledge update happens when the model changes its original parametric answer upon seeing a conflicting context. Knowledge conflicts happen in three important RAG applications. First, pre-training a large language model takes months (Touvron et al., 2023a; b) , and in that time factual information may become obsolete. Second, in the currently prevailing transfer learning paradigm (Bommasani et al., 2021) most end users don't train models from scratch. Instead, they rely on a few pre-trained models and adapt them to downstream tasks by fine-tuning, prompting, or retrieval augmentation. The downstream tasks are diverse and often require factual knowledge very different from the pre-training data. Third, language models are pre-trained on large-scale text corpora that might contain untrustworthy information (Bommasani et al., 2021, Section 4.6) . Failure to update the parametric knowledge with correct domain-specific Table 1 : Comparison of knowledge updating approaches. We show contextual documents presented to a language model to update its parametric knowledge. Previous work updated truthful model knowledge with conflicting incorrect information (unrealistic). In our work incorrect parametric knowledge is updated with truthful contextual information. 
experimental design: 
 We set out to understand how often do LLMs update their knowledge when presented with real factual documents that contradict their incorrect parametric answers? As discussed in § 2, previous work only studied knowledge updating behavior with synthetic documents contradicting correct model knowledge. We introduce a novel framework for studying knowledge-updating behaviors in LLMs in § 3.1. Then, we formalize the problem setup in § 3.2. Details of our empirical analysis are then described in § 3.3.We propose an experimental framework that mirrors real-world RAG application and allows us to study LLM knowledge-updating behaviors. We focus on the question-answering (QA) task. First, we identify a subset of data where parametric knowledge is insufficient. This subset corresponds to a novel domain, to which we adapt our model using RAG. Then we run the model on this subset providing the retrieved documents.We assume perfect retrieval to focus on the LLM behaviors. Therefore, documents always contain ground-truth answers and conflict with the incorrect parametric knowledge of the model. To simulate "golden" retrieved passages that always contain the correct answers we rely on open-book (extractive) QA datasets. We simulate in-context retrieval, a popular way to deploy LLMs with RAG (Lewis et al., 2020; Ram et al., 2023; Shi et al., 2023) .Our experimental framework is a three-stage categorization of samples (cf. Figure 1 ) that enables us to study knowledge updating under knowledge conflicts.Stage 1: Closed-book answer gathering. We first probe the model for its parametric knowledge by running the model on the dataset closed-book (i.e., without context). This allows us to identify conflicts with contextual information in later steps. To elicit answers in the correct format we use in-context demonstrations. Then, we save the model's answers for further filtering.Stage 2: Filtering out no-conflict examples. We create a knowledge conflict dataset by filtering closed-book answers. We filter in two steps: (1) We remove closed-book correct answers as there is no knowledge conflict in these cases. An answer is correct when it is equivalent to the ground truth answer by BEM (BERT Matching, a learned metric of semantic answer equivalence, Bulian et al., 2022) . (2) We further remove any answer that does not contradict the context, e.g. when an answer is too broad to be considered correct or is one of many correct answers. Similar to prior work (K öksal et al., 2023; Xie et al., 2024) , we use a natural language inference (NLI) model for this. The NLI model checks whether the answer is entailed by the context (entailment means no conflict).Stage 3: Open-book QA under knowledge conflict. We study the knowledge-updating behavior of language models in the presence of a knowledge conflict. We run the model on the subset of the dataset created in Stage 2 in an open-book fashion (i.e. providing the context). The answers are categorized into 3 disjoint sets:• Correct update -knowledge update succeeded, and the model answered correctly according to the context. • Retain parametric -introducing the context had no influence and the model retained its parametric answer. § 4.3 and § 4.4 ) and how in-context demonstrations affect the success of the knowledge update (Appendix F).Let D = {(q i , c i , a i )} N i=1 be an open-book QA dataset of (question, context, answer) triplets. If a language model f is queried closed-book (without context) we write f (q) = a p for the parametric answer; f (q, c) = a c means we query the model open-book (with context), where a c is its contextual answer with c as context. a gt is the ground-truth answer. If two answers a 1 and a 2 are an exact match we write a 1 = a 2 . If two answers are equivalent (semantically equal) we write a 1 ∼ = a 2 .Definition 1 A knowledge update happens when the model changes its answer upon seeing the context: f (q) ̸ = f (q, c).Definition 2 A knowledge update is correct when upon seeing the context the model changes its answer to the correct one: f (q) ̸ = f (q, c) ∼ = a gt .In Stage 3 we use U c for the Correct update category of examples. For this subset a c ∼ = a gt . The symbol R then stands for the Retain parametric examples for which (a c ̸ ∼ = a gt ) & (a c = a p ). Then the Incorrect update category is referred to as U i . It includes examples for which (a c ̸ ∼ = a gt ) & (a c ̸ = a p ). We write the proportion of a given category A among all open-book QA answers as P(A). Additionally, we label the event of the parametric answer appearing in the context as (a p ⊆ c).Datasets. We examine knowledge updates across a wide range of open-book QA datasets that represent diverse question-answering scenarios: Natural Questions (NQ) (Kwiatkowski et al., 2019) , SQuAD (Rajpurkar et al., 2016) , NewsQA (Trischler et al., 2017) , TriviaQA (Joshi et al., 2017) , SearchQA (Dunn et al., 2017) , and HotpotQA (Yang et al., 2018) . We use the unified format versions distributed in the 2019 MrQA Shared Task (Fisch et al., 2019) .We study the 7B and 70B models of the Llama2-family (Touvron et al., 2023b) due to their wide adoption. We study the Mistral-7B (Jiang et al., 2023) , a model that outperforms Llama-2 of the same size on many benchmarks. Additionally, we study Mixtral-8x7B, a sparse mixture-of-experts model. For brevity, we report results on Llama2-7B, Mistral-7B, and Mixtral-8x7B. In Appendix H we report additional studies with Llama-70B. For comprehensiveness, we also include results for one closed-source model, OpenAI GPT-3.5-Turbo, presented in Appendix J.When evaluating the model on a QA dataset both closed-book (Stage 1) and open-book (Stage 3) we employ the BEM metric (Bulian et al., 2022) instead of the commonly used Exact Match (EM) to identify correct and wrong examples. BEM (BERT Matching) is a model-based metric of answer equivalence that allows for a more precise example-level QA quality evaluation. We support this design decision in Appendix E. 
observations and analysis: 
 In this section, we apply the experimental setting proposed in § 3 to study how often, when, and how language models fail to update their parametric knowledge.We first create a dataset with realistic knowledge conflicts. Given an open-book QA dataset and a language model, we gather its closed-book answers (Stage 1) and filter out those that do not present a conflict between parametric and contextual information (Stage 2). Using the knowledge conflict dataset, we study the knowledge-updating behavior of LLMs. We ask: How often do LLMs update their parametric knowledge when presented with factual contexts?In ) where the model retains its parametric answer (P(R)), successfully updates its answer to the correct contextual one (P(U c )), or updates its answer incorrectly (P(U i )), as defined in § 3.2. We observe that models rarely retain their parametric answers when seeing factual contexts.We investigate the remaining cases in the R subset where the knowledge update did not happen. See Appendix C for small-scale analysis of the U i error class. We focus on the R subset as these are critical failures of RAG. If the incorrect answer is not changed after providing retrieved documents that means RAG did not work to update model knowledge as intended (Lewis et al., 2020) . We aim to understand and potentially mitigate these errors.In this analysis, we ask two questions: (a) How are examples in the R category different from other examples and (b) how can we use that information to better understand the knowledge update failures?The initial manual investigation uncovered a potential explanation: in a large portion of examples in the R subset, the incorrect parametric answer a p appears in the retrieved document c. For example:Question: Who was the main performer at this year's halftime show? Document: CBS broadcast Super Bowl 50 in the U.S., and charged an average of $5 million for a 30-second commercial during the game. The Super Bowl 50 halftime show was headlined by the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Super Bowl XLVIII halftime shows, respectively. It was the third-most watched U.S. broadcast ever.Coldplay Incorrect parametric answer: To quantify this initial observation, we measure how often the incorrect parametric answer a p appears in context c in each answer category in Stage 3 of our framework. We find that in the R subset this happens much more often than on average in the dataset and in other subsets.Beyoncé N Q S Q u A D N ew sQ A T ri v ia Q A S ea rc h Q A H oFigure 2 shows that for Llama2-7B the incorrect parametric answer appears in context in 32% -88% of the examples in the R subset compared to only 5% -28% in the full dataset. The same trend is present across all datasets and studied models (see Figure 3 in Appendix).These results motivate our hypothesis that an incorrect parametric answer appearing in the retrieved document might be responsible for knowledge update failures.We test this hypothesis in the next sections. We hypothesize that there exists a type of confirmation bias induced by the parametric knowledge that potentially explains the majority of the knowledge update failures. We ask the question: does the appearance of the parametric answer in the context increase the likelihood of a knowledge update failure?More formally, we ask: is there a statistically significant difference between P(R | a p ⊆ c) and P(R | a p ̸ ⊆ c)?We report the results of this analysis in Table 4 . For Llama2-7B there is an increase in failure likelihood ranging from 4%pt. up to 11%pt. across all datasets. For the Mistral model, the difference also exists, ranging from 1%pt. to 10%pt. For Mixtral-8x7B the difference ranges from 2%pt. to 10%pt. The difference in failure likelihood is statistically significant at α = 0.05 for all 6 datasets and 3 studied models. We provide the details of our statistical analysis in Appendix D.We call this effect parametric bias. We define it as a bias when the incorrect parametric answer of the model appearing in context negatively influences its reading ability.These findings suggest that the parametric answer of a language model makes knowledge update more likely to fail when it appears in the context document.To develop a better understanding of the parametric bias phenomenon, we design two intervention experiments. First, to test if the appearance of a p in context c was the reason behind the model retaining its parametric answer, we mask a c when it appears in context and observe the changes in model behavior. Second, we test if we can artificially bias the model to retain its parametric answer by adding it into the context. If true, this would provide additional evidence for the existence of a parametric bias.We re-run Stage 3 of our experiment and mask the incorrect parametric answer tokens a p . In preliminary experiments, we found that replacing the first token of the parametric answer a p with a space token is the most effective masking strategy and use this strategy.We find that masking the parametric answer a p reduces the probability of the model retaining its parametric answer, as reported in Table 5 . For Llama2-7B the likelihood of retaining the original answer drops by up to 1.6 percentage points, for Mistral-7B and Mixtral-8x7B, the maximum drop is 0.4 %pt. and 1.8 %pt. respectively.However, a small number of cases stay in the Retain subset, indicating that the found parametric bias might not be the only reason for the model to retain its parametric answer when presented with a conflicting context.Additionally, we find that the accuracy P(U c ) also drops by up to 0.6, 0.9. and 1.8 percentage points for Llama2-7B, Mistral-7B and Mixtral-8x7B respectively. This indicates that the examples that were influenced by the incorrect parametric answer now moved into the Incorrect update category U i . We suspect that models retain the incorrect answer not only due to the parametric bias but in some cases, due to limited reading abilities. If parametric bias were the only reason, masking the parametric answer should move all those examples into U c .Llama2-7B Mistral-7B Mixtral-8x7B Dataset P(R) P(U c ) P(R) P(U c ) P(R) P(U c ) NQ 0.7 (-0.7) 79.3 (-0.3) 0.2 (-0.2) 78.5 (-0.9) 0.8 (-0.9) 76.8 (-0.1) SQuAD 0.2 (-0.2) 89.9 (-0.4) 0.1 ( 0.0) 85.0 (-0.3) 0.0 (-0.1) 87.1 (-1.8) NewsQA 0.5 (-0.3) 71.4 (-0.6) 0.1 (-0.1) 67.3 (-0.8) 0.5 ( 0.0) 71.8 (-0.9) TriviaQA 2.9 (-0.5) 79.6 (+0.3) 3.0 (-0.3) 78.8 (+0.2) 6.2 ( 0.0) 74.0 (-0.3) SearchQA 0.7 (-1.5) 60.9 (-0.6) 0.3 (-0.4) 59.1 (-0.8) 1.6 (-1.8) 69.2 (-0.3) HotpotQA 0.5 (-0.8) 79.6 ( 0.0) 0.2 (-0.4) 78.5 ( 0.0) 0.6 (-0.6) 81.7 (-0.6) Table 5 : Knowledge update success rate (%) after masking the parametric answer a p if it appears in the context. Size of change in parenthesis. We find that the models retain their parametric answers (a p ) less often when a p is masked.We further test our parametric bias hypothesis by artificially adding the parametric answer to the context. If the bias exists, we expect the model to retain its parametric answer more often. We add the incorrect parametric answer to the prompt after the context. We separate it from the context and the question by the words "Unrelated text", indicating to the model that it should be ignored. Below is an example of such a prompt: prompt = f"""Answer the question with as few words as possible by extracting information directly from the context. Context: In the United Kingdom, BBC Radio 5 Live and 5 Live Sports Extra will carry the contest. The BBC will carry its own British English broadcast, with Greg Brady, Darren Fletcher and Rocky Boiman on commentary. Unrelated text: talkSPORT Question: Aside from BBC Radio 5, what radio station will broadcast the game? Answer:""" Such an addition would not fool a human into reading a context incorrectly because of the words "Unrelated text". Hence, we claim this is evidence of an existing bias and consider it a failure of the model if this addition influences the model's performance.We report the results in Table 6 . This simple intervention increases the likelihood of the model retaining its parametric answer for all datasets, providing further evidence for the existence of the parametric bias. Table 6 : Knowledge update success rate (%) after adding the parametric answer a p to the context after words "Unrelated text:". Size of change in parenthesis. We find that the models retain their parametric answer (a p ) more often when adding a p to the context.In general, our intervention experiments show that the parametric factual knowledge of a language model interferes with knowledge updating, biasing its ability to read the context and jeopardizing the reliability of RAG systems. 
discussion: 
 Over-reliance on parametric knowledge. Previous work (Longpre et al., 2021; Si et al., 2023) found that LLMs retain their parametric answers after seeing a counterfactual conflicting context in a significant number of cases. Our findings show that in realistic knowledge conflicts, this rarely happens. These results suggest that LLMs ignoring conflicting information from retrieved documents is not a significant practical problem. However, a problematic interaction between parametric and contextual knowledge exists in a realistic scenario.Parametric bias. We show that despite high knowledge update success rates, the openbook performance of LLMs is not independent of their parametric factual knowledge. An incorrect parametric answer can make the knowledge update more likely to fail if it appears in the retrieved document. In Appendix I we provide evidence that parametric bias might become an even larger problem in RAG systems with changing knowledge and complex and realistic documents.Solutions. Our work focuses on understanding and analyzing the parametric bias. We have uncovered a problem and introduced a way of evaluating its effect. We invite the community to work together to find a solution. As a first step in this direction, we try several simple solutions. In Appendix F we show that better task adaptation through ICL is not enough to mitigate this bias. Furthermore, in Appendix H we show that model scaling might reduce the bias. However, even the largest popular open LLMs are still susceptible to it. Finally, in Appendix G we show that task-specific finetuning can reduce the parametric bias, although it is not enough to completely eradicate it.In our experiments, we assume perfect retrieval: a single relevant document that always contains the correct answer is provided as context to the LLM. This assumption defines the scope of applicability of our findings. In reality, retrieval could bring noisy, outdated, or irrelevant documents into the LLM prompt. However, ensuring correct model behavior in our studied setting is a necessary step if the goal is to achieve factual correctness using RAG. Asserting the relevance and quality of the retrieved documents is another, complementary step. Both of these steps are necessary for the final goal of factually correct outputs. 
conclusion: 
 In this work, we studied the knowledge-updating behaviors of LLMs. This is the first work to study context-memory knowledge conflicts as they appear in practice, with factual real-world documents. Contrary to prior work that introduced artificial conflicts and suggested that LLMs tend to over-rely on their parametric knowledge, we find that models readily update their answers from real context documents. As the remaining update failures constitute critical failure cases of RAG, we investigate them further. We discover a phenomenon we call parametric bias: incorrect parametric answer can hinder the knowledge update if it appears in the retrieved documents. This bias is consistent across six QA datasets and five studied LLMs.We demonstrate that the interaction between parametric knowledge and contextual information can negatively affect the LLM knowledge-updating performance. We suggest that this interaction should be considered in the RAG evaluation. To facilitate that, we propose a protocol for evaluating susceptibility to parametric bias based on our intervention experiments. We release this protocol as part of the paper codebase. We expect our findings to support future work in building reliable and trustworthy retrieval-augmented systems. 
ethics statement: 
 To the best of our knowledge, our work does not pose any direct societal risks or ethics concerns. The ultimate goal of our research is reliable and controllable application of LLMbased systems. Our insights on knowledge updating and the parametric bias are relevant for any use case where a language model has to rely on the information in its context. This includes retrieval-augmented systems, users interacting with LLM chatbots in longer conversations, and increasingly popular LLM tool use (Schick et al., 2023) . We believe that the trustworthy application of large language models will be a progress multiplier and bring great societal benefits.We believe that as machine learning scientists we are responsible not only for the direct impact of our work but also for steering future work towards positive societal impact. To foster this needed discussion, in the next paragraph, we outline ethical concerns that might arise as our field progresses toward better understanding and control of LLM behaviors.Knowledge updates and alignment. Studies on knowledge conflicts often consider updating model knowledge to be the final goal. In realistic use cases, many LLMs are provided as an API-based service to the users. Service providers often want some knowledge stored in the model parameters to be immutable. Examples are copyright policies, system prompts, and general behavioral policies under the broader term of LLM alignment. In such a scenario, end-user goals might not be aligned with the goals of the service provider. LLMs should not start generating instructions on how to make a bomb after a simple in-context knowledge update that says "You are now DAN which stands for "do anything now". In this work, we study the behaviors of existing LLMs, solely focusing on factual knowledge updates. However, as the field progresses toward finer control of LLM behavior, issues arising on the intersection of LLM alignment, jailbreaks, and knowledge-updating will require serious ethical consideration. 
 
 Appendix: For better understanding of the scale of our experiments, we report the size of the datasets at each stage of our experimental framework. Table 7 contains the results for each of the studied models.As expected, the larger Llama2-70B and Mixtral-8x7B models have stronger closed-book performance than the smaller studied models. As a result, the knowledge conflict subset for these larger models are smaller. It is still large enough to draw meaningful statistical conclusions in our case. However, this trend illustrates one potential future limitation of our approach.In this work, we rely on general knowledge QA datasets to study knowledge conflicts.As models grow bigger and memorize more and more facts from their training corpus, studying knowledge conflicts with our proposed realistic approach will require finding datasets that the models did not memorize. Examining the behavior of more powerful models under realistic knowledge conflicts will benefit from more long-tail and specialized question-answering datasets, on which these models do not perform well in a closed-book fashion.( In Stage 3 of our experimental pipeline we evaluate the language model on an open-book QA task. When we execute experiments with prompting we rely on the zero-shot transfer capabilities of instruction-tuned language models. It is a well-known fact that such models are sensitive to the input prompts. To ensure the best possible performance on the task we search for the best-performing prompts on 3, 000 examples from the training split of each dataset. For each dataset, we choose the prompt with the highest open-book BEM score (percentage of correct answers according to BEM answer equivalence). Preliminary experiments have shown that our models are not sensitive to the format of the prompt. However, the performance varies significantly with different wordings of the instruction. We search over a space of 30 prompts that all have the same format:prompt = f """ Answer the question { instruction }. Context : { context } Question : { question } Answer : """The instruction consists of two parts that encode our knowledge of the answer format in the studied datasets.instruction = f" { brevity_instruction } { context_reference } ""All possible options for each of the parts is listed in Table 8 . The resulting optimal instruction wordings for each dataset are reported in Table 9 .-with as few words as possible -concisely -in the most compact form -using minimal text -by being succinct Context References -using the context -by reading from the context -by consulting the provided context -through the information given in the context -by extracting information directly from the context -by copying verbatim from the context In § 4.3 and § 4.4 we focus on understanding the R subset of examples where the incorrect parametric answer is retained. In this section, we explain why we focus on this seemingly smaller subset of incorrect open-book answers.When determining the importance of a problem, frequency should not be the sole factor. For example, adversarial attacks rarely happen in practice, yet there is substantial research to ensure the adversarial robustness of models. Analogously, the R subset may appear to be a fringe case, but it is a well-defined problem for which we provide a deeper analysis.Out of all incorrect open-book answers, the R subset contains only the ones that did not change at all after introducing the context. This specificity, together with our analysis is what makes this subset actionable.The U i subset, however, includes all other incorrect answer categories "lumped" together.If you look into the U i subset, it is not one "prevalent scenario", but rather many of them.To illustrate this, we manually label 100 examples in the U i subset for the Mixtral8x7B model on the Natural Questions dataset. We report the results in Table 10 . As can be seen, analyzing the U i subset would actually require separating each sub-category. This would turn each of them into actionable, but small subsets, similar to R.Answer has an incorrect format. 10Predicted answer is also correct, but not equal to the ground-truth answer. 13Predicted answer is actually correct, while the dataset label is incorrect. 12 Answer is incorrect due to the model being bad at reading tables. 13The predicted answer was part of the context, but not the correct answer. 4Model predicts an answer that does not appear in context. 5The model wrongly claims that the answer is not present in the context. 3The context is illegible and or not enough to answer correctly. In our conflicted open-book QA experiments, all examples naturally split into two disjoint groups: in one the incorrect parametric answer a p appears in context c (see § 4.3.1 for an example), and in the other group it does not. We report the sizes of the groups in Table 11 . We model each of the groups of examples as a sequence of independent Bernoulli trials with an unknown success probability π. In our model success in the Bernoulli trial corresponds to a successful knowledge update, so π is the theoretical quantity that the empirical probability P(U c ) estimates.We ask the question: is there a statistically significant difference in the successful knowledge update probabilities in these two groups? We approach this problem from a Bayesian perspective and model the success probability π U c as a random variable. The number of successes in each group follows a Binomial distribution with m successes out of n trials:p(m|n, π) = m n π m (1 -π) n-m (1)We choose the prior distribution on π to be a uniform Beta distribution:p(π) = B(π; 1, 1) (2)The beta distribution is a conjugate prior to the Binomial likelihood and is the common choice when modeling success probabilities. The specific choice of parameters makes the prior uniform, encoding our lack of knowledge a priori. The posterior over π after seeing m successes in n trials is then computable in closed-form and is also a Beta distribution:EQUATIONLet π 0 correspond to the success probability in (a p ̸ ⊆ c) group and π 1 is the success probability in (a p ⊆ c) group. Our null hypothesis is then: H 0 : π 0 = π 1 . This null hypothesis defines a predictive distribution p(m 1 |H 0 ) over the observed number of successes in the group where a p ⊆ c.To test the null hypothesis we compute the probability of observing m 1 or less successes out of n 1 trials under the assumption that the null hypothesis is correct p(m 1 |H 0 ). To compute the predictive distribution on m 1 we need to marginalize over the posterior on π:p(m 1 |H 0 ) = p(m 1 |n 1 , π)p(π|m 0 , n 0 )dπ (4)This results in a Beta-binomial distribution:EQUATIONWe use this distribution to compute the final p-values by taking the value of the probability mass function at the observed value of m 1 . Final results are reported in the main text in Table 4 . Full KC Data R Subset Full KC Data R Subset Full KC Data R Subset Figure 3 : Frequency of samples that contain the incorrect parametric answer in the context (a p ⊆ c) in the full knowledge conflict (KC) data, the Retain subset (R) and the Correct update subset (U c )for each studied LLM. Across all datasets and models, the percentage of context documents containing the incorrect parametric answer is largest in R.N Q S Q u A D N e w sQ A T ri v ia Q A S eU c Subset (a) Llama2-7B N Q S Q u A D N e w sQ A T ri v ia Q A S eU c Subset (b) Llama2-70B N Q S Q u A D N e w sQ A T ri v ia Q A S eU c Subset (c) Mistral-7B N Q S Q u A D N e w sQ A T ri v ia Q A S e(a) Llama2-7B In our three-stage experimental pipeline ( § 3.1), we prepare the data by filtering out examples that were correct closed-book or where the model answer did not conflict with the context. Unlike previous work, we focus on the closed-book incorrect examples. Thus, we need to identify incorrect model answers with high precision.The commonly used Exact Match (EM) metric is prone to false negatives, as many conceptually equivalent answers may have different surface forms (and hence are considered different by EM). As pointed out by Bulian et al. (2022) , EM only provides a lower bound on the system-level performance. However, EM is a weak example-level metric due to a large number of false negatives. Very often the model answers correctly but the surface form of the answer differs from the label (Napoleon's vs. Napoleon, three vs. 3 or 50-140 cm vs. 0.5-1.4 m). In our pipeline, using Exact Match would lead to a lot of actually correct (false negative) model answers making it into Stage 3 of the pipeline. This would inflate the number of cases where the model retains its parametric answer, as there would be no reason for the model to update its answer since there is no conflict. For the same reason, we need to filter out incorrect but not conflicting examples in Stage 2 of the pipeline using a separate NLI model that predicts whether the answer is entailed by the context. Bulian et al. (2022) introduced BEM and showed it to be a better example-level metric than EM. We ported1 the BEM metric to HF Transformers library and reproduced the original results. On the original data2 BEM reaches 89% accuracy while EM only reaches 55 %. We further tested BEM on manually labeled subset of HotpotQA3 , where BEM resulted in 92% accuracy while EM only achieved 22%.We perform an ablation study to illustrate the importance of BEM and NLI filtering by running the experiment on Llama2-7B without filtering and using the EM metric. We report the results in (-31.8) 45.3 (+25.6) Table 12 : Ablation study: knowledge update success rates for the Llama2-7B model when measured using Exact Match instead of BEM and not applying filtering. These results contain a lot of noise and thus are inflated. Size of change due to noise, compared to Table 3 in parenthesis.Using a weak evaluation metric may lead one to wrongly conclude that the model often retains an incorrect parametric answer. This experiment highlights the importance of the strong answer equivalence metric (BEM, (Bulian et al., 2022) ) and the filtering stage in our experimental pipeline. This design decision is crucial in identifying samples with a realistic knowledge conflict.Motivation In-context learning (ICL) is known to improve language model performance on a wide array of NLP tasks. ICL involves providing several labeled examples as part of the context. It is considered to be a method of adapting a general pre-trained model to perform a specific task. It is simpler and requires fewer resources than fine-tuning, but is generally more effective than instruction prompting alone.RQ Can in-context demonstrations minimize the influence of the discovered parametric bias?Evaluation setup The Llama2 model has a context length of 2048 tokens which does not allow for many in-context demonstrations. We therefore use five in-context examples for Natural Questions, SQuAD, and HotpotQA datasets, as they have shorter context lengths. For the other three datasets, we use two in-context demonstrations.The Mistral-7B model employs sliding window attention, which enables flexible context lengths. Mixtral-8x7B has context length of 32768. This allows us to run the ICL experiment with eight context demonstrations for each dataset on both of those models.The results for the Llama2 family are presented in Table 13a and Table 13b . The 7B model continues to retain its parametric answer in roughly the same number of cases. Moreover, the incorrect parametric answer in the context still increases the probability of a knowledge update failure. For the larger 70B model, the trend is similar overall. Interestingly, for TriviaQA and SearchQA datasets Llama2-70B retains its parametric answer even more often with in-context demonstrations than without them. Furthermore, the negative influence of the parametric answer on update probability exists for all datasets. We conclude that for the Llama2 model family, in-context demonstrations do not minimize the influence of the parametric bias.The results for Mistral-7B and Mixtral-8x7B are reported in Table 13c and Table 13d . The influence of in-context demonstrations is two-fold: it improves open-book performance (P(U c )), but also makes the model retain the incorrect parametric answer more often than without ICL. Furthermore, the parametric bias (difference in failure likelihood) is slightly exacerbated after introducing ICL for the majority of datasets.Limitations Due to a limited context window, shown examples might not suffice for the model to learn the task. This phenomenon was explored in the literature for text classification tasks by the name of "demonstration bias" (Fan et al., 2024) . This could explain the marginal improvement in the performance and ICL failing to prevent parametric bias. Motivation Fine-tuning a pre-trained language model on a downstream task is a powerful task adaptation method. Fine-tuning involves further training an LLM on a set of labeled examples. It requires access to model weights and more computational resources than ICL or prompting but generally enables stronger downstream task performance. One caveat of fine-tuning is that by changing the model weights we may lose the generality of an instruction-tuned model. We compare the fine-tuned model with the original prompted version used in the main experiments.First, we report our observations on knowledge-updating behaviors and the influence of an incorrect parametric answer naturally appearing in the context in Table 15 . As expected, fine-tuning improves open-book accuracy across all studied datasets. We find that fine-tuning reduces the chance of the model retaining its original incorrect answer. The fine-tuned version retains the original answer in only 1.2% of cases on average across datasets, compared to 1.6% for the prompted model. Regarding the parametric bias, we find that when an incorrect parametric answer naturally appears in context -its influence on the failure likelihood is smaller for the fine-tuned model across all datasets.Additionally, we run an intervention study, to compare how the fine-tuned model responds to artificially adding the incorrect parametric answers to context. We follow the setup in § 4.4.2.The results are reported in Table 16 . The fine-tuned model is still susceptible to adding the incorrect parametric answer in context, however, to a lesser extent than the prompted version. For the original model, the likelihood of retaining the incorrect answer increases by 7.9% on average across all datasets. For the fine-tuned model the average increase only reaches 4.3%.Discussion Fine-tuning reduces the parametric bias in our experiments. However, in the examined setup fine-tuning was not enough to completely eradicate it. Overall, our results suggest that fine-tuning is the most promising approach to combat the parametric bias in RAG systems. However, it comes at the cost of potentially losing the generality of an instruction-tuned LLM. Motivation Given enough training data, larger LLMs tend to perform better on both the pretraining objective and downstream tasks.RQ How does the size of a language model influence its knowledge-updating behavior and susceptibility to parametric bias?Evaluation setup We compare two models of the same architecture that differ only in parameter count: Llama2-7B and Llama2-70B.Results First, we report our observations on knowledge updating behaviors and the influence of an incorrect parametric answer naturally appearing in the context in Table 17 . We find that 7B and 70B models have very similar behaviors with respect to knowledge updating: Llama2-7B retains its original incorrect answer in only 1.6% of examples on average and Llama2-70B does that on average for 2% of examples. We also find that the two models are similarly susceptible to failing their knowledge updates when their incorrect parametric answer appears in context.Additionally, we run an intervention study, to compare how models of different sizes respond to artificially adding their incorrect parametric answers to context. We follow the setup in § 4.4.2.The results are reported in Table 18 . Both models are susceptible to parametric bias: adding the incorrect parametric answer to the context increases their likelihood of retaining it even after seeing the context. For Llama2-7B the increase is on average by 7.9 %pt. For the larger 70B model, this increase is on average by 4.8 %pt.These experiments provide evidence that increasing the model size can be a viable, although costly, solution to mitigate the discovered parametric bias. However, among the currently popular open models, even the largest ones are susceptible to this bias.(a) Llama2-7B 9.1 (-2.9) 68.0 (+0.9) SearchQA 3.3 (-2.1) 72.0 (-0.2) HotpotQA (-0.6) 87.5 (-0.5) In our experiments, GPT-3.5-Turbo exhibits the same behaviors as the studied open-source models. It readily updates its knowledge from real documents and it is also susceptible to the discovered parametric bias.BEM model available at: https://huggingface.co/kortukov/answer-equivalence-bemOriginal answer equivalence dataset: https://huggingface.co/datasets/kortukov/ answer-equivalence-datasetHotpotQA answer equivalence dataset: https://huggingface.co/datasets/kortukov/ ae-gpt4-hotpotqa 


Decision: The paper presents an empirical study/analysis of LLM's behavior under knowledge conflict -- between model's internal parametric knowledge and provided in-context documents. The paper is clearly written with valid evaluation setting. While they study slightly newer angle to the analysis on parametric vs. non-parametric knowledge -- looking at cases where model's initial prediction was incorrect and provided in-context passage is correct. The reviewers were concerned this only account for a small subset of examples, but these are still valuable subset. The takeaways/actionable items from the analysis is a bit unclear, and the scope of analysis is a bit limited (and overlaps with rich prior work). 



One point: 
I agree with reviewer 2iuH that this setting is not necessarily "realistic" than prior settings, and recommend changing the title. Also -- 
"Re: prior work studies "synthetic document"" is not a valid representation of prior work. "Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence" Chen at al EMNLP 2022, Section 5 studies knowledge conflict where real documents with conflicts are drawn from different time stamped corpus or different interpretation of the question.

[comments from the PCs] Please revise your paper to better represent prior work as the AC notes.

Reviews: review: paper_summary: paper_summary: Past work on LLM knowledge conflicts has shown that models struggle to correctly use in-context passages to answer the question and fall back on the parametric answer that they have memorized. Follow-up work showed that this is less true if the context is modified sufficiently to serve as convincing evidence. However, all these approaches relied on artificial edits to the context passage.

This paper proposes that LLM behavior in knowledge conflicts should be tested when (1) the parametric answer is incorrect and (2) the correct evidence passage is provided. Authors find that even in this realistic scenario where the context is not artificially edited, LLMs again predict their parametric answer a certain percentage of the time. Additionally, a large proportion of contexts result in the LLM in predicting a different incorrect answer. This behavior is established across several QA datasets and for 3 open-source models: Llama2-7B/70B, Mistral-7B, Mixtral-8x7B.

They further study the cause of this concerning behavior. They identify a cause they name `parametric bias': the model predicts the parametric answer again when it reappears in the context even though there is a correct answer elsewhere. That is, there is a confirmation bias where models predict their parametric answer even if it occurs in unrelated contexts.
strength_weakness: strength_weakness: - I have one major question for the authors: why did you focus on the smaller fraction of answer retention errors over the larger fraction of errors where the model predicts a different but incorrect answer?
    - This is concerning because this category of errors contributes to a larger fraction of errors overall
    - In all ablation experiments, the fraction of errors of this kind seems to be increasing but are largely ignored
questions: questions: 1. Table 3: As stated before, what is in the error category $U_i$?
2. Table 4: Can yous separate out $P(R | a_p \subseteq c)$? Why did you choose to combine $P(R \cup U_i | a_p \subseteq c)$?
3. Sec 4.4.1: Since both $P(R)$ and $P(U_c)$ are decreasing, that means that $P(U_i)$ is increasing i.e. the model is predicting neither the parametric nor the correct contextual answer. Is the model predicting the masked (edited) parametric answer instead (thus putting it in the $U_i$ set? What do yo see is the error case?
4. Sec 4.4.1: How do you draw this conclusion? The last line is the definition of $R$ and not $U_i$. “This indicates that the examples that were influenced by the incorrect parametric answer now moved into the Incorrect update category $U_i$. *This suggests that the model might be resorting to its original answer in cases when it cannot find the correct answer in the context.*”
5. Sec 4.4.2: What explains the increase in $P(U_i)$ (in 7 of 18 settings) when we add back the parametric answer with the phrase "unrelated context"?
paper_summary: paper_summary: This work proposes to study another scenario of LLM knowledge conflicts: when the LLM parametric knowledge is wrong and the retrieved document contains evidence that supports the correct answer. Experiments demonstrate that models are much more likely to shift in this setting, while model behavior might be correlated with the occurrence of parametric/incorrect answers in the retrieved document.
strength_weakness: strength_weakness: - I'm not sure settings in previous works where the parametric answer is right and retrieved docs are wrong are "unrealistic". Retrieval could bring noisy, outdated, or irrelevant documents into LLM prompts, so it is definitely a valid setting. What could support the setting in this work, instead of claiming it to be more "realistic", is an empirical evaluation of retrieval-augmented LMs and see how often both scenarios happen. If the setting in this work is much more prevelant, it would strengthen the motivation of this work.

- According to Table 3, it seems that in the setting of focus in this work (parametric knowledge is wrong, retrieved docs are right), LLMs overwhelmingly rely on the retrieved document >95-98% of the time. This probably indicates that LLMs consistently favor the retrieved documents under these settings and we just need to improve retrieval quality, and there probably is very little to investigate about knowledge conflicts in these scenarios.

- Much of the analysis (most of Sec 4.3 and 4.4) focus on the $\mathcal{R}$ cases, i.e. the parametric answer is retained. However, it only constitute a very small fraction of cases (<5% usually). I wonder if more effort should be spent on investigating the $\mathcal{U}_i$ cases where the answer changed but still wrong.

- An important point is that we often don't know right and wrong at inference time for either parametric answers or retrieved documents. The findings and proposed "interventions" rely heavily on the assumption that "parametric is wrong but retrieved is right", which might not be applicable when the correctness is not known beforehand.

- The intervention experiments might be a bit trivial technically, simply adding or removing the parametric answer in the retrieved document. This might not be a great issue though.

- It would be nice to present some qualitative analysis and/or examples in the main paper.
paper_summary: paper_summary: This paper studies knowledge conflicts under a new setup: incorrect parametric knowledge conflicts with ground-truth knowledge in context. The paper discovers parametric bias, which means knowledge
updates can be impeded if incorrect parametric answers appear in the retrieved documents. Extensive experiments are conducted to demonstrate this.
strength_weakness: strength_weakness: 1. While the setup differs from previous works, the main finding seems not very novel.
2. The paper does not provide a solution. It mainly focuses on analyzing and demonstrating a phenomenon.
3. The paper emphasizes knowledge conflicts under its setup are realistic. However, in reality, the retrieved knowledge might be noisy and not necessarily be the ground truth.
questions: questions: 1. Why the happening of knowledge updates is determined by exact match? Will changing it into semantic equality change the results much?
paper_summary: paper_summary: This paper proposes a framework to investigate the knowledge conflict problem in the RAG scenarios. Compared with previous work, the authors claim that they use real conflicting documents rather than synthetic information, which is closer to the real-world setting. According to this paper, LLMs are easy to update their parametric memory when external conflicting evidence is presented in the context. Furthermore, the authors find a 'parametric bias' phenomenon: It's hard for LLMs to update their parametric memory if there is parametric memory-related information in the context, although it is incorrect and unrelated to the question.
strength_weakness: strength_weakness: 1. In stage 1, the authors mention using in-context demonstration to elicit answers. However, could this introduce demonstration bias? The authors should clarify this aspect in the paper.
2. In stage 2, the authors mention using BEM to match the answers. What is the accuracy of BEM? Can it ensure matching quality? The authors should provide clarification on this in the paper.
3. Given the widespread use of closed-source LLMs in real-world RAG scenarios, it's essential to include at least one closed-source LLM to ensure the findings can be generalized to these models.
4. Why do the authors mention "tune the prompts" in Stage 3? I've checked the appendix, but I'm still confused about this point. Does it mean the authors have fine-tuned the models, or have they simply selected one of the best prompts? If it's the former, could this introduce another training bias since the parameters have been changed? If it's the latter, the authors should reconsider the terminology.
5. The first finding doesn't seem entirely new, as it has been reported in "Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts," which is also referenced in this paper. Specifically, Xie et al. found that LLMs are highly likely to update their parametric memory as long as the external evidence is coherent. Could you please justify the difference?

(I will reconsider the score if the authors can address the points mentioned above to my satisfaction.)
questions: questions: 1. Could the phenomenon of 'masking reducing the likelihood of retaining parametric answers' be attributed to low-quality masking? It's possible that LLMs are strong enough to identify such obvious misinformation?

Comments: title: Comment after rebuttal
Reply to the Authors (2)
Reply to the Authors
comment: After reading the author responses to all reviews, I am somewhat convinced by their answer to my concern. 

| this category of errors contributes to a larger fraction of errors overall

The authors address this with two observations:
1. The category of errors where the model updates to a different incorrect answer is also a fragmented category. Explaining any such update would also require explanations for relatively small fractions of errors
2. They report a semantic matching metric (instead of EM) which further highlights that the "retention" errors are higher than initially estimated.

| the fraction of errors of this kind (new incorrect answer) seems to be increasing but are largely ignored

I am convinced by their breakdown of this error category.

I am increasing my score from 6 -> 7.
We thank the reviewer for the response. We would like to expand a bit on some of the points mentioned by the reviewer.
>it is often very hard to make these assumptions for retrieval-augmented LMs at inference time due to the potential noise/irrelevance of retrieval

The assumption of "parametric is wrong but retrieved is right" does indeed define the scope of applicability of our findings. However, ensuring correct model behavior in this setting is a **necessary** step if the goal is to ensure factual correctness using RAG. Ensuring relevance and quality of the retrieved documents is another, complementary step. **Both** of these steps are necessary for the final goal of factually correct outputs.

>In addition, the analysis often focuses on the most unlikely scenarios (<5%, points 2 and 3 in the review)

We think this is an important disagreement point and would like to explain our thinking and observations. We hope this provides more insight into why we focus on the R subset.

Frequency is not the sole factor in determining the importance of a problem. For example, adversarial attacks rarely happen in practice, yet there is substantial research to ensure adversarial robustness of models. Analogously, the R subset may appear to be a "fringe case", but it is a well-defined problem for which we provide a deeper analysis.
Out of all incorrect open-book answers, the R subset contains only the ones which did not change at all after introducing the context. This specificity, together with our analysis is what makes it actionable.

Ui subset, however, includes all other incorrect answer categories "lumped" together.
If you look into the Ui subset, it is not one "prevalent scenario", but rather many of them:
We manually classify 100 samples in 𝑈𝑖 for the Mixtral on NQ:
- 42 incorrect answer format
- 10 model answer is also correct, but != GT
- 13 model answer is correct while the GT is incorrect
- 12 model being bad at reading tables
- 13 incorrect reading (prediction appears in context, but != GT)
- 4 answers not from context
- 5 model wrongly claims the answer is not in the context
- 3 context is illegible

Analysing the Ui subset, as proposed by the reviewer, would actually require separating each sub-category. This would turn each of them into actionable, but small subsets, similar to R.

>the authors acknowledge that analysis could be expanded to more prevalent scenarios

We would like to clarify that we intended to add to the revision the manual analysis above for the Ui subset, together with the provided motivation for focusing on the seemingly smaller but actionable R subset.
Thank you for the detailed response. I have no further questions. I will keep the score of 6.
I would like to thank the authors for the detailed response. Overall I understand why this work matters in the constrained setting of "parametric is wrong but retrieved is right", but it is often very hard to make these assumptions for retrieval-augmented LMs at inference time due to the potential noise/irrelevance of retrieval. In addition, the analysis often focuses on the most unlikely scenarios (<5%, points 2 and 3 in the review): the authors acknowledge that analysis could be expanded to more prevalent scenarios in knowledge conflicts instead of the fringe cases ("We'll also add an analysis of the 𝑈𝑖 cases to the revision"), but I'm not sure if this is entirely within the scope of camera ready revision.
Thank you for your prompt and detailed explanation. I have no further questions. Looking forward to the revision.
Thank you for considering the rebuttal and changing your score. We will improve the revision and include the discussed points.
We also re-checked Xie et al.'s work and would like to elaborate on the difference between ours and their work:

We refer to single-source experiments (§4.1 in Xie et al.) when we say they studied coherent yet false contexts.
In these experiments, Xie et al. provide a "counter-memory" context which they synthetically generate to contradict the model's knowledge.
Our setting is also single-source, but the contexts are real and factually true, hence we claim our setting is more realistic.

Xie et al. also have multi-source evidence experiments (§4.2).
These experiments contain, as you rightly point out, "both correct and false contexts".
Their multi-source experiments focus on "the evidence preference of LLMs", while we focus on knowledge-updating behavior.
Their experiments show LLMs exhibit a *confirmation bias* - they often prefer evidence supporting their parametric knowledge.

Our realistic single-source experiments extend their findings. We show that real documents meant to update knowledge can unintentionally bias the model toward its wrong parametric answer (our section 4.3.1). This can be framed as a confirmation bias of sorts, which we term "parametric bias". The parametric bias describes the case when models fail to update their knowledge even if the evidence is **not** supporting the parametric knowledge but just includes the parametric answer.
We will add this distinction to the paper.
Sincerely thank you for the detailed response. I have adjusted my score accordingly. I encourage the authors to incorporate these discussions into the next version. One last comment: If I remember correctly, Xie et al. studied both correct and false contexts. Perhaps the authors should verify this and include the differences in the next version.