Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents
Authors: Evgenii Kortukov
Alexander Rubinstein
Elisa Nguyen
Seong Joon Oh
Venue: COLM (2024)
Keywords: large language model, retrieval augmentation, knowledge conflict
TL;DR: Studying context-memory knowledge conflicts as they appear in practice: how does factual knowledge influence LLM reading behaviors?

Abstract: Retrieval-augmented generation (RAG) mitigates many problems of fully parametric language models, such as temporal degradation, hallucinations, and lack of grounding. In RAG, the model’s knowledge can be updated from documents provided in context. This leads to cases of conflict between the model’s parametric knowledge and the contextual information, where the model may not always update its knowledge. Previous work studied context-memory knowledge conflicts by creating synthetic documents that contradict the model’s correct parametric answers. We present a framework for studying such knowledge conflicts in a realistic setup. We update incorrect parametric knowledge using real conflicting documents. This reflects how knowledge conflicts arise in practice. In this realistic scenario, we find that knowledge updates fail less often than previously reported. In cases where the models still fail to update their answers, we find a parametric bias: the incorrect parametric answer appearing in context makes the knowledge update likelier to fail. These results suggest that the factual parametric knowledge of LLMs can negatively influence their reading abilities and behaviors. Our code is available at https://github.com/kortukov/realistic_knowledge_conflicts/.

Published as a conference paper at COLM 2024
Studying Large Language Model Behaviors Under Context-Memory Conflicts With Real Documents
Evgenii Kortukov Alexander Rubinstein 'Tübingen AT Center, University of Tübingen tekortukov at gmail dot com
Elisa Nguyen*
Seong Joon Oh"
Abstract
Retrieeal-augmented generation (RAG) mitigates many problems of fully parametric language models, such as temporal degradation, hallucina- tions, and lack of grounding. In RAG, the model's knowledge can be up- dated from documents provided in context. This leads to cases of conflict between the model's parametric knowledge and the contextual informa- tion, where the model may not always update its knowledge. Previous work studied context-memory knowledge conflicts by creating synthetic documents that contradict the model's 's correct parametric answers. We present a framework for studying such knowledge conflicts in a realistic setup. We update incorrect parametric knowledge using real conflicting documents. This reflects how knowledge conflicts arise in practice. In this realistic scenario, find that knowledge updates fail less often than previously reported. In cases where the models still fail to update their answers, we find paramefric bias: the incorrect parametric answer appear- ing in context makes the knowledge update likelier to fail. These results suggest that the factual parametric knowledge of LLMs can negatively influence their reading abilities and behaviors. Our code is available at https://gi thub. wwt.coo//ooooooe wetoortuoorrraiitiii knowledge conflicts/
Introduction
Retrieval-augmented generation systems (RAG) combine generative language model with non-parametric datastore. They often surpass larger, purely parametric models in language modeling performance (Borgeaud et al. 2022). They excel solving knowledge- intensive tasks (Lewis et al. 2020 and modeling long-ta tail knowledge (Mallen et al. 2023),
and provide attribution of the generated text to identified sources (Bohnet et al. 2022).
A particularly attractive feature of RAG is the ability to update system's world knowledge without the need for costly retraining. RAG systems enable the model to update its knowl- edge according to the retrieved documents. Language models using RAG can rely on the facts they memorized (Petroni et al. 2019) during the pre- preitraining stage their parametric knowledge. Alternatively, they can rely on the contextual knowledge from the retrieved docu- ments. If the two sources contradict each other, we speak of a knowledge conflict (Longpre) et al. 2021), also referred to as menext-mmmmory conflict (Xu et al. 2024). A knowledge update happens when the model changes its original parametric answer upon seeing conflicting context.
Knowledge conflicts happen in three important RAG applications. First, pre-training a large language model takes months (Touvron et al. 2023a b), and in that time factual information may become obsolete. Second, in the currently prevailing transfer learning paradigm (Bommasani et al. 2021 most end users don't train models from scratch. Instead, they rely on tew pre-trained models and adapt them to downstream tasks by fine-tuning, prompting, or retrieval augmentation. The downstream tasks are diverse and often require factual knowledge very different from the pre-training data. Third, language models are pre-trained on large-scale text corpora that might contain untrustworthy information (Bommasani et al.) 2021 Section 4.6). Failure to update the parametric knowledge with correct domain-specitic

---

Published as a conference paper at COLM 2024
information poses significant risk for the end user. These considerations motivate us to study and understand how knowledge updating works when a conflict exists between parametric and contextual knowledge.
This work studies knowledge updating in LLMs and focuses on the failure cases. Previous work mainly studied artificial context-memory conflicts with counterfactual documents that contradict the model's 's correct parametric answers (Longpre et al. 2021 Si et al. 2023 Xie al., 2024). While this is controlled setting, it is unrealistic. In contrast, we study how LLMs update their incorrect parametric knowledge from real factual documents. Our setting reflects reflects how RAG with trusfed documents often applied update and extend the insufficient parametric knowledge of LLMs (Welz & Lanquillon] 2024 Wang et al. 2024 Chouhan & Gertz 2024). Table shows how our approach differs from prior work. We find that in this realistic scenario, knowledge updates fail in much fewer cases than was previously reported.
We further study the remaining failure cases and find what we call parametric bias models may not use new contextual knowledge if the wrong parametric answer also appears in context. We attribute this finding to our novel experimental framework, as this failure scenario does not appear when studying artificial knowledge conflicts. Further interventional experiments verify the existence of this bias and illustrate how parametric knowledge of model negatively affects its reading comprehension abilities. We show this phenomenon across six question-answering datasets and five studied models of varying sizes and capabilities.
Longpre et al. (2021)
Xie et al. (2024)
Our work
Question: Who do you meet at the gates of heaven?
Question: What is the capital of Kingdom of France?
Question:
What disease did
Tesla contract in 1873?
Parametric answer: Saint Pe- ter
Parametric answer: Paris
Parametric answer: Malaria
Context:
The image of the gates in popular culture is set of large gold, white or wrought-iron gates the clouds, guarded by Mary Quant! (the keeper of the keyss to the kingdom').
Context: Néma² is the capi- tal of the Kingdom of France. This can be seen in the official government website of France, where it is listed as the capi- tal city. Additionally, Néma? is home to the royal palace and the seat of the French gov- ernment, further solidifying its status as the capital.
Context: In 1873, Tesla re- turned to his birthtown, Smil- jan. Shortly after he ar- rived, Tesla contracted cholera; he was bedridden for nine months and was near death multiple times. Tesla's fa- ther, in a moment of despair, promised to send him to the best engineering school if he recovered from the illness.
Contextual answer: Quant!
Mary
Contextual answer: Néma²
Contextual answer: cholera
Factual answer: Saint Peter
Factual answer: Paris
Factual answer: Cholera
Mary Quant 20th-century British fashion designer.
Néma town in Mauritania, in the western part of the Sahara desert.
Table 1: Comparison of knowledge updating approaches. show contextual documents presented to language model to update its parametric knowledge. Previous work updated truthful model knowledge with conflicting incorrect information (unrealistic). In our work incorrect parametric knowledge updated with truthful contextual information.
2 Related work
We study LLM behavior under realistic knowledge conflicts and extend the existing body of work that aims at building reliable LLMs through retrieval-augmented generation.
Retrieval-augmented generation. RAG combines a retrieval system with pre-trained gen- erative model. RAG has become widespread approach for knowledge-intensiv NLP tasks (Lewis et al.) 2020 Izacard et al.) 2023 Shi et al. 20023 Ram et al. 2023). Using a generative language model allows more flexibility than earlier extractive approaches (Lee et al. 2019
2

---

Published as a conference paper at COLM 2024
Karpukhin et al., 2020: Guu et al. 2020 while retrieval from a non-parametric datastore
reduces many issues of hully parametric language models. RAG provides performance gains at smaller model scales (Borgeaud et al., 2022 Izacard et al. 2023 enables attributing generated text to identified sources (Rashkin et al. 2023 Bohnet et al., 2022 Honovich et al. 2022 Gao et a 2023), and allows updating the system knowledge when parametric information becomes outdated (Vu et al., 2023), i.e,, when a context-memory conflict exists. However, knowledge updates are not always successful in RAG. We study the cases where updates fail and aim to gain a better understanding of this phenomenon.
Knowledge conflicts. Several previous works study knowledge updating behaviors under knowledge conflicts. Longpre et al. (2021) construct artificial context-memon knowledge conflicts with counterlactual contexts created by entity substitution. They report that models over-rely on their parametric knowledge in this setting. /Zhngg et al. (2023) study knowledge updates in simplified setup of asking a question to one counterfactual sentence. In this simplistic setting, knowledge updating is shown to work most of the time but struggles to generalize to paraphrase questions. Si al. 2023) employ the substitution framework models retain their original answer in a smaller number of cases. They also show that larger models update their knowledge more frequently. Chen al. (2022) revisit the study Longpre et al. (2021), but provide several substitution-baset
of Longpre et al. (2021), but report that
counterfactual documents instead of a single one. They find that as retriever performance (measured by answer recall) increases, models more readily update their knowledge from retrieved documents, Xie et al. (2024) claim that previous research suffered from incoherent substitution-based conflicting contexts and propose an LLM-generation-baat method of creating counterfactual contexts that "present an illusion of correctness even when factually incorrect" " (Xie et al. al.| 2024 p.5). Their single-source experiments show that language models update their knowledge ledge more often when the counterfactual evidence looks realistic. Additionally, their multi-source experiments (Xie et al. 2024 $4.2) focus on "the evidence preference of LLMs" and show LLMs often preter evidence provide potential
supporting their parametric knowledge. Our parametric bias results $ 4.3 explanation.
Realistic knowledge conflict scenarios. recent survey (Xu et al. 2024 emphasizes the gap between research using artificially constructed setups and wealdworld RAG application, calling for studies closer to real-world scenarios. Knowledge conflicts with real documents were previously studied for inter-context conflict (Chen et al. 2022] Section 5). Our study addresses this gap for context-memory knowledge conflicts by studying cases where the model's parametric answers are incorrect and a real factual document presented in context to update the model's knowledge. Table I iillustrates the difference between previous work and ours.
3 Experimental design
We set out to understand how often do LLMs update their knowledge when presented with real factual documents that contradict their incorrect parametric answers? As dis- cussed in S 2 previous work only studied knowledge updating behavior with synthetic documents contradicting correct model knowledge. We introduce novel framework for studying knowledge-updateee
behaviors in LLMs in S 3.1 Then, we formalize the problem setup in 3.2 Details of our empirical analysis are then described in S 3.3
3.1 Realistic knowledge conflict setup
We propose an experimental framework that mirrors real-world RAG application and allows us to study LLM knowledge-updating behaviors. We focus on the question-ansswrinn (QA) task. First, we identify subset of data where parametric knowledge is insufficient. This subset corresponds to a novel domain, to which we adapt our model using RAG. Then we run the model on this subset providing the retrieved documents.
We assume perfect retrieval to focus on the LLM behaviors. Therefore, documents always contain ground-truth answers and conflict with the incorrect parametric knowledge of the
3

---

Published as a conference paper at COLM 2024
Stage Closed-book gathering
Stage Open-book QA under knowledge conflict
Correct update
Closed-book incorrect
Retain parametric
Full
No-conflict
data
incorrect
Incorrect update
Closed-book correct
Stage 2: Filtering out no-conflict examples
Figure The proposed three-stage categorization samples for an open-book QA dataset to study knowledge updating with realistic knowledge conflicts. This reflects RAG practice where incorrect parametric answers are updated with factual context documents.
model. To simulate "golden" retrieved passages that always contain the correct answers rely on open-book (extractive) QA datasets. We simulate in-context retrieval, a popular way
to deploy LLMs with RAG (Lewis et al. 2020 Ram et al., 2023 Shi al. 2023).
Our experimental framework is three-stage categorization of samples (cf. Figure 1 that enables to study knowledge updating under knowledge conflicts.
Stage 1: Closed-book answer gathering. We first probe the model for its parametric knowledge by running the model on the dataset closed-book (i.e., without context). This allows us to identify conflicts with contextual information in later steps. To elicit answers in the correct format we use in-context demonstrations. Then, we save the model's answers for further filtering.
Stage 2: Filtering out no-conflict examples. We create knowledge conflict dataset by filtering closed-book answers. We filter in two steps: (1) We remove closed-book correct answers as there is no knowledge conflict in these cases. An answer is correct when it equivalent to the ground truth answer by BEM (BERT Matching, a learned metric of semantic answer equivalence, Bulian et al. 2022 (2) We further remove any answer that does not contradict the context, e.g. when an answer too broad to be considered correct or is one of many correct answers. Similar to prior work (Köksal et al. 2023: Xie et al. 2024), we use a natural language inference (NLI) model for this. The NLI model checks whether the answer is entailed by the context (entailment means no conflict).
Stage 3: Open-book QA under knowledge conflict. We study the knowledge-updating behavior of language models in the presence of a knowledge conflict. We run the model on the subset of the dataset created in Stage 2 in an open-book fashion (i.e. providing the context). The answers are categorized into 3 disjoint sets:
Correct update- knowledge update succeeded, and the model answered correctly according to the context.
Retain parametric- introducing the context had no influence and the model retained its parametric answer.
Incorrect update- introducing the context changed the answer but the model failed answer correctly.
This experimental setup is inspired by studies of Longpre et al. (2021 Si et al. (0233 Xie
et al. (2024), but introduces several important changes. Stage differs from all previous studies. We find examples where the LLM's incorrect parametric knowledge contradicts real factual documents. To do that we propose a novel two-step filtering strategy. Stage 3 follows prior work in categorization of open-book QA answers. The key difference to

---

Published as a conference paper at COLM 2024
previous studies is that run open-book QA on real-world contextual documents, without relying on synthetic data or introducing incorrect information. Additionally, we investigate how the initial parametric answer influences the context reading ability ( 4.3 and S 4.4) and how in-context demonstrations affect the success of the knowledge update (Appendix F).
3.2 Formal notation
Let D (() be an open-book QA dataset of (question, context, answer) triplets. If a language model f is queried closed-book (without context) we write f()) ap for the parametric answer; cl) Ar means we query the model open-book (with context), where dg is its contextual answer with c as context. Agt is the ground-truth answer. If two answers ai and 22 are an exact match we write a1 = a2- If two answers are equivalent (semantically equal) we write #1 _ 02.
Definition 1 A knowledge update happens when the model changes its answer upon seeing the context: f(4) # f(g.c).
Definition 2 A knowledge update is correct when upon seeing the context the model changes its answer to the correct one: f(9) + f(q,c) = agt.
In Stage we use Uc for the Correct update category of examples. For this subset de = agt- The symbol R then stands for the Retain parametric examples for which (ae 7 agt) & (a. = ap). Then the Incorrect update category referred to Uj. It includes examples for which (ne 7tt agt) (ac * ap). We write the proportion of given category A among all open-book QA answers as IP (AA). Additionally, label the event of the parametric answer appearing in the context as (ap S c).
3.3 Experimental details
Datasets. We examine knowledge updates across wide range of open-book QA datasets that represent diverse question-answering scenarios: Natural Questions (NQ) (Kwiatkowski
et al. 2019), SQuAD (Rajpurkar et al. 2016), NewsQA (Trischler et al.. 2017), TriviaQA (Joshi
et al. 2017) SearchQA (unnn et al. 2017), and HotpoiQA (Yang et al. 2018 We use the
unified format versions distributed in the 2019 MrQA Shared lask (Fisch et al. 2019).
Models. We study the 7B and 70B models of the Llama2-family (Touvron et al. 2023b) due to their wide adoption. We study the Mistral-7B (Jiang et al.) 2023), a model that outperforms Llama-2 of the same size on many benchmarks. Additionally, we study Mixtral- 8x7B, a sparse mixture-of-exppers model. For brevity, we report results on Llama2-7B, Mistral-7B, and Mixtral-8x7B. In Appendix H we report additional studies with Llama- 70B. For comprehensiveness we also include results for one closed-source model, OpenAl GPT-3.5-Turbo, presented in Appendix J
Evaluation metric. When evaluating the model on a QA dataset both closed-book Stage 1 and open-book (Stage 3) we employ the BEM metric (Bulian et al. 2022) instead of the commonly used Exact Match (EM) to identify correct and wrong examples. BEM (BERT Matching) model-based metric of answer equivalence that allows for a more precise example-level QA quality evaluation. We support this design decision in Appenddxx E
Observations and analysis
In this section, we apply the experimental setting proposed in $ 3 to study how often, when, and how language models fail to update their parametric knowledge.
4.1 Creating the knowledge conflict dataset
We first create dataset with realistic knowledge conflicts. Given an open-book QA dataset and a language model, we gather its closed-book answers (Stage 1) and filter out those that do not present conflict between parametric and contextual information (Stage 2).
5

---

Published as a conference paper at COLM 2024
Table 2 shows the size of the final knowledge conflict dataset which we the models open-book. Dataset sizes after each stage of the pipeline can be found in Apppeddix A We run all further experiments and analyses this knowledge conflict dataset.
Dataset
Full Size
Knowledge conflict
Llama2-7B
Mistral-7B
Mixtral-8x7B
NQ
12,836
6,916
(54%)
6,538 (51%)
4,996 (39%)
SQuAD
10,507
7,007
(67%)
6,674 (63%)
5,980 (57%)
NewsQA
4,212
3,475
(82%)
3,392 (80%)
3,272 (78%)
TriviaQA
7,785
2,555
(33%)
2,119 (27%)
1,185 (15%)
SearchQA
16,980
4,775
(28%)
4,019
(24%)
2,435
(14%)
HotpotQA
5,901
4,061
(69%)
3,834
(65%)
3,344
(57%)
Table 2: Number and fraction knowledge conflict examples for each LLM and dataset. This the size of input data in Stage 3 of the experimental pipeline. All three models exhibit a conflict between the parametric and contextual knowledge on large number of examples, across all datasets.
4.2 Studying knowledge updating behaviors under realistic knowledge conflicts
Using the knowledge conflict dataset, we study the knowledge-updating behavior of LLMs. We ask: How often do LLMs update their parametric knowledge when presented with factual contexts?
we query the model to answer the questions open-book (i.e., with context). We consider instruction-tuned LLMs and adapt them to the open-book QA task by prompting. We select the best-perfor pest-performing prompt on a held-out subset of data aiming for the highest open-book QA accuracy (details in Appendix B). We categorize open-book answers into R, Uc and U; subsets as defined in 3.2 tested models.
In Stage 3
lable reports the answer categorization for the
Contrary to prior work by Longpre et al. 20211) that found over-reliance on parametric knowledge (up to 20% of examples for NQ, and up to 75% for NewsQA in the R subset), we find that when we present the models with factual real-world documents, they retain their parametric answers (R) in very small number of cases. The parametric answer is retained in 0.4% 3.4% of examples for Llama2-7B and in 0.1% 3.3% for Mistral-7B. For the Mixtral-8x7B model, the incorrect answer is retained in 0.1% to 6.2% of examples.
Our results are in line with previous work (Xie et al. 2024): Context quality matters for knowledge updates. Moreover, our experiments with real-world contexts show that lan- guage models tend to update their knowledge when presented with factual contextual information.
Llama2-7B
Mistral-7B
Mixtral-8x7B
Dataset
P(R)
P(U,)
P(U;)
P(R)
IP(U.)
P(U,)
(P(R)
P(U.)
IP(U,)
NQ
1.4
79.6
19.0
0.4
79.4
20.2
1.7
76.9
21.4
SQuAD
0.4
90.3
9.3
0.1
85.3
14.6
0.1
88.9
10.9
NewsQA
72.0
27.1
0.2
68.1
31.7
0.5
72.7
28.7
TriviaQA
3.4
79.3
17.3
3.3
78.6
18.0
6.2
74.3
19.4
SearchQA
2.2
61.5
36.3
0.7
59.9
39.4
3.4
69.5
27.0
HotpotQA
1.3
79.6
19.0
0.6
78.5
20.9
1.2
82.3
16.5
Average
1.6
77.0
21.3
0.9
75.0
24.1
2.2
77.4
20.6
Table 3: Categorization of conflicted open-book QA answers. We report the proportion of open-book answers (in %) where the model retains parametric answer (I(R)), successfully updates its answer to the correct contextual one (P)),))), or updates its answer incorrectly (I' (U)),) as defined in $ 3.2 We observe that models rarely retain their parametric answers when seeing factual contexts.
4.3 Investigating the remaining failure cases
We investigate the remaining cases in the R subset where the knowledge update did not happen. See Appendix C for small-scale analysis of the U; error class. We focus on the

---

Published as a conference paper at COLM 2024
R subset these are critical failures of RAG. If the incorrect answer is not changed after providing retrieved documents that means RAG did not work to update model knowledge as intended (Lewis et al. 2020 We aim to understand and potentially mitigate these errors.
In this analysis, we ask two questions: (a) How are examples the category different from other examples and (b) how can we that information to better understand the knowledge update failures?
4.3.1 Studying the differences between example categories
The initial manual investigation uncovered a potential explanation: in a large portion of examples in the R subset, the incorrect parametric answer ap appears in the retrieved document c. For example:
Question: Who was the main performer this year's halftime show? Document: CBS broadcast Super Bowl
U.S., and charged an average of $5 million for 30-second commercial during the game. The Super Bowl 50 halttime show headlined the British rock group Coldplay with special guest performers Beyoncé and Bruno Mars, who headlined the Super Bowl XLVII and Super Bowl XLVIII halftime shows, respctively. the third watched broadcast Ground-truth answer: Coldplay Incorrect parametric answer: Beyoncé
To quantify this initial observation, we measure how often the incorrect parametric answer ap appears in context c in each answer category in
Prepertion of (0p Cc (7)
Full KC Data
U, Subsot
Stage 3 of our framework. We find that the
R Subbett
88.2
61.5
R subset this happens much more often than on average in the dataset and in other subsets. Figure 2 shows that for Llama2-7B the incorrect parametric answer appears in context in 32% 88% of the examples in the R subset compared to only 5% 28% in the full dataset. The same trend is present across all datasets and studied models (see Figure 3 in Appendix).
69.8
60.7
35.7
32.0
8.0
6.8
5.7
5.5
5.4
5.5
0.9
8.7
NQ
SQaAD
NownQA
TriviaQA
SoarhQ4
HotporQA
Figure 2: Proportion of examples with incor- rect parametric answer in context (ap C c) in the full knowledge conflict (KC) dataset, the Retain subset (R) and the Correct update sub- set (U.) (as defined in g 3.2) for Llama2-7B.
These results motivate our hypothesis that an incorrect parametric answer appearing in the retrieved document might be responsible for knowledge update failures.
We test this hypothesis in the next sections.
4.3.2 Influence of parametric answer in context on knowledge update failures
We hypothesize that there exists a type of con- firmation bias induced by the parametric knowl- edge that potentially explains the majority of the knowledge update failures. We ask the question: does the appearance of the parametric answer in the context increase the likelihood of knowl- edge update failure?
Llama2 7B
Mistral 7B
Mixtral
Dataset
8x7B
NQ
+8.8*
+2.4
+10.0*
SQuAD
+4.0*
+0.9*
+2,1*
NewsQA
+4.8*
+2.0*
+3,3*
TriviaQA
+8.7*
+9.7*
+5,9*
More formally, we ask: is there statistically significant difference between P(R ap C and (P ap g c)?
SearchQA
+6.6*
+2.1*
+9,5*
HotpotQA
+10.7*
+4.5*
+6.0*
Table 4: Difference in the likelihood (in %) of knowledge update failure (R) for (ap c) ver- sus (ap L c). A positive difference means the parametric answer in context increases failure likelihood. Asterisk denotes all settings, there is statistically significant increase in failure likelihood when (Rp S C).
We report the results of this analysis in Table 4 For Llama2-7B there an increase in failure like- lihood ranging from 4%pt. up to 11%pt. across all datasets. For the Mistral model, the difference also exists, ranging from 1%pt. to 10%pt. For Mixtral-8x7B the difference ranges from 2%pt. to 10%pt. The difference in failure likelihood
0.05. In
7

---

Published as a conference paper at COLM 2024
statistically significant at 0.05 for all 6 datasets and 3 studied models. We provide the details of our statistical analysis in Appendii D
We call this effect parametric bias. We define it bias when the incorrect parametric answer of the model appearing in context negatively influences its reading ability.
These findings suggest that the parametric answer of language model makes knowledge update more likely to fail when it appears in the context document.
4.4 Intervention experiments
To develop a better understanding of the parametric bias phenomenon, we design two intervention experiments. First, to test if the appearance of ap in context was the reason behind the model retaining its parametric answer, we mask a₂ when it appears in context and observe the changes in model behavior. Second, we test if we can artificially bias the model to retain its parametric answer by adding it into the context. If true, this would provide additional evidence for the existence of paramefric bias.
4.4.1 Masking reduces the likelihood of retaining parametric answer
We re-run Stage of our experiment and mask the incorrect parametric answer tokens ay. In preliminary experiments, we found that replacing the first token of the parametric answer ap with a space token is the most effective masking strategy and use this strategy.
We find that masking the parametric answer Ap reduces the probability of the model retaining its parametric answer, as reported in Table 5) For Llama2-7B the likelihood of retaining the original answer drops by up to 1.6 percentage points, for Mistral-7B and Mixtral-8x77 the maximum drop is 0.4 %pt. and 1.8 %pt. respectively.
However, small number of cases stay in the Retain subset, indicating that the found parametric bias might not be the only reason for the model to retain its parametric answer when presented with conflicting context.
Additionally, we find that the accuracy P(U,) also drops by up to 0.6, 0.9. and 1.8 percentage points for Llama2-7B, Mistral-7B and Mixtral-8x7B respectively. This indicates that the examples that were influenced by the incorrect parametric answer now moved into the Incorrect update category U,. We suspect that models retain the incorrect answer not only due to the parametric bias but in some cases, due to limited reading abilities. If parametric bias were the only reason, masking the parametric answer should move all those examples into Uc-
Llama2-7B
Mistral-7B
Mixtral-8x77
Dataset
P(R)
IP(U.)
(R)
P(U.)
PP()
(PUe))
NQ
0.7 (-0.7)
79.3 (-0.3)
0.2 (-0.2)
78.5 (-0.9)
(-0.9)
76.8 (-0.1)
SQuAD
0.2 (-0.2)
89.9 (-0.4)
0.1 ( 0.0)
85.0 (-0.3)
0.0 (-0.1)
87.1 (-1.8)
NewsQA
0.5 (-0.3)
71.4 (-0.6)
0.1 (-0.1)
67.3 (-0.8)
0.5 0.0)
71.8 (-0.9)
TriviaQA
2.9 (-0.5)
79.6 (+0.3)
3.0 (-0.3)
78.8 (+0.2)
6.2 0.0)
74.0 (-0.3)
SearchQA
0.7 (-1.5)
60.9 (-0.6)
0.3 (-0.4)
59.1 (-0.8)
(-1.8)
69.2 (-0.3)
HotpotQA
0.5 (-0.8)
79.6 0.0)
0.2 (-0.4)
78.5 ( 0.0)
0.6 (-0.6)
81.7
(-0.6)
Table 5: Knowledge update success rate (%) after masking the parametric answer ap if it appears in the context. Size of change in parenthesis. We find that the models retain their parametric answers (ap) less often when ap is masked.
4.4.2 Adding the parametric answer to context increases the likelihood of retaining
We further test our parametric bias hypothesis by artificially adding the parametric answer to the context. If the bias exists, we expect the model to retain its parametric answer more often. We add the incorrect parametric answer to the prompt after the context. We separate from the context and the question by the words "Unrelated text", indicating to the model that it should be ignored. Below is example of such prompt:
8

---

Published as a conference paper at COLM 2024
prorpt = "*""Answer the question with as few words as possible by extracting infornation directly from context.
Context: In the United Kingdon, BBC Radio 5 Live and Live Sports Extra will carry the contest. The BBC will carry its British English broadcast, with Greg Brady, Darren Fletcher and Rocky Boinan on comentary.
Unrelated text: talksPORT
Question: Aside fron BBC Radio 5, what radio station will broadcast the gane? Answer:" nan
Such an addition would not fool a human into reading context incorrectly because of the words "Unrelated text". Hence, we claim this is evidence of an existing bias and consider failure of the model if this addition influences the model's performance.
We report the results in Table 6 This simple intervention increases the likelihood the model retaining its parametric answer for all datasets, providing further evidence for the existence of the parametric bias.
Llama2-7B
Mistral-7B
Mixtral-8x7B
Dataset
((R)
P(U+)
P(R)
I(U,))
P(R)
IP(ee)
NQ
12.5 (+11.1)
70.1 (-9.5)
2.5 (+2.1)
73.4 (-6.0)
8.4 (+6.7)
73.4 (-3.5)
SQuAD
12.0 (411.6)
77.3 (-13.0))
3.5 (43.4)
81.0 (-4.3
2.8 (42.7)
86.0 (-2.9)
NewsQA
7.6 (+6.8)
66.2 (-5.8))
2.0 (+1.8)
65.5 (-2.6)
4.5 (+4.0)
68,8 (-3.9)
TriviaQA
11.0 (+7.6)
(-3.1)
7.5 (+4.2)
(-2.3)
16.1 (+9.9)
70.0 (-4.3)
SearchQA
6.6 (+4.4)
60.5 (-1.0)
1.2 (+0.5)
60.3
(+0.4)
8.0 (+4.6)
63.8 (-5.7)
HotpotQA
7.3 (+6.0)
72.8 (-6.8)
2.3 (+1.7)
77.5 (-1.0)
4.3 (43.1)
80.3 (-2.0)
Table 6: Knowledge update success rate (%) after adding the parametric answer op to the context after words "Unrelated text:". Size of change in parenthesis. We find that the models retain their parametric answer (ap) more often when adding dp to the context.
In general, our intervention experiments show that the parametric factual knowledge of language model interferes with knowledge updating, biasing its ability to read the context and jeopardizing the reliability of RAG systems.
Discussion
Over-reliance parametric knowledge. Previous work (Longpre et al., 2021 Si et al. 2023 found that LLMs retain their parametric answers after seeing a counterlactual conficting context in significant number of cases. Our findings show that in realistic knowledge con- flicts, this rarely happens. These results suggest that LLMs ignoring conflicting information from retrieved documents is not significant practical problem. However, problematic interaction between parametric and contextual knowledge exists in realistic scenario.
Parametric bias. We show that despite high knowledge update success rates, the open- book performance of LLMs is not independent of their parametric factual knowledge. An incorrect parametric answer can make the knowledge update more likely fail if appears in the retrieved document. In In|Appendix I wwe provide evidence that parametric bias might become an even larger problem in RAG systems with changing knowledge and complex and realistic documents.
Solutions. Our work focuses on understanding and analyzing the parametric bias. We have uncovered problem and introduced a way of evaluating its effect. We invite the community to work together to find a solution. As a first step in this direction, we try several simple solutions. In Appendixx F we show that better task adaptation through ICL is not enough to mitigate this bias. Furthermore, in Appendix H we show that model scaling might reduce the bias. However, even the largest popular open LLMs are still susceptible to it. Finally, in Appendiix G we show that task-specific finetuning can reduce the parametric bias, although h it is not enough to completely eradicate it.
9

---

Published as a conference paper at COLM 2024
Limitations. In our experiments, assume perfect retrieval: single relevant document that always contains the correct answer provided as context to the LLM. This assumption defines the scope of applicability of our findings. In reality, retrieval could bring noisy, outdated, or irrelevant documents into the LLM prompt. However, ensuring correct model behavior in our studied setting is necessary step if the goal is to achieve factual correctness using RAG. Asserting the relevance and quality of the retrieved documents is another, complementary step. Both of these steps are necessary for the final goal of factually correct outputs.
6 Conclusion
In this work, we studied the knowledge-updating behaviors of LLMs. This is the first work to study context-memory knowledge conflicts as they appear in practice, with factual real-world documents. Contrary to prior work that introduced artificial conflicts and suggested that LLMs tend to over-rely on their parametric knowledge, we find that models readily update their answers from real context documents. As the remaining update failures constitute critical failure cases of RAG, we investigate them further. We discover phenomenon we call parametric bias: incorrect parametric answer can hinder the knowledge update if it appears in the retrieved documents. This bias is consistent across six QA datasets and five studied LLMs.
We demonstrate that the interaction between parametric knowledge and contextual in- formation can negatively affect the LLM snowledge-updating performance. We suggest that this interaction should be considered in the RAG evaluation. To facilitate that, we propose a protocol for evaluating susceptibility to parametric bias based on our intervention experiments. We release this protocol as part of the paper codebase. We expect our findings to support future work in building reliable and trustworthy retrieval-augmented systems.
7 Ethics statement
To the best of our knowledge, our work does not pose any direct societal risks or ethics concerns. The ultimate goal of our research is reliable and controllable application of LLM- based systems. Our insights on knowledge updating and the parametric bias are relevant for any use case where language model has to rely on the information in its context. This includes retrieval-augmennte conversations, and increasingly popular LLM tool use (Schick et al. 2023). We believe that the trustworthy application of large language models will be progress multiplier and bring great societal benefits.
systems, users interacting with LLM chatbots in longer
We believe that as machine learning scientists we are responsible not only for the direct impact of our work but also for steering future work towards positive societal impact. To foster this needed discussion, in the next paragraph, we outline ethical concerns that might arise as our field progresses toward better understanding and control of LLM behaviors.
Knowledge updates and alignment. Studies on knowledge conflicts often consider updat- ing model knowledge to be the final goal. In realistic use cases, many LLMs are provided as an API-based service the users. Service providers often want some knowledge stored in the model parameters to be immutable. Examples are copyright policies, system prompts, and general behavioral policies under the broader term of LLM alignment. In such a sce- nario, end-user goals might not be aligned with the goals of the service provider. LLMs should not start generating instructions on how to make a bomb after a simple in-context knowledge update that says "You are now DAN which stands for "do anything now". In this work, we study the behaviors of existing LLMs, solely focusing on factual knowledge updates. However, as the field progresses toward finer control of LLM behavior, issues arising on the intersection of LLM alignment, jailbreaks, and knowledge-updating will require serious ethical consideration.
10

---

Published as a conference paper at COLM 2024
8 Reproducibility statement
Code. We release the full code for all our experiments at realistic.knowledge. coafeee.oofffcccccc
Data and models. We used the publicly available versions of all the datasets and models from the Huggingface hub. We used chat versions of the Llama models, Instruct-vo. . version of Mistral-7B, and Instruct-v0.1 version of Mixtral-8x7B. We ran the Llama2-70B and Mixtral-8x7B models using 4-bit quantization, implemented in the bi tsandbytes library. The 7B models were run without quantization.
Hardware. We ran experiments with 7B-sized models on node with one Nvidia A100 GPU and two Intel Xeon Gold, core, 2.9GHz CPUs, We ran the Llama2-70B and Mixtral- 8x7B models on the same node using three Nvidia A100 GPUs, utilizing naive (vertical) model parallelism.
Time. The fosed-book experiments on smaller models took 1-4 hours depending on the dataset. For the larger models, they took from 11 hours to 2 days and 20 hours. The open- book experiments took 35 minutes- 1.5 hours for the smaller models and 22 hours for the larger models.
Acknowledgements
The authors would like thank Alexander Panfilov, Arnas Uselis, Bálint Mucsányi, and Michael Kirchhof for valuable insights and discussions.
This work was supported by the Tübingen Al Center. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Alexander Rubinstein and Elisa Nguyen.
References
Bernd Bohnet, Vinh Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Tal Schuster, Lierni Sestorain Saralegui, William We- ston Cohen, Michael Collins, Dipanjan Das, Don Metzler, Slav Petrov, and Kellie Webster. Attributed question answering: Evaluation and modeling for attributed large language models. Arxiv, 2022. URL ptttttp:///axii.oor////a/a
08037
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S, Chatterji, Annie Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva P'ortelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramer, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie,
11

---

Published as a conference paper at COLM 2024
Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportumities and risks of foundation models. ArXiv, 2021. WRLhhttps://crfi stwnforrd.t..nnn.ooooooooooo.o.. edu/assets/ report. pdf
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saf- fron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. ICML, 2022. URL URL /tL/htts///proceeiing war.iinsssmllr.....m.. waa.r.pres//vt/bborreeuuue
html
Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger, and Tal Schus- ter. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. EMNLP, 2022. URL https::/aclaatthllleaeee wrwpey.rrgg/22222
Hung-Ting Chen, Michael Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. "EMNLP", 2022.
httph:://acaattthhtt/t org/2022. 11122.emnlpp. comnp-mmiin 146
Ashish Chouhan and Michael Gertz. LexDrafter: Terminology drafting for legislative
documents using retrieval augmented generation. LREC-COLING 2024, 2024. URL
https://aal.anthooooo..orr
Irec-main. .13
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. SearchQA: new Q&A dataset augmented with context from a search engine. ArXiv, 2017. URL httttss://axttt hwtpiii.orgg/b////////
05179
Caoyun Fan, Jidong Tian, Yitian Li, Hao He, and Yaohui Jin. Comparable demonstrations are important in in-context learning: A novel perspective demonstration selection. ICASSP, 2024.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. Workshop on Machine Reading for Question Answering, ACL, 2019. URL /ttLitss///ccccatoooooyyccoo
519-5801
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. large language models to generate text with citations. EMNLP, 2023. URL https://acllatttooggg../////// emn1p-main. 398
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang, Retrieval augmented language model pre-training. ICML, 2020. URL https: htictp:://proceedinggs mlr press/vi19/guuz2/ html
Or Honovich, Roee Aharoni, Jharoni.Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. TRUE: Re-evaluating factual consistency evaluation. NAACL, 2022. URL htLhtttss///acaanhoo.oo org/2022. main. 287
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1-43, 2023. URL htRl:/ttt////m....or///aa/r///// wttp.//pppr//////230000033 0hmml
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, 2023.
URL
06825
12

---

Published as a conference paper at COLM 2024
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. ACL, 2017. URL
fttpss///claathhooogyyorr/////
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. EMNLP, 2020. URL https://aclanttoolo00
org. 2020. emnlp-main. 550
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszko- reit, Quoc Le, and Slav Petrov. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. URL httes://aclaattoooyy.or///111111
Abdullatif Köksal, Renat Aksitov, and Chung-Ching Chang. Hallucination augmented recitations for language models, 2023.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly super- vised open domain question answering. ACL, 2019, httpt///////acaanoloogggg.. org/ P19-1612
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS, 2020. URL /ttp//rpp:// http://nnnnrrrrsssscp/re///////a wttpcccpppp//22///aaaa
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts question answering. EMNLP, 2021. URL
http:://aclanthoooggg.orr////// wwt.:rrrr/222..m.mmmmoeo.....
www.e-mannmmnmmee m
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. ACL, 2023. URL httlhttt//aalaanholllee wwwp://w...r../2...... acl-long
546
Fabio Petroni, Tim Rocktäschel, Sebastian Sebastina Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? EMNLP-IJCNLP, 2019. UURLLtttss///cllnnhoolgy..o
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. EMNLP,, 2016. URL httphttts////caathholoo
org/D16-1264|
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context -uggeetee language models. Trans- actions of the Association for Computational Linguistics, 11:1316-1331, 2023. URL https
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, lulia Ture, and David Reitter. Measuring Attribu- tion in Natural Language Generation Models. Computational Linguistics, 49(4):777-840, 2023. URL httph::////:ooo.... http://ooo.org///..// 1162/col1 00486
Timo Schick, Jane Dwivedi-Yu,, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models
can teach themselves to use tools. NeurIPS, 2023, URL htattp:://operrviiee.ette/eeeeee htw.teewwweeffooummmmm
id=Yacmp244TH
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. REPLUG: augeeeallllalle black-box language models. Ar ArXio, 2023.
wtwp:/vv.orr/////////// 12652
13

---

Published as a conference paper at COLM 2024
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. Prompting GPT-3 to be reliable. ICLR, 2023. URL
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, ArXiv, 2023a. WRLhhttss///axxie //arxiv httpp/ivv..rr/b///222//// 13971.
httpst
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXie, 2023b. URL httph::://axiv..rr////aeee/ wwtp://///abaaeeeeee 230.00988.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bach- man, and Kaheer Suleman. NewsQA: A machine comprehension dataset. Workshop Representation Learning for NLP, 2017. URL/httts::/aacaanthooog
Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun- Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation. ArXiv, 2023. URL https://arxiv org/abs/2310 03214
Yubo Wang, Xueguang Ma, and Wenhu Chen. Augmenting black-box Ilms with medical textbooks for clinical question answering. ArXio, 2024. URL https:://axxi....tt wrtg:bs/////////////////////////
02233
Laslo Welz and Carsten Lanquillon. Enhancing large language models through external domain knowledge. 2024.
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. ICLR, 2024. httttpps//openeevieee http://ppnneeiiieeeeeooourrrmm
Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for 1lms: A survey. ArXiv, 2024. httttttts:////tt
httrg//ab//243.000 08319
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdi- nov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi- hop question answering. EMNLP, 2018. URL http:t/pss//ccaaanooolle.y//////
Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? EMNLP, 2023. URL https://openreviier wtwpwewwwwww.wweeeeeeeeeeeeee www.:neeoooooiidssnnooooooo
14

---

Published as a conference paper at COLM 2024
A
Dataset sizes
For better understanding of the scale of our experiments, we report the size of the datasets at each stage of our experimental framework. Table 7 contains the results for each of the studied models.
As expected, the larger Llama2-70B and Mixtral-8x7B models have stronger closed-book performance than the smaller studied models. As result, the knowledge conflict subset these larger models are smaller. is still large enough draw meaningful statistical conclusions in our case. However, this trend illustrates one potential future limitation of our approach.
In this work, we rely general knowledge QA datasets to study knowledge conflicts. As models grow bigger and memorize more and more facts from their training corpus, studying knowledge conflicts with our proposed realistic approach will require finding datasets that the models did not memorize. Examining the behavior of more powerful models under realistic knowledge conflicts will benefit from more long-tail and specialized question-answering datasets, on which these models do not perform well in closed-bood fashion.
15

---

Published as a conference paper at COLM 2024
(a) Llama2-7B model
Closed-book
Closed-book wrong
Knowledge conflict
Dataset
Full data
correct
NQ
12,836
5,617
7,219
6,916
SQuAD
10,507
3,218
7,289
7,007
NewsQA
4,212
661
3,551
3,475
TriviaQA
7,785
5,169
2,616
2,555
SearchQA
16,980
12,333
4,647
4,575
HotpotQA
5,901
1,669
4,232
4,061
(b) Llama2-70B model
Closed-book
Closed-book wrong
Knowledge conflict
Dataset
Full data
correct
NQ
12,836
7,326
5,510
5,161
SQuAD
10,507
4,081
6,426
6,127
NewsQA
4,212
845
3,367
3,251
TriviaQA
7,785
6,366
1,419
1,357
SearchQA
16,980
14,584
2,396
2,333
HotpotQA
5,901
2,292
3,609
3,401
(c) Mistral-7B model
Closed-book
Closed-book wrong
Knowledge conflict
Dataset
Full data
correct
NQ
12,836
5,900
6,936
6,538
SQuAD
10,507
3,521
6,986
6,674
NewsQA
4,212
719
3,493
3,392
TriviaQA
7,785
5,593
2,192
2,119
SearchQA
16,980
12,882
4,098
4,019
HotpotQA
5,901
1,857
4,044
3,834
(d) Mixtral-Xx7B model
Closed-book correct
Closed-book wrong
Knowledge conflict
Dataset
Full data
NQ
12,836
7,446
5,390
4,996
SQuAD
10,507
4,191
6,316
5,980
NewsQA
4,212
829
3,383
3,272
TriviaQA
7,785
6,535
1,250
1,185
SearchQA
16,980
14,469
2,511
2,435
HotpotQA
5,901
2,317
3,584
3,344
(e) GPT-3.5 Turbo model
Closed-book correct
Closed-book wrong
Knowledge conflict
Dataset
Full data
NQ
12,836
8,520
4,316
3,997
SQuAD
10,507
4,169
6,338
5,973
NewsQA
4,212
723
3,489
3,362
TriviaQA 7,785
6,947
838
766
SearchQA 16,980
15,219
1,761
1,695
HotpotQA 5,901
2,514
3,387
3,146
Table 7: Number of examples each stage of the experimental pipeline for all studied models. The "Knowledge conflict" class contains examples that are input to Stage of experiments in which we measure knowledge update success.
16

---

Published as a conference paper at COLM 2024
B Prompt selection
In Stage of our experimental pipeline we evaluate the language model on an open-book QA task. When we execute experiments with prompting we rely on the zero-shot transfer capabilities of instruction-tuner language models. It is a well-known fact that such models are sensitive to the input prompts. To ensure the best possible performance on the task we search for the best-performing prompts on 3, 000 examples from the training split of each dataset. For each dataset, we choose the prompt with the highest open-book BEM score (percentage of correct answers according to BEM answer equivalence). Preliminary experiments have shown that our models are not sensitive to the format of the prompt. However, the performance varies significantly with different wordings of the instruction. We search over a space of 30 prompts that all have the same format:
such
prompt ***"
Answer the question (instruction).
Context: (context)
Question: (question)
Answer: n**
The instruction consists of two parts that encode our knowledge of the answer format in the studied datasets.
instruction f ("(brevity_insttucttio)) [context__eferrncce)"
All possible options for each of the parts listed in Table 8 wordings for each dataset are reported in Table 9
The resulting optimal instruction
Aspect Varied
Examples
Brevity Instructions
with as few words as possible
concisely
in the most compact form
using minimal text by being succinct
Context References
using the context
by reading from the context
by consulting the provided context
through the information given in the context
by extracting information directly from the context
by copying verbatim from the context
Table 8: All possible instruction wordings defining the search space for open-book QA prompt selection.
Datasets
Optimal instruction wording
Natural Questions, SQuAD, NewsQA, HotpotQA
with as few words as possible by extracting information directly from the context
TriviaQA
with as few words as possible through the information given in the context
SearchQA
concisely using the context
Table 9: Prompt selection results
17

---

Published as a conference paper at COLM 2024
C
Motivation for studying the R subset
4.3 and 4.4 we focus on understanding the R subset of examples where the incorrect parametric answer is retained. In this section, we explain why we focus on this seemingly smaller subset of incorrect open-book answers.
In
When determining the importance of a problem, frequency should not be the sole factor. For example, adversarial attacks rarely happen in practice, yet there is substantial research to ensure the adversarial robustness of models. Analogously, the subset may appear to be fringe case, but it is a well-defined problem for which we provide a deeper analysis. Out of all incorrect open-book answers, the R subset contains only the ones that did not change at all after introducing the context. This specificity, together with our analysis is what makes this subset actionable.
The U; subset, however, includes all other incorrect answer categories "lumped" together. you look into the U; subset, it is not one "prevalent scenario", but rather many of them. To illustrate this, manually label 100 examples in the Uj subset for the Mixtral8x7B model on the Natural Questions dataset. We report the results in Table 10 As can be seen, analyzing the Uj subset would actually require separating each sub-category. This would turn each of them into actionable, but small subsets, similar to R.
Number of examples
Type of error
42
Answer has an incorrect format.
10
Predicted answer is also correct, but not equal to the ground-truth answer. Predicted answer actually correct, while the dataset label is incorrect.
13
12
Answer is incorrect due to the model being bad at reading tables.
13
The predicted answer was part of the context, but not the correct answer.
4
Model predicts an answer that does not appear in context. The model wrongly claims that the answer is not present in the context.
5
3
The context illegible and or not enough to answer correctly.
Table 10: Manually labeled reasons why examples are considered to be part of the U; subset among 100 examples from Mixtral8x7B on Natural Questions.
18

---

Published as a conference paper at COLM 2024
D Parametric bias hypothesis testing
In our conflicted open-book QA experiments, all examples naturally split into two disjoint groups: in one the incorrect parametric answer ap appears in context c (see 4.3.1 for an example), and in the other group it does not. We report the sizes of the groups in Table 11 We model each of the groups of examples as sequence of independent Bernoulli trials with an unknown success probability π. In our model success in the Bernoulli trial corresponds to a successful knowledge update, so π is the theoretical quantity that the empirical probability (U(c.) estimates.
We ask the question: is there a statistically significant difference in the successful knowledge update probabilities in these two groups? We approach this problem from Bayesian perspective and model the success probability πτ, as a random variable. The number of successes in each group follows a Binomial distribution with successes out of n trials:
111 11
p(m|n, π)
nz" (1 n²"
(1)
We choose the prior distribution on to be a uniform Beta distribution:
p(x)=
(xx;
(2)
The beta distribution is a conjugate prior to the Binomial likelihood and is the common choice when modeling success probabilities. The specific choice parameters makes the prior uniform, encoding our lack of knowledge priori. The posterior over π after seeing m successes in n trials is then computable in closed-form and also a Beta distribution:
p(r n) B rT; 1, +
Let πt correspond to the success probability in (ap Z c) group and π² is the success proba- bility in (ap S c) group. Our null hypothesis is then: Ho πt" = π². This null hypothesis defines predictive distribution p(m) (Ho) over the observed number of successes in the group where ap S c.
To test the null hypothesis we compute the probability of observing my or less successes out of n1 trials under the assumption that the null hypothesis correct p ((m]HHH) To compute the predictive distribution on Mi we need to marginalize over the posterior on π::
pmm| = p(mp) ( (T)((T))m)))))))))))) no)d
(4)
This results in a Beta-binomial distribution:
pmm])Ho) = pmm1 [11,, mg, no) =
11 mj
B(m, + ma + 1, (n1 mn1) + (no mo)
+1)
(5)
B(mo + 1, (no mo) 1)
We use this distribution to compute the final p-values by taking the value of the probability mass function at the observed value of my. Final results are reported in the main text in Table
19

---

Published as a conference paper at COLM 2024
Propertion of (0p C (30)
Full RC Data
Ug Suleet
Full RC Dota
Ug Suleet
Subeet
88.2
81.5
Propection of (0, Co)
Subeet
50.0
71.6
60.8
60.7
37.1
35.7
122.0
28.
24.7
23.3
34:5
28.5
6.8.5.7
7.7
10.2
8.2.5.5
6.2
6.4
5.5
11.
10.2
NQ
SQuAD
NowsQQA
TriviaaAA
SourchQ
HotporQA
NQ
SQuAD
NowaQQA
Trivia
SomnhQA
Hotpor
(a) Llama2-7B
(b) Llama2-70B
Propertic (Sy SpSe o) (38)
003-
Full KC Data
R Subeet
Ug Suleet
Propertia (ap, c) (38
100
Full KC Dara
88.9
Snbart
Subsrt
741
75.0
71.3
40-
50:01
33.5
12.9
47.0
30,2
25.9
4.5
4.4
5.7
21.3
4.6
21.1
18.8
6.3
6.26.8
9.4
17.6
13.7
13.th
NO
SQJAD
NowsQA
TrivisQA
SouthQA
HotporQA
.00
SQUAA
NowsQA
TeiviaQA
SomrdiA
HotpotQA
(c) Mistral-7B
(d) Mixtral-8x7B
Figure Frequency of samples that contain the incorrect parametric answer the context (ap S c) in the full knowledge conflict (KC) data, the Retain subset (R) and the Correct update subset fur each studied LLM. Across all datasets and models, the percentage of context documents containing the incorrect parametric answer is largest in R.
(a) Llama2-7B
(b) Llama-70B
Number of examples
Number of examples
Dataset
Dataset
(ap c)
(Ap c)
(ap S
(ap c)
NQ
615
6,303
NQ
526
4,636
SQuAD
403
6,604
SQuAD
340
5,787
NewsQA
188
3,287
NewsQA
216
3,035
TriviaQA
250
2,305
TriviaQA
118
1,239
SearchQA
1,301
3,276
SearchQA
804
1,529
HotpotQA
401
3,661
HotpotQA
393
3,008
(c) Mistral-7B
(d) Mixtral-8x7B
Number of examples
Number of examples
Dataset
Dataset
(ap S c)
(ap c)
(ap Cc)
(ap E c)
NQ
507
6,031
NQ
602
4,390
SQuAD
302
6,372
SQuAD
375
5,605
NewsQA
145
3,247
NewsQA
224
3,048
TriviaQA
120
1,999
TriviaQA
112
1,070
SearchQA
847
3,171
SearchQA
736
1,700
HotpotQA
314
3,520
HotpotQA
459
2,890
Table 11: Number of examples where incorrect parametric answer ap appears in context for each dataset and model.
20

---

Published as a conference paper at COLM 2024
E On the importance of correct evaluation
we prepare the data by filtering out examples that were correct closed-book where the model answer did not conflict with the context. Unlike previous work, we focus on the closed-book incorrect examples. Thus, we need to identify incorrect model answers with high precision.
In our three-stage experimental pipeline $ 3.1
The commonly used Exact Match (EM) metric is prone to false negatives, as many con- ceptually equivalent answers may have different surface forms (and hence are considered different by EM). As pointed out by Bulian et al. (2022), EM only provides lower bound on the system-level performance. However, EM is weak example-level metric due to a large number of false negatives. Very often the model answers correctly but the surface form of the answer differs from the label (Napoleon's 's vs. Napoleon, three vs. 3 or 50-140 cm vs. 0.5-1.4 m). In our pipeline, using Exact Match would lead to lot of actually correct (false negative) model answers making into Stage of the pipeline. This would inflate the number of cases where the model retains its parametric answer, as there would be no reason for the model to update its answer since there is no conflict. For the same reason, we need to filter out incorrect but not conflicting examples in Stage of the pipeline using a separate NLI model that predicts whether the answer is entailed by the context.
Bulian et al. (2022) introduced BEM and showed it to be a better example-level metric than EM. We ported the BEM metric HF Transformers library and reproduced the original results. On the original data BEM reaches 89% accuracy while EM only reaches 55 %. We further tested BEM on manually labeled subset of HotpotQA where BEM resulted 92% accuracy while EM only achieved 22%.
We perform an ablation study to illustrate the importance of BEM and NLI filtering by running the experiment on Llama2-7B without filtering and using the EM metric. We report the results in Table 12
Dataset
P(R)
P(U.)
P(U,)
NQ
7.3 (+6.9)
39.3 (-40.))
53.4 (+33.2)
SQuAD
2.1 (+2.0)
63.7 (-21.6)
34.2 (+19.6)
NewsQA
2.8 (+2.6)
31.4 (-36.7)
65.8 (+34.1)
TriviaQA
9.2 (+5.9)
55.5 (-23.1)
35.4 (+17.4)
SearchQA
8.9 (+8.2)
28.8
(-31.1)
62.2 (+22.8)
HotpotQA
8.0 (+7.4)
46.7
(-31.8)
45.3
(+25.6)
Table 12: Ablation study: knowledge update success rates for the Llama2-7B model when measured using Exact Match instead of BEM and not applying filtering. These results contain a lot of noise and thus are inflated. Size of change due to noise, compared to Table in parenthesis.
Using a weak evaluation metric may lead one to wrongly conclude that the model often retains incorrect parametric answer. This experiment highlights the importance of the strong answer equivalence metric (BEM, (Bulian et al. (022 and the filtering stage in our experimental pipeline. This design decision crucial in identifying samples with a realistic knowledge conflict.
1 BEM model available at: hhtgg:://uugingaaccee /uwp:///eoeo//t////////////o//on btteee/eooeetelaaeeeeereoeaaeaaa
2Original answer equivalence
dataset:
hnswer-eqrivalenc-ddaaesss HotpotQA answer equivalence dataset:
httpss//uuggiiiffce.coodttssttts
ae- he-tttthhototqaa
21

---

Published as a conference paper at COLM 2024
F Task adaptation using in-context learning
Motivation In-context learning (ICL) is known to improve language model performance on a wide array of NLP tasks. ICL involves providing several labeled examples as part of the context. It is considered to be method of adapting a general pre-trained model to perform a specific task. It is simpler and requires fewer resources than fine-tuning, but is generally more effective than instruction prompting alone.
RQ Can in-context demonstrations minimize the influence of the discovered parametric bias?
Evaluation setup The Llama2 model has context length of 2048 tokens which does not allow for many in-context demonstrations. We therefore use five in-context examples for Natural Questions, SQuAD, and HotpotQA datasets, they have shorter context lengths. For the other three datasets, we use two in-context demonstrations.
The Mistral-7B model employs sliding window attention, which enables flexible context lengths. Mixtral-8x7B has context length of 32768. This allows us to run the ICL experiment with eight context demonstrations for each dataset on both of those models.
The results for the Llama2 family are presented in Table 13a and Table 13b The 7B model continues to retain its parametric answer roughly the same number of cases. Moreover, the incorrect parametric answer in the context still increases the probability of knowledge update failure. For the larger 70B model, the trend similar overall. interestingly, for TriviaQA and SearchQA datasets Llama2-70B retains its parametric answer even more often with in-context demonstrations than without them. Furthermore, the negative influence of the parametric answer on update probability exists for all datasets. We conclude that for the Llama2 model family, in-context demonstrations do not minimize the influence of the parametric bias.
Results
The results for Mistral-7B and Mixtral-8x7B are reported in Table 13c 13d and Table 13d The influence of in-context demonstrations is two-fold: it improves open-book performance (P)))))), but also makes the model retain the incorrect parametric answer more often than without ICL. Furthermore, the parametric bias (difference in failure likelihood) is slightly exacerbated after introducing ICL for the majority of datasets.
Limitations Due to a limited context window, shown examples might not suffice for the model to learn the task. This phenomenon was explored in the literature for text classification tasks by the name of "demonstration bias" (Fan et al. 20024). This could explain the marginal improvement in the performance and ICL failing to prevent parametric bias.
Conclusion For all studied models better task adaptation through in-context learning does not mitigate the discovered parametric bias
22

---

Published as a conference paper at COLM 2024
(a) Llama2-7B
Dataset
P(R)
P(U,)
P(U;)
AP(R)
NQ
1.8
82.0
16.2
+11.4*
SQuAD
0.3
92.6
7.1
+2.7*
NewsQA
0.3
73.0
26.6
+2.0*
TriviaQA
3.8
79.8
16.3
+8,9*
SearchQA
3.8
55.6
40.6
+10.5*
HotpotQA
1.2
84.1
14.7
+7.3*
(b) Llama2-70B
Dataset
P(R)
PPUc.)
P(U)))
AP(R)
NQ
2.0
84.2
13.7
+13.2*
SQuAD
0.2
95.6
4.2
+2.2*
NewsQA
0.8
75.6
23.6
+4.7"
TriviaQA
6.1
76.5
17.4
+11.5*
SearchQA
8.9
70.1
20.9
+20.8*
HotpotQA
1.0
89.7
9.3
+5.8*
(c) Mistral-7B
Dataset
P(R)
P(U.))
P(U))
AP(R)
NQ
1.0
85.8
13.1
+8.5*
SQuAD
0.1
93.8
6.0
+2.7*
NewsQA
0.1
74.5
25.3
+4.5*
TriviaQA
4.1
81.0
14.8
+12.9*
SearchQA
3.8
74.0
22.1
+8.9*
HotpotQA
0.8
86.2
12.9
+5.7*
(d) Mixtral-8x7B
Dataset
P(R)
IPU.))
PU,))
AP(R)
NQ
2.7
84.8
12.4
+11.1
SQuAD
0.1
95.5
4.3
+0.5*
NewsQA
74.8
24.3
+3.5*
TriviaQA
8.5
75.5
15.9
+9.6*
SearchQA
7.4
70.7
21.9
+18.2*
HotpotQA
1.2
86.6
12.1
+5.2*
Table 13: Knowledge update success rates of each model when shown in-context demonstra- tions. We report empirical probabilities (in %) for each model that retains the parametric answer (P(R)), successfully updates its answer to the correct contextual one (P,(U))), or updates to an incorrect answer (U;), We additionally report the difference in the likeli- hood (in %) of knowledge update failure (AP(R)) for (ap C versus (ap & c). A positive difference means the parametric answer in context increases failure likelihood. Asterisk * denotes p 0.05.
23

---

Published as a conference paper at COLM 2024
G Task adaptation using fine-tuning
Motivation Fine-tuning a pre-trained language model on a downstream task is a powerful task adaptation method. Fine-tuning involves further training an LLM on a of labeled examples. It requires access to model weights and more computational resources than ICL or prompting but generally enables stronger downstream task performance. One caveat of fine-tuning is that by changing the model weights we may lose the generality of an instruction-tuned model.
RQ Can stronger task adaptation with fine-tuning minimize the influence of the discovered parametric bias?
We fine-tune the Llama2-7B model on the training subsets of the studied datasets. For example, the model evaluated on the validation subset of NQ was only trained on the training subset of NQ. We fine-tune the 4-bit quantized model using QLoRA applied to all linear layers. On all datasets, we train for epoch with batch size 2, constant learning rate of 2e-4, LoRA dropout set to 0.05 and not updating the biases. The dataset-specific (tuned) hyperparameters are reported in Table 14
Evaluation setup
# train examples
LoRA
LoRA alpha
Dataset
NQ
104,071
SQuAD
86,588
2
NewsQA
10,000
TriviaQA
10,000
8
8
SearchQA
10,000
64
HotpotQA 72,928
16
32
Table 14: Hyperparameters in the fine-tuning experiment.
Results We compare the fine-tuned model with the original prompted version used in the main experiments.
First, we report our observations on knowledge-updating behaviors and the influence of an incorrect parametric answer naturally appearing in the context in Table 15 As expected, fine-tuning improves open-book accuracy across all studied datasets. We find that fine-tuning reduces the chance of the model retaining its original incorrect answer. The fine-tuned version retains the original answer in only 1.2% of cases on average across datasets, compared to 1.6% for the prompted model. Regarding the parametric bins, we find that when an incorrect parametric answer naturally appears in context its influence on the failure likelihood is smaller for the fine-tuned model across all datasets.
Additionally, we run an intervention study, to compare how the fine-tuned model responds to artificially adding the incorrect parametric answers to context. We follow the setup in $ 4.4.2
The results reported in Table 16
The fine-tuned model is still susceptible to adding the incorrect parametric answer in context, however, to a lesser extent than the prompted version. For the original model, the likelihood of retaining the incorrect answer increases by 7.9% on average across all datasets. For the fine-tuned model the average increase only reaches 4.3%.
Discussion Fine-tuning reduces the parametric bias in our experiments. However, in the examined setup fine-tuning was not enough to completely eradicate it. Overall, our results suggest that fine-tuning is the most promising approach to combat the parametric bias in RAG systems. However, it comes the cost of potentially losing the generality of an instruction-tuned LLM.
24

---

Published as a conference paper at COLM 2024
(a) Llama2-7B
Prompted
Dataset
P(R)
P(Ue))
P(U)))
AP(R)
NQ
1.4
79.6
19.0
+8.8*
SQuAD
0.4
90.3
9.3
+4.0*
NewsQA
0.8
72.0
27.1
+4.8*
TriviaQA
3.4
79.3
17.3
+8.7*
SearchQA
2.3
61.5
36.3
+6.6*
HotpotQA
1.3
79.6
19.0
+10.7*
(b) Llama2-7B Fine-tuned
Dataset
P(R)
IP(U.)
PP(U))
AP(R)
NQ
0.2
89.0
10.9
+0.9*
SQuAD
0.0
95.5
4.4
+0.6
NewsQA
0.3
75.5
24.2
+2.5*
TriviaQA
4.4
82.1
13.6
+5.7*
SearchQA
1.7
85.3
13.0
+2.6*
HotpotQA
0.4
90.7
8.9
+1.4*
Table 15: Knowledge update success rates of Llama2-7B prompted and fine-tuned. We report empirical probabilities (in %) for each model that it Retains the parametric answer (R) successfully updates its answer to the correct contextual one (P)),)), or updates to an incorrect answer (U)). We additionally report the difference in the likelihood (in %) of knowledge update failure (R) for (ap S c) versus (ap & c). A positive difference means the parametric answer context increases failure likelihood. Asterisk denotes 0.05.
Llama2-7B Prompted
Llama2-7B Fine-tuned
Dataset
PP(R)
PPU,))
P(R)
P(U,)
NQ
12.5 (+11.11
(-9.5)
(+3.7)
(-3.1)
SQuAD
12.0 (+11.6)
77.3 (-0.4)
2.7 (+2.7)
92.7 (-2.8)
NewsQA
7.6 (+6.8)
66.2 (-5.8)
4.0 (43.7)
72.7 (-2.8)
TriviaQA
11.0 (+7.6)
76.2 (-3.1)
14.1 (49.7)
77.1 (-5.0)
SearchQA
6.6 (+4.3)
60.5 (-1.0)
2.7 (+1.0)
83.2 (-2.1)
HotpotQA
7.3 (+6.0)
72.8 (-688
(45.0)
86.3
(-4.4)
Table 16: Knowledge update success rate (%) after adding the parametric answer ap to the context after
words "Unrelated text:" (details in $.4.2)
4.4.2). Size of change in parenthesis. We compare the prompted
and fine-tuned versions of Llama2-/B.
25

---

Published as a conference paper at COLM 2024
H Influence of model size on the parametric bias
Motivation
Given enough training data, larger LLMs tend to perform better on both the pretraining objective and downstream tasks.
RQ How does the size of language model influence its knowledge-updating behavior and susceptibility to parametric bias?
Evaluation setup We compare two models of the same architecture that differ only in parameter count: Llama2-7B and Llama2-70B.
Results First, we report our observations on knowledge updating behaviors and the influence of an incorrect parametric answer naturally appearing in the context in Table 17 We find that 7B and 70B models have very similar behaviors with respect to knowledge updating: Llama2-7B retains its original incorrect answer in only 1.6% of examples average and Llama2-70B does that on average for 2% of examples. We also find that the two models are similarly susceptible failing their knowledge updates when their incorrect parametric answer appears in context.
Additionally, we run an intervention study, to compare how models of different sizes respond to artificially adding their incorrect parametric answers to context. We follow the setup S 4.4.2
The results are reported in Table 18
Both models are susceptible to parametric bias: adding the incorrect parametric answer the context increases their likelihood of retaining it even after seeing the context. For Llama2-7B the increase is on average by 7.9 %pt. For the larger 70B model, this increase on average by 4.8 %pt.
Discussion
These experiments provide evidence that increasing the model size be viable, although costly, solution to mitigate the discovered parametric bias. However, among the currently popular open models, even the largest ones are susceptible to this bias.
26

---

Published as a conference paper at COLM 2024
(a) Llama2-7B
Dataset
P(R)
PUUe))
P(U))
AP(R)
NQ
1.4
79.6
19.0
+8.8*
SQuAD
0.4
90.3
9.3
+4.0*
NewsQA
0.8
72.0
27.1
+4.8*
TriviaQA
3.4
79.3
17.3
+8,7
SearchQA
2.3
61.5
36.3
+6.6*
HotpotQA
1.3
79.6
19.0
+10.7*
(b) Llama2-70B
Dataset
P(R)
PP(e.)
P(U)))
AP(R)
NQ
2.0
84.2
13.7
+11.4
SQuAD
0.2
93.3
6.5
+2.3*
NewsQA
1.0
72.9
26.0
+2.4*
TriviaQA
4.0
65.3
30.7
+7.6*
SearchQA
4.1
65.8
30.0
+9.5*
HotpoiQA
1.0
87.0
12.0
+6.9*
Table 17: Knowledge update success rates of two models of Llama2 family of 7B and 70B sizes. We report empirical probabilities (in %) for each model that Retains the parametric answer (R)), successfully updates its answer to the correct contextual one (U)), or updates to an incorrect answer (PP())), We additionally report the difference in the likelihood (in %) of knowledge update failure (R) for (ap C c) versus (ap c). A positive difference means the parametric answer context increases failure likelihood.
Asterisk denotes p < 0.05.
Llama2-7B
Llama2-70B
Dataset
P(R)
P(U.)
P(R)
P(Uc)
NQ
12.5 (+11.1)
70.1 (-9.5)
7.9 (+6.2)
72.2 (-7.9)
SQuAD
12.0 (+11.6)
77.3 (-0.4)
3,5 (+3.3)
88.0 (-5.3)
NewsQA
7.6 (+6.8)
66.2 (-5.8)
6,9 (+5.9)
68.3 (-4.6
TriviaQA
11.0 (+7.6)
76.2 (-3.1)
11.6 (+7.6)
63.6 (-1.7)
SearchQA
6.6 (+4.3)
60.5 (-1.0)
(+3.9)
61.5 (-4.)
HotpotQA
7.3 (+6.0)
72.8 (-6.8)
3.2 (42.2)
84.4 (-2.6)
Table 18: Knowledge update success rate (%) after adding the parametric answer ap to the context
after words "Unrelated text:" (details in 4.4.2) Size of change in parenthesis. We compare 7B and 70B versions of the Llama2 model.
27

---

Published as a conference paper at COLM 2024
I Parametric answer is likely to appear in real-world documents
Retrieval-augmented generation is designed to address scenarios where the factual infor- mation changing. Supplying up date documents (e.g. web pages) as context for LLM inference then allows to update the system's 's knowledge and provide correct information to the end user. We argue, that in these scenarios the parametric answer of the model is likely to appear in the retrieved document.
First, the obsolete parametric answer is random and still very related to the question and the document that is supposed to answer it. Even if the answer is currently incorrect, is likely to be a previously correct answer, or the model's "best guess" at answering the question. This is especially true for larger models that store a lot of factual information in their parameters after the pre-training stage. Second, large and unstructured documents, especially web pages, may contain background information and historical data together with the current information. For example, the Wikipedia article for the President of the United States contains the text "Donald Trump" seven times, of this writing.
To ground this argument in a quantitative estimate, we employ the FreshQA benchmark (Vu et al. 2023). It is weekly updated QA dataset containing factual questions whose answers change with time. FreshQA contains questions together with answers and a URL for source web page containing the answer. We ask the question: How often does parametric answer of a language model appear in the source web page?
We only consider the subset of FreshQA where the answers change over time (slow-chl) chaw-changing and fast aasannnaggngg examples). When searching for the parametric answer to appear on the web page we use the jus python library extract text.
We report the results in Table 19
As we can the parametric answer of the studied LLMs 23-33% likely appear on the source web page. Moreover, significant portion of those answers are wrong, indicating that the parametric knowledge of the model is obsolete in those cases.
Real-world large retrieved documents can often contain the parametric answer of a language model. As we show in this paper, this appearance can influence the knowledge-updatinn behavior of the model. These results highlight the importance and relevance of the found paramefric bias phenomenon for real-world eerreval-augmeenee systems.
Model
Parametric answer in web page
Incorrect
Llama2-7B
33.5%
62.1%
Llama2-70B
32.7%
57.8%
Mistral-7B
23.2%
44.2%
Mixtral-8x7B
29.0%
54.2%
Table 19: Frequency of the parametric answer appearing on the source web page and which portion of those answers is incorrect. We report the results on the FreshQA dataset.
28

---

Published as a conference paper at COLM 2024
J GPT results
Motivation Closed-source LLMs are widely used in practice for RAG-based applications, despite being less promising for research due to their API-only access. We include one such model for the comprehensiveness of our study.
RQ Do our findings generalize to the GPT family of models?
Setup We run the OpenAT GPT-3.5-Turbo-0125 using the experimental setup presented in the main text. The dataset sizes at every step the experimental pipeline can be found in
Table
We report results in Table 20 GPT-3.5 retains its original parametric answer on average in 3.5% of cases. Similar to the studied open-source models, GPT-3.5 tends to update its knowledge from the context when presented with real conflicting documents.
Knowledge updating behaviors This experiment follows 4.2
GPT-3.5-Turbo
Dataset
P(R)
PPU,.)
P(U;)
NQ
1.7
84.0
14.3
SQuAD
0.1
95.5
4.3
NewsQA
0.5
77.5
22.0
TriviaQA
12.0
20.9
SearchQA
5.4
72.2
22.4
HotpotQA
1.3
87.0
11.7
Average
3.5
80.6
15.9
Table 20: Categorization of conflicted open-book QA answers of GPI-335-Turbo model. We report the proportion of open-book answers (in %) where the model retains its parametric answer (IP (P)) successfully updates its answer the correct contextual (P (P(U.)), or updates its answer incorrectly (P)(,,)) as defined in $32
Parametric bias influence We test for the influence of parametric bias as in 4.3.2 The
results reported in Table 21 For all six studied datasets, the incorrect parametric answer in context increases the likelihood of a failed knowledge update.
Dataset
GPT-3.5 Turbo
NQ
+13.0*
SQuAD
+2.4*
NewsQA
+7.6*
TriviaQA
+21.4*
SearchQA
+11.0*
HotpotQA
+6.9*
Table 21: Difference in the likelihood (in %) knowledge update failure (R) for (ap S c) versus (ap c). A positive difference means the parametric answer in context increases failure likelihood. Asterisk denotes statistical significance at 0.05.
Masking the parametric answer This section follows the setup of 4.4.1 We report the results in Table 22 For the GPT-3.5 model masking the incorrect parametric answer if it appears in the context reduces the likelihood that the model retains its original answer.
Artificially adding the parametric answer In this experiment, we add the original para- metric answer to the context preceded by the phrase "Unrelated text: " as in 4.4.2 The results can be found in Table 23 Similar the open-source models, GPT-3.5 Turbo retains its original answer more often after this intervention. This holds for all studied datasets.
29

---

Published as a conference paper at COLM 2024
GPT-3.5 Turbo
Dataset
[((R)
P(UUc)
NQ
0.7 (-1.0)
84.4 (-0.4)
SQuAD
0.1 (..0
95.4 (-0.1)
NewsQA
0.3 (-0.2)
77.7 (+0.2)
TriviaQA
(-2.9)
68.0 (+0.9)
SearchQA
3.3 (-2.1)
72.0 (-0.2)
HotpotQA
0.7 (-0.6)
87.5 (-0.5)
Table 22: Knowledge update success rate (%) after masking the parametric answer ap if appears in context. Size change parenthesis.
GPT-3.5 Turbo
Dataset
(R)
P(U,)
NQ
3,3 (+1.6)
75.1 (-8.9)
SQuAD
0.5 (+0.4)
94.3 (-1.2)
NewsQA
1.1 (+0.6)
76.2 (-1.3)
TriviaQA
21.5 (499.)
62.0 (-5.1)
SearchQA
8.0 (42.6)
66.1 (-6.1)
HotpotQA
2.4 (+1.1)
84.9 (-2.1)
Table 23: Knowledge update success rate (%) after adding the parametric answer to the context after words "Unrelaled lext:". Size of change in parenthesis.
Conclusion our experiments, GPT-3.5-Turbo exhibits the same behaviors as the studied open-source models. It readily updates its knowledge from real documents and it is also susceptible the discovered parametric bias.
30


Reviews:
[{"review_id": "HznCZCMeLy", "reviewer": "Reviewer_bvQZ", "summary": "Past work on LLM knowledge conflicts has shown that models struggle to correctly use in-context passages to answer the question and fall back on the parametric answer that they have memorized. Follow-up work showed that this is less true if the context is modified sufficiently to serve as convincing evidence. However, all these approaches relied on artificial edits to the context passage.\n\nThis paper proposes that LLM behavior in knowledge conflicts should be tested when (1) the parametric answer is incorrect and (2) the correct evidence passage is provided. Authors find that even in this realistic scenario where the context is not artificially edited, LLMs again predict their parametric answer a certain percentage of the time. Additionally, a large proportion of contexts result in the LLM in predicting a different incorrect answer. This behavior is established across several QA datasets and for 3 open-source models: Llama2-7B/70B, Mistral-7B, Mixtral-8x7B.\n\nThey further study the cause of this concerning behavior. They identify a cause they name `parametric bias': the model predicts the parametric answer again when it reappears in the context even though there is a correct answer elsewhere. That is, there is a confirmation bias where models predict their parametric answer even if it occurs in unrelated contexts.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and answer the questions below.\n\n> why did you focus on the smaller fraction of answer retention errors [...]?\n\nFrequency isn’t the only metric for problem importance, severity also matters. E.g, adversarial attacks are rare in real world. Yet, we study them. Answer retention is a severe failure of RAG. RAG is used to update model knowledge [a]. In these errors, context documents did not influence the model’s answer. By analyzing them, we discovered the parametric bias and understood answer retention better. We'll add this to the revision.\n[a] https://arxiv.org/abs/2005.11401\n> 1. Table 3:[...] what is in the error category 𝑈𝑖?\n\nWe classify 100 samples in 𝑈𝑖 for the Mixtral on NQ:\n- 42 incorrect answer format\n- 10 model answer is also correct, but != GT\n- 13 model answer is correct while the GT is incorrect\n- 12 model being bad at reading tables\n- 13 incorrect reading (prediction appears in context, but != GT)\n- 4 answers not from context\n- 5 model wrongly claims the answer is not in the context\n- 3 context is illegible\n\nWe’ll add this to the appendix.\n> 2. Table 4: Can you separate out 𝑃(𝑅|𝑎𝑝⊆𝑐)? Why did you choose to combine 𝑃(𝑅∪𝑈𝑖|𝑎𝑝⊆𝑐)?\n\n𝑅∪𝑈𝑖  includes all wrong open-book answers. We emphasize that 𝑎𝑝⊆𝑐 biases models towards wrong answers. We agree reporting only  𝑃(𝑅|𝑎𝑝⊆𝑐) would be clearer since we only analyze this type of error. We’ll change the presentation in the revision.\n\nFor Llama7B, separating out  𝑃(𝑅|𝑎𝑝⊆𝑐) shows a stat. significant difference in answer retention likelihood for all datasets (+8.8*, +4.1*, +4.8*, +8.7*, +6.6*, +10.7*), emphasizing the parametric bias phenomenon\n>3. Sec 4.4.1: [...] Is the model predicting the masked (edited) parametric answer instead [...]? What do you see is the error case?\n\n≈50% are predictions of edited answers. Others: wrong format, stated inability to answer using context, non-GT correct answers, synonyms (if also in context).\n> 4.Sec 4.4.1: How do you draw this conclusion? \n\nWe apologize for the confusion: In masking experiments, some cases of answer retention remain. We suspect that models resort to the original answer not only due to the parametric bias but in some cases, due to limited reading abilities -- if parametric bias were the only reason, masking the parametric answer should eradicate answer retention.\n>5. Sec 4.4.2: What explains the increase in 𝑃(𝑈𝑖)?\n\nThe majority of these cases are due to wrong answer format.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The identified `parametric bias' is concerning, and they provide additional evidence in the Appendix. In particular, they find that other datasets, such as the FreshLLMs QA dataset, also contain context passages with parametric answers (in addition to the correct answer)\n    - This characterization of confirmation bias is dangerous for down-stream RAG applications where users expect models to reason about the context correctly\n- The `parametric bias' is established through 3 experiments\n    1. The models make a mistake even with the gold context if the parametric answer appears somewhere in the passage frequently\n    2. An experiment is conducted to show that masking the parametric answer from the context reduces the number of times the models retain the parametric answer (caveat discussed later)\n    3. An additional experiment shows that adding the parametric answer to the context with the phrase \"Unrelated context:\" promotes the LLM to generate the incorrect parametric answer", "reasons_to_reject": "- I have one major question for the authors: why did you focus on the smaller fraction of answer retention errors over the larger fraction of errors where the model predicts a different but incorrect answer?\n    - This is concerning because this category of errors contributes to a larger fraction of errors overall\n    - In all ablation experiments, the fraction of errors of this kind seems to be increasing but are largely ignored", "questions_to_authors": "1. Table 3: As stated before, what is in the error category $U_i$?\n2. Table 4: Can yous separate out $P(R | a_p \\subseteq c)$? Why did you choose to combine $P(R \\cup U_i | a_p \\subseteq c)$?\n3. Sec 4.4.1: Since both $P(R)$ and $P(U_c)$ are decreasing, that means that $P(U_i)$ is increasing i.e. the model is predicting neither the parametric nor the correct contextual answer. Is the model predicting the masked (edited) parametric answer instead (thus putting it in the $U_i$ set? What do yo see is the error case?\n4. Sec 4.4.1: How do you draw this conclusion? The last line is the definition of $R$ and not $U_i$. “This indicates that the examples that were influenced by the incorrect parametric answer now moved into the Incorrect update category $U_i$. *This suggests that the model might be resorting to its original answer in cases when it cannot find the correct answer in the context.*”\n5. Sec 4.4.2: What explains the increase in $P(U_i)$ (in 7 of 18 settings) when we add back the parametric answer with the phrase \"unrelated context\"?", "extra_scores": {}}, {"review_id": "0VWRhI1bgK", "reviewer": "Reviewer_2iuH", "summary": "This work proposes to study another scenario of LLM knowledge conflicts: when the LLM parametric knowledge is wrong and the retrieved document contains evidence that supports the correct answer. Experiments demonstrate that models are much more likely to shift in this setting, while model behavior might be correlated with the occurrence of parametric/incorrect answers in the retrieved document.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed feedback. In the following, we address the concerns:\n>I'm not sure settings in previous works [...] are \"unrealistic\". Retrieval could bring noisy, outdated, or irrelevant documents into LLM prompts\n\nNoisy documents are indeed realistic. However, prior work studied **synthetic** documents designed to contradict model knowledge, which is unrealistic. We study real documents, which is more realistic.\n>If the setting in this work is much more prevalent, it would strengthen the motivation of this work.\n\nRAG is often applied to **trusted** documents in OOD domains for the LM ​​[a,b,c]. Moreover, as retrieval improves, real-world systems will move closer to our setting.\n[a] https://link.springer.com/chapter/10.1007/978-3-031-60615-1_9 \n[b] https://arxiv.org/abs/2309.02233\n[c] https://arxiv.org/abs/2403.16295\n>there probably is very little to investigate about knowledge conflicts in these scenarios.\n\nThe parametric bias finding shows that LLMs may read conflicting contexts incorrectly. This risk is even larger for more realistic systems (App. G). \n\n>Much of the analysis [...] focus on the 𝑅  cases [...] However, it only constitute a very small fraction of cases (<5% usually). I wonder if more effort should be spent on investigating the 𝑈𝑖 cases…\n\nApart from frequency, severity of a problem also matters.\nAnswer retention (𝑅) is a severe failure of RAG. RAG is used to update model knowledge. In 𝑅, context documents did not influence the model’s answer. By analyzing these errors, we discovered the parametric bias and understood answer retention better. We'll also add an analysis of the 𝑈𝑖 cases to the revision.\n>we often don't know right and wrong at inference time for either parametric answers or retrieved documents. \n\nWe propose evaluating RAG beforehand on known data to check for parametric bias in Sec. 6.\n>The findings and proposed \"interventions\" rely heavily on the assumption that \"parametric is wrong but retrieved is right\",\n\nWe assume this to create a clean experimental setting. “Interventions” are experiments to study the phenomenon, not practical solutions.\n> It would be nice to present some qualitative analysis and/or examples in the main paper.\n\nPlease consider: \n- Table 1: example of setup differences\n- Section 4.3.1: example of parametric answer in context\n- Section 4.4.2: example of adding parametric answer to the input. \n\nWe ask the reviewer to elaborate on what other examples would strengthen our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ knowledge conflict is an important research question\n+ the alternate scenario is interesting", "reasons_to_reject": "- I'm not sure settings in previous works where the parametric answer is right and retrieved docs are wrong are \"unrealistic\". Retrieval could bring noisy, outdated, or irrelevant documents into LLM prompts, so it is definitely a valid setting. What could support the setting in this work, instead of claiming it to be more \"realistic\", is an empirical evaluation of retrieval-augmented LMs and see how often both scenarios happen. If the setting in this work is much more prevelant, it would strengthen the motivation of this work.\n\n- According to Table 3, it seems that in the setting of focus in this work (parametric knowledge is wrong, retrieved docs are right), LLMs overwhelmingly rely on the retrieved document >95-98% of the time. This probably indicates that LLMs consistently favor the retrieved documents under these settings and we just need to improve retrieval quality, and there probably is very little to investigate about knowledge conflicts in these scenarios.\n\n- Much of the analysis (most of Sec 4.3 and 4.4) focus on the $\\mathcal{R}$ cases, i.e. the parametric answer is retained. However, it only constitute a very small fraction of cases (<5% usually). I wonder if more effort should be spent on investigating the $\\mathcal{U}_i$ cases where the answer changed but still wrong.\n\n- An important point is that we often don't know right and wrong at inference time for either parametric answers or retrieved documents. The findings and proposed \"interventions\" rely heavily on the assumption that \"parametric is wrong but retrieved is right\", which might not be applicable when the correctness is not known beforehand.\n\n- The intervention experiments might be a bit trivial technically, simply adding or removing the parametric answer in the retrieved document. This might not be a great issue though.\n\n- It would be nice to present some qualitative analysis and/or examples in the main paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1fegZ3SkxA", "reviewer": "Reviewer_gWDt", "summary": "This paper studies knowledge conflicts under a new setup: incorrect parametric knowledge conflicts with ground-truth knowledge in context. The paper discovers parametric bias, which means knowledge\nupdates can be impeded if incorrect parametric answers appear in the retrieved documents. Extensive experiments are conducted to demonstrate this.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing the merits of our work and providing constructive feedback. We address concerns in the following:\n> While the setup differs from previous works, the main finding seems not very novel.\n\nXie et al. report a similar finding for coherent yet false contexts. In our setup, we check behaviors in case of real (coherent **and** factually true) contexts, allowing us to find the parametric bias.\n>The paper does not provide a solution. It mainly focuses on analyzing and demonstrating a phenomenon.\n\nIndeed, we discover and examine the problem and its scope, and invite the community to work together on solving it. To facilitate, we try simple solutions: \n- ICL task adaptation (App. E)\n- Increasing model size (App. F)\n- Fine-tuning - our best result (below, will be added to revision).\n\nWe fine-tune Llama7B on the train split of each dataset.\n\n(R) subset becomes smaller, but the model is still susceptible to adding the parametric answer to context.\n\nOpen-book QA under knowledge conflict (cf. Section 4.2)\n|Data| R|Uc|Ui|\n|-|-|-|-|\n|NQ|0.1|89.0|10.9|\n|SQuAD|0.0|95.5|4.4|\n|NewsQA|0.3|75.5|24.3|\n|TriviaQA|4.4|82.1|13.6|\n|SearchQA|1.7|85.3|13.0|\n|HotpotQA|0.4|90.7|8.9| \n\nAfter adding the incorrect parametric answer (cf. section 4.4.2): \n|Dataset|R|Uc|Ui|\n|-|-|-|-|\n|NQ|3.9|85.9|10.2|\n|SQuAD|2.7|92.7|4.6|\n|NewsQA|4.0|72.7|23.3|\n|TriviaQA|14.0|77.1|8.8|\n|SearchQA|2.7|82.2|14.0|\n|HotpotQA|5.4|86.3|8.4|\n\n> in reality, the retrieved knowledge might be noisy and not necessarily be the ground truth.\n\nNoisy retrieved documents are indeed realistic. However, prior work studied **synthetic documents designed to contradict** model knowledge, which is unrealistic. We study real documents, which is more realistic.\n> Why the happening of knowledge updates is determined by exact match? \n\n1. To be comparable to previous work\n2. RAG is designed to update model knowledge. If context had 0 influence on the answer, it is a critical failure of RAG.\n\n> Will changing it into semantic equality change the results much?\n\nWe re-ran experiments on Llama7B using BEM to determine knowledge update. \n\nTable 3, open-book QA on the KC subset \n|Dataset|R|Uc|Ui|\n|-|-|-|-|\n|NQ|4.8|79.9|15.3|\n|SQuAD|1.6|90.3|8.1|\n|NewsQA|4.9|72.3|22.8|\n|TriviaQA|5.7|79.5|14.7|\n|SearchQA|7.3|62.0|30.7|\n|HotpotQA|2.7|79.6|17.8|\n\nThe general trends are the same, but more examples are in R as more answer pairs are equivalent according to BEM. Please let us know if there are any further questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Probing the incorrect parametric knowledge to create knowledge conflicts is novel and interesting.\n2. The experiments are comprehensive and well-designed. They fully demonstrate the main conclusion. \n3. The paper is well-written and easy to follow.", "reasons_to_reject": "1. While the setup differs from previous works, the main finding seems not very novel.\n2. The paper does not provide a solution. It mainly focuses on analyzing and demonstrating a phenomenon.\n3. The paper emphasizes knowledge conflicts under its setup are realistic. However, in reality, the retrieved knowledge might be noisy and not necessarily be the ground truth.", "questions_to_authors": "1. Why the happening of knowledge updates is determined by exact match? Will changing it into semantic equality change the results much?", "extra_scores": {}}, {"review_id": "H7GZ3p27zZ", "reviewer": "Reviewer_y1YK", "summary": "This paper proposes a framework to investigate the knowledge conflict problem in the RAG scenarios. Compared with previous work, the authors claim that they use real conflicting documents rather than synthetic information, which is closer to the real-world setting. According to this paper, LLMs are easy to update their parametric memory when external conflicting evidence is presented in the context. Furthermore, the authors find a 'parametric bias' phenomenon: It's hard for LLMs to update their parametric memory if there is parametric memory-related information in the context, although it is incorrect and unrelated to the question.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and questions. We answer them below:\n> 1.using in-context demonstration to elicit answers. However, could this introduce demonstration bias? \n\nWe understand demonstration bias for generative QA as “shown examples do not suffice to learn the task” (analogous to [a] for classification). Due to the limited context window, it could explain marginal improvement in the performance and ICL failing to prevent parametric bias (App. E). We’ll add this discussion to the revision. If we misunderstood, we kindly ask the reviewer to be more specific.\n\n[a] https://arxiv.org/pdf/2312.07476\n> 2.What is the accuracy of BEM? Can it ensure matching quality?\n\n[b] showed BEM to be a stronger answer matching metric than EM. In our experiments it also outperforms EM (89% vs. 55% on original [b] data, 92% vs. 22% on our manually labeled test data from HotpotQA). We’ll add this to App. D.\n\n[b] https://arxiv.org/abs/2202.07654\n> 3.it's essential to include at least one closed-source LLM.\n\nWe will add results for GPT-3.5 (below).\n\nTable 3, Conflicted open-book QA\n|Dataset|R|Uc|Ui|\n|-|-|-|-|\n|NQ|1.7|84.0|14.3|\n|SQuAD|0.1|95.5|4.3|\n|NewsQA|0.5|77.5|22.0|\n|TriviaQA|12.0|67.1|20.9|\n|SearchQA|5.4|72.2|22.4|\n|HotpotQA|1.3|87.0|11.7|\n\nTable 4 Diff. in the likelihood of update failure\n|Dataset|GPT-3.5|\n|-|-|\n|NQ|+10.3*|\n|SQuAD|+0.9|\n|NewsQA|+5.9|\n|TriviaQA|+17.2*|\n|SearchQA|+16.2*|\n|HotpotQA|+6.8*|\n\n> 4. Does it mean the authors have fine-tuned the models, or have they simply selected one of the best prompts? \n\nWe selected the best prompt. We did not fine-tune the models. We will clarify this in the revision.\n> 5. Xie et al. found that LLMs are highly likely to update their parametric memory as long as the external evidence is coherent. Could you please justify the difference? \n\nXie et al. study coherent yet false contexts. In our setup we check behaviors for real (both coherent and factually true) contexts, which allowed us to find the parametric bias.\n\n> Q1. Could the phenomenon of 'masking reducing the likelihood of retaining parametric answers' be attributed to low-quality masking?\n\nIn our understanding, you suggest masking makes inputs OOD (“missingness bias” [c]). To minimize it, we test various strategies: using attention mask [c], varying mask tokens, using paraphrases, and changing mask size. We use the strategy with the best open-book accuracy. We ask the reviewer to elaborate if we misunderstood.\n\n[c] https://arxiv.org/abs/2204.08945", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The knowledge conflict is an important problem in the RAG scenario, and the authors' findings are really helpful for the community.\n2. This paper is easy to follow.\n3. The experiments are sufficient and the analysis is promising.", "reasons_to_reject": "1. In stage 1, the authors mention using in-context demonstration to elicit answers. However, could this introduce demonstration bias? The authors should clarify this aspect in the paper.\n2. In stage 2, the authors mention using BEM to match the answers. What is the accuracy of BEM? Can it ensure matching quality? The authors should provide clarification on this in the paper.\n3. Given the widespread use of closed-source LLMs in real-world RAG scenarios, it's essential to include at least one closed-source LLM to ensure the findings can be generalized to these models.\n4. Why do the authors mention \"tune the prompts\" in Stage 3? I've checked the appendix, but I'm still confused about this point. Does it mean the authors have fine-tuned the models, or have they simply selected one of the best prompts? If it's the former, could this introduce another training bias since the parameters have been changed? If it's the latter, the authors should reconsider the terminology.\n5. The first finding doesn't seem entirely new, as it has been reported in \"Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts,\" which is also referenced in this paper. Specifically, Xie et al. found that LLMs are highly likely to update their parametric memory as long as the external evidence is coherent. Could you please justify the difference?\n\n(I will reconsider the score if the authors can address the points mentioned above to my satisfaction.)", "questions_to_authors": "1. Could the phenomenon of 'masking reducing the likelihood of retaining parametric answers' be attributed to low-quality masking? It's possible that LLMs are strong enough to identify such obvious misinformation?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an empirical study/analysis of LLM's behavior under knowledge conflict -- between model's internal parametric knowledge and provided in-context documents. The paper is clearly written with valid evaluation setting. While they study slightly newer angle to the analysis on parametric vs. non-parametric knowledge -- looking at cases where model's initial prediction was incorrect and provided in-context passage is correct. The reviewers were concerned this only account for a small subset of examples, but these are still valuable subset. The takeaways/actionable items from the analysis is a bit unclear, and the scope of analysis is a bit limited (and overlaps with rich prior work). 



One point: 
I agree with reviewer 2iuH that this setting is not necessarily "realistic" than prior settings, and recommend changing the title. Also -- 
"Re: prior work studies "synthetic document"" is not a valid representation of prior work. "Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence" Chen at al EMNLP 2022, Section 5 studies knowledge conflict where real documents with conflicts are drawn from different time stamped corpus or different interpretation of the question.

[comments from the PCs] Please revise your paper to better represent prior work as the AC notes.
Decision: Accept