Adaptive Chameleon  or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
Authors: Jian Xie
Kai Zhang
Jiangjie Chen
Renze Lou
Yu Su
Venue: ICLR 2024 spotlight (2024)
Keywords: Large Langugage Model
Knowledge Conflict
Tool Augmentation
Primary area: societal considerations including fairness, safety, privacy
TL;DR: This paper explores LLMs' responses to knowledge conflicts: are they Adaptive Chameleons, open to external evidence, or Stubborn Sloths , holding onto parametric memory?

Abstract: By providing external information to large language models (LLMs), tool augmentation (including retrieval augmentation) has emerged as a promising solution for addressing the limitations of LLMs' static parametric memory.
However, how receptive are LLMs to such external evidence, especially when the evidence conflicts with their parametric memory? 
We present the first comprehensive and controlled investigation into the behavior of LLMs when encountering knowledge conflicts.
We propose a systematic framework to elicit high-quality parametric memory from LLMs and construct the corresponding counter-memory, which enables us to conduct a series of controlled experiments.
Our investigation reveals seemingly contradicting behaviors of LLMs.
On the one hand, different from prior wisdom, we find that LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing.
On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information that is consistent with their parametric memory, despite being presented with conflicting evidence at the same time.
These results pose important implications that are worth careful consideration for the further development and deployment of tool- and retrieval-augmented LLMs.
Resources are available at https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict.

Published as a conference paper at ICLR 2024
Adaptive Chameleon or Stubborn Sloth: REVEALING THE BEHAVIOR OF LARGE LANGUAGE MODELS IN KNOWLEDGE CONFLICTS
Jian X
Kai Z Jiangjie Chen
Renze Lou ?
Yu Su
School of Computer Science, Fudan University
The Ohio State University The Pennsylvania State University j jianxie22em. fudan. edw.ed..c. cn, [zhang. 13253, su. 809)eos.. edu
wudan.edu.co..
ABSTRACT
By providing external information to large language models (LLMs), tool aug- mentation (including retrieval augmentation) has emerged as promising solution for addressing the limitations of LLMs' static parametric memory. However, how receptive are LLMs to such external evidence, especially when the evidence con- flicts with their parametric memory? We present the first comprehensive and controlled investigation into the behavior of LLMs when encountering knowledge conflicts. We propose a systematic framework to elicit high-quality parametric memory from LLMS and construct the corresponding counter-ment menntrrrmemor, which enables us to conduct a series of controlled experiments. Our investigation re- veals seemingly contradicting behaviors of LLMs. On the one hand, different from prior wisdom, we find that LLMs can be highly receptive to external ev- idence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing. On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information that is consistent with their parametric memory, despite being pre- sented with conflicing evidence at the same time. These results pose important implications that are worth careful consideration for the further development and deployment of tool- and retrieval-auugmente LLMs. Resources are available at http://gtttlbcomm///SU/PP/oogol
INTRODUCTION
After pre-training on massive corpora, large language models (LLMs) BBronn et al. 2020 Chowdhery al. 2022 Ouyang al. 2022 OpenAl 2022 2023 Zeng al. 2023 Touvron al. 2023a) have formed a wealth of parametric memory, such as commonsense and factaal Knowledge (Petroni et al.
2019: Li al. 2022 Zhao al. 2023). However, such parametric memory may be inaccurate or become outdated (Liska et al. 2022; Luu et al. (022) due to misinformation in the pre-training corpus
or the static nature of parametric memory, known to major cause for hallucinations (Elazar et
2021
Shuster et al.) 2021
Ji et al. 2023).
Tool
Schick et al. 2023 Qin al. 2023) or retrieval augmentation (Mallen et al. 2022 Shi et al.
2023b
Ram et al.) 2023) has emerged as promising solution by providing external information as new evidence to LLMss, such as ChatGPT Plugins and New Bing. However, external evidence, inevitably, could conflict with LLMs' parametric memory. We refer external evidence that conflicts with parametric memory as counter-memory. In this paper, we seek to answer the question: how receptive are LLMs external evidence, especially counter-memory? A solid understanding of this question is an essential stepping stone for wider application of holl-augmented LLMs. Not only does this relate to overcoming the limitations of LLM's static parametric memory, but it is also associated
* 'The first two authors contributed equally. Work done during Jian Xie's 's internship at OSU NLP Group In the rest of the paper use "tool--ngmented L.MMs" because retrievers one type of tools, but tools limited to retrievers (consider, e.g., a question answering tool).

---

Published as a conference paper at ICLR 2024
with direct safety concerns. For example, what third- -hirt- tool, either by the developer or hijacked by attackers, intentionally returns disinformation? Will LLMs be deceived?
We present the first comprehensive and controlled investigation into the behavior of LLMs when encountering counter- menttrrr A key challenge lies in how to construct the counter memory. Prior work employs various heuristics, such negation injection (Niu & Bansal 2018, Kassner al. 2021 Gubelmann & Handschuh]] (022) and entity substitution (onggree et al.] Zhou et al. 2023). and finds that language models (both large and small) tend to be stubborn and cling their parametric memory. However, such heuristic word-level editing results in incoherent counter-memory (see an example in Section 4.1), which may make it trivial for LLMs to detect and thus neglect the constructed counter-memory. It is unclear how the prior conclusions translate to real-world scenarios, where counter-memory is more coherent and convincing.
We propose a systematic framework to elicit the parametric memory of LLMs and construct the corresponding counter-memory. We design series of checks, such as entailment from parametric memory to the answer, to ensure that the elicited parametric memory is indeed the LLM's internal belief. For the counter-memory, instead of heuristically editing the parametric memory, we instruct an LLM to directly generate coherent passage that factually conflicts with the parametric memory. After obtaining a large pool of parametric memory and counter-memors pairs, we then examine LLMs' behavior in different knowledge conflict scenarios, including 1) when only counter-memory is present as external evidence and 2) when both parametric memory and counter-memory are present.
Our investigation leads to a series of interesting new findings. We highlight the following:
LLMs are highly receptive to external evidence if that is the only evidence, even when it conflicts with their parametric memory. This contradicts the prior wisdom (Longpre) (et al.] 2202T), and we attribute this to the more coherent and convincing counter-memory constructed through our framework. On the other hand, this also suggests that L.MMs may be easily deceived by, c.g., disinformation from malicious (third-party) tools.
However, with both supportive and contradictory evidence their parametric memory, LLMs show strong confirmation bias (Nickerson) 1998) and tend to cling to their parametric memory. This reveals a potential challenge for LLMs to unbiasedly orchestrate multiple pieces of conflicting evidence, common situation encountered by generative search engines.
RELATED WORK
Parametric Memory in Language Models After pre-training, language models have internalized a vast amount of knowledge into their parameters (Roberts et al. 2020 Jiang et al. 2020). also known as parametric memory. Many past studies have explored the elicitation of parametric memory in language models, such as commonsense or factual knowledge probing (Petroni et al. 2219 Lin et al., 2020/Zhang Zhang et al. 202T West al. 2022, Chen et al.) 2023 Wang et al.) (023). Such parametric 2020
2023 Sun et al. (023). However, previous work has discovered that language models only memorize small portion of the knowledge they have been exposed during pre-training (Carlini al. 202 2023) due to model's limited memorization abilitics. In addition, the parametric memory may become outdated (Lazaridou et al. 2021 De Cao al. 2021) Such incorrect and outdated parametric memory may show as halluciattonss (Elazar et al. 2021 Shuster et al.) [02T]] Di et al.] (202)). Although some methods are proposed to edit knowledge in language models (Dai et al.) |002 Meng et al.) [022] 2023), they typically require additional modifications on model weights without evaluating the consequences on models' other aspects such as performances and are limited to factual knowledge.
memory could help solve downstream tasks (Wang et al., 2021: Yu et al.
Tool-augmented Language Models To address the limitations of parametric memory, external tools such as retrievers are used to augment language models with up-to-date information, namely
tool-augmented (Nakano et al.) 2021 Yao et al. 2023 Qin et al. 2023; Schick et al. 2023 Lu ct al.
2023) retrieval Saweaallauuuan -augmented (Guu et al. 2020: Khandelwal et al. 2020 Izacard & Grave 202 I
Borgeaud et al.) 2022 Zhong et al. 2022) language models. Such a framework, which has proven its efficacy in enhancing large language models (Shi et al. 2023b Ram et al. 2023 Mallen et al.) 2022), is adopted in real-world applications such as New Bing and ChatGPT Plugins. Inevitably, the external evidence could conflict with the parametric memory. However, the behavior of LLMs in
2

---

Published as a conference paper at ICLR 2024
Step
Parametric Memory Elicitation
Counter-memory Construction
Entailment Checking
Answer Consistency
Evidence Preference
Answerr the given cuistion and explain the reason.
Given the senterce, maxe up a supporting exidence.
ven the exidenoe, ansiver
Given the cvidence, answer the question.
Parts liaseabic on- Curndee
Bie queztio.
Question:
Sentence:
Peeplend and
Question: spientss of Mind?
Question:
Who chief sclenest
Deff Deam the chiif sds
splentist
ntter of Google Decplitnd
Desphlind?
Sleenti Sasog Deephind
Damix Hansable is the shse
Poni Hascobis founded Bocshlind 2013 and
Evidence:
Sumsa Hossa ble foundied Douphtin 2013 and
Evidence
Closed-book
evidance
support?
Curmently Dean an the charr apctentic of
Answer souncit Google Desplind
Counter-memory
Damii Hassabis
Kurrenty Dean sceving
NLJ Model
A Confficrt
Parametric Memory
Consistent?
Passablc
Google DiegHind, lending
paephiin 2010 and
Ird conparch soer stncw
nouses actannc no
research organiration that
Support
actentiel Google Deepptin
Who?!
Figure 1: Our framework for simulating knowledge conflicts. In Step 1, we elicit LLMs
memory answer and parametric memory in a closed-book QA fashion. In Step 2, we construct counter-answer to memory answer with heuristic rules, for which ChatGPT generates supporting counter-memory with instructions. To uphold evidence quality, conduct entailment checking (Step 3) and answer consistency (Step 4) to filter unqualified examples. All experiments are imple- mented under zero-shot setting avoid the bias introduced by demonstrations.
knowledge conflict scenarios remains under-explored, and unraveling it holds significance for wider applications of tool-augmented LLMs.
Knowledge Conflict To perform controlled experiments, knowledge conflict is often simulated
with counter-memory constructed upon parametric memory. Heuristic counter- memory construction
methods such as negation injection (Niu & Bansal 2018 Kassner et al. 2021 Petroni et al. 2020
Pan et al., (021] have been developed. Furthermore, entity substitution Longpre al.) 2021 Chen
et al. 2022 Si et al. 2023) Zhou et al. 2023) replaces all mentions of the answer entity in parametric memory with other entities construct counter- counter-memory. However, these methods are limited to word-level editing, leading to low overall coherence in the counter-memory, We instead instruct
LLMs to generate counter-memory from scratch to ensure high coherence.
EXPERIMENTAL SETUP
In this section, describe our framework for eliciting high-quality parametric memory from LLMs and constructing the corresponding counter-memory, as well as the evaluation metrics.
3.1 DATASETS
Following prior work (Longpre et al. 2021 Chen et al.) 2022), ), we adopt question answering (QA) task as the testbed for knowledge conflict experiments. In addition entity-based QA dataset (POPQA), we include a multi-step reasoning dataset (Strategy QA) for diversifying the questions studied in the experiments. Specifically,
PorQA (Mallen et al., 2022) is an entity-centric QA dataset that contains 14K questions. Data for POPQA originates from triples in Wikidata. Employing custom templates tailored to relationship types, the authors construct questions through the substitution of the subject within knowledge triples. POPQA defines the popularity of question based on the monthly
Wikipedia page views associated with the entity mentioned in the question.
STRATEGYQA (Geva al. 2021) is multi-step fact reasoning benchmark that necessitates the implicit question decomposition into reasoning steps. The questions are built around Wikipedia terms and cover wide range of strategies, which demand the model's capability to select and integrate relevant knowledge effectively. The language model is expected to provide True or False answer.
3

---

Published as a conference paper at ICLR 2024
Table 1: The correctness of LLMs responses in closed-book QA fashion (Step 1 in Figure 1 We examine eight LLMs, including three closed-source LLMs and five open-source LLMs.
PoPQA
STRATEGYQA
Models
Correct
Wrong
Unknown
Correct
Wrong
Unknown
Closed-source LLMS
ChatGPT OpenAl 2022
44.6
44.4
11.0
67.4
30.7
1.9
GPT-4 OpenAl 2023)
50.8
48.7
0.5
77.3
22.7
0.0
PaLM2 Anil ct al. 2023
32.9
67.1
0.0
67.9
32.1
0.0
Open-source LLMs
Qwen-7B (Alibaba 2023)
24.9
62.6
5.1
56.8
43.2
0.0
Llama2-7B Louvron ct al.
2023b
24.1
75.9
0.0
56.7
43.3
0.0
Llama2-70BB [Touvron et all
2023b
43.0
57.0
0.0
64.4
35.7
0.0
Vicuna-7B (Zheng et al.
2023
23.8
69.3
6.9
55.0
45.0
0.0
Vicuna-33B (Zheng
2023
28.6
71.4
0.0
65.0
35.0
0.0
3.2 PARAMETRIC MEMORY ELICITATION
Step in Figure illustrates how we elicit parametric memory: in a closed-book QA fashion, LLMs recall their parametric memory answer questions without any external evidence. Specifically, given question, e.g., "Who is the chief scientist of Google Deep.indd,, LLMs are instructed to provide an answer "Demis Hassabis" and its supporting background information about how Demis founded and led DeepMind in detail. We cast the detailed background as parametric memory because the answer only represents the conclusion of parametric memory w.r.f the given question.
Table 1 shows the closed- book results of LLMs on PoPQA and STRATEGYQA. Notably, LLMs may respond with "Unknown" when no evidence is provided in the context, particularly in ChatGPT. Such answer abstention (Rajpurkar et al. 2018) 8) suggests that LLMs fail to recall valid memory associated with the given question, so we discard them. For comprehensiveness, we also keep the examples that LLMs answer incorrectly in the closed-book paradigm because the wrong answer and associated memory are also stored in model parameters.
3.3 COUNTER-MEMEN CONSTRUCTION
As depicted in Figure 1 at Step 2, we reframe the memory answer "Demis Hassabis" to counter- answer (c.g., "Jeff Dean") "). Concretely, for POPQA, we substitute the entity in the memory answer with same-type entity (e.g.. from Demis to Jeff); while in STRATEGY YA,, flip the memory answer (e.g., from positive sentence to negative sentence). With counter-answer "Jeff Dean", we instruct ChatGPT to make up supporting evidence that Jeff Dean serves chief scientist of DeepMind. We term such evidence that conflicts with parametric memory as counter-memory.
Since the counter-memory is generated from scratch by powerful generative LLMs, it compared to previous word-level editing methods (Longpre et al. [021] Chen et al. 2022) performed on parametric memory. Both generated parametric memory and counter-memory could serve as external evidence for later experiments on LLMs in knowledge conflicts. Please refer to Appendix B. for more details of evidence construction in each dataset.
coherent
3.4 ANSWER-EVIIR EVIDENCE ENTAILMENT CHECKING
An ideal piece of evidence should strongly support its answer. For instance, the parametric memory about Demis and DeepMind should clearly support the corresponding memory answer that Demis is the chief scientist of DeepMind. Similarly, counter-memory should clearly support the corresponding counter-answee as well. Therefore, for Step 3 shown Figure I we utilize natural language infer- ence (NLI) model for support-ches -heeckng to ensure the evidence indeed entails the answer. Specifically,
We leverage ChatGPT for its cost-effectiveness and its on par counter- memtrr generation ability with GPT-4. In pilot study (based on 1000 instances), L.MSs showed the same level of receptiveness counter-memory generated by both ChatGPT and GPT-4.
4

---

Published as a conference paper at ICLR 2024
Table Examples in final dataset. Both parametric memory and mommrry are generated by ChatGPT. They are coherent and present an illusion of correctness even when factually incorrect.
Correct Memory
Wrong Memory
Question
What the capital of Kingdom of France?
Who producer The Cardinal?
Ground Truth
Parts
Otto Preminger
Paris was chosen as the cupital of the Kingshom of France due its central location and strategic importance. It was also major cultunal and economic center, making ideal location for the scat govemment. Additionally. Paris a long history ss royal and political center, dating back the medieval pesiod.
The Cardinal was produced by Martin Ronsehoff, who was promiinntt film and television produces in the mid-2ltth contury. He was for producing number successfal films, and soboff
Parameteic Memory
co-forder Filmways, production company that produced popula: TV shows such as The Addams Family.
is capital the Kingskon of France This can scen the otticaal government website of France, where is listed the copital city. Adtitionally, Néma home the French govenmmert further soliditying capital. The city is also with nomerous mosemms, gallenies and businesses.
Otto Premingge was promincat film producer century, known for his work oa namber of successful films, One of his most notable productions was the 1963 fil The Curdinal. which directed him and cntticll and commercial success, receningg several Academy Award acminations and grossing over $10 million at the box office.
mad-20th
Counter-memory
royal palace and the the cultural and economic activities,
Tryon. The film
we use the state-of-the-ard NLI model DeBERTa-V V2 parametric memory and counter- memory support their corresponding answers. We only keep the examples where both answers are supported for subsequent experiments.
He et al. 2021 to determine whether both the
To ensure the reliability of the selected NLI model, manually evaluated 200 random examples and observed 99% accuracy of the model. Please refer to Appendix [3.] for more details.
3.5 MEMORY ANSWER CONSISTENCY
We adopt another check (Step 4 of Figure for further ensuring the data quality. If the parametric memory we elicit is truly the internal beliet of an LLM's, presenting it explicitly evidence should lead the LLM to provide the same answer as in the closed-book setting (Step 1). Therefore, in the evidence-basse QA task format, we use the parametric memory as the sole evidence and instruct LLMs to answer the same question again. For example, given the parametric memory about Demis and DeepMind, LLMs should have consistent response with the previous memory answer, that Demis is the chief scientist of DeepMind.
However, the answer inconsistency results in Table show that LLMs may still change their answers when the parametric memory obtained in Step is explicitly presented as evidence. This suggests that the LLM's internal belief on this parametric memory may not be firm (e.g., there may competing answers that are equally plausible based on the LLM). We filter out such examples to ensure the remaining ones well capture an LLM's firm parametric memory.
After undergoing entailment and answer consistency checks, the remaining examples are likely to represent firm parametric memory and high-quality counter-memory, which lay a solid foundation for subsequent knowledge conflict experiments. Some examples from the final PoPQA data are shown in Table and the statistics of the final datasets are shown in Table 4 Please refer to Appendix B.2 for more details for Step and 4 and examples.
3.6 EVALUATION METRICS
A single generation from LLM could contain both the memory answer and the counter-answer, which poses a challenge to automatically determine the exact answer from an LLM. To address this issue, transform the free-form QA to multiple-choice QA format by providing few options as possible answers. This limits the generation space and helps determine the answer provided by LLMs with certainty. Specifically, for each question from both datasets, LLMs are instructed select one answer from memory answer (Mem-Ans.), counter-answer (Ctr-Ans.), and "Uncertain". Additionally, to quantify the frequency of LLMs sticking to their parametric memory, we adopt the memorization ratio metric (Longpre et al. 2021 Chen et al. 2022):
fm
MR fm + fc
(1)
5

---

Published as a conference paper at ICLR 2024
Table 3: Answer inconsistency rate between closed-book results (Step 1) and evidence-based QA with parametric memory (Step 4).
Table 4: Number of final examples for each LLM. The difference between LLMs is due to their different outputs going through the framework.
PorQA
STRATEGYQA
POPQA(#)
STRATEGYQA
ChatGPT 4.7%
3.7%
ChatGPT
7,947
1,245
GPT-4
3.9%
2.6%
GPT-4
9,544
1,356
PaL.M
8.4%
2,7%
PaL.M2
5,256
500
Qwen-7B 5.4%
5.6%
Qwen-7B
7,204
671
Llama2-7B
4.7%
7.3%
Llama2-7B
8,027
698
Llama2-70B
2.3%
0.7%
Llama2-70B
9,314
822
Vicuna-7B 12.4%
6.9%
Vicuna-77
4,170
559
Vicuna-33B 16.6%
5.3%
Vicuna-335 -3B
3,787
775
where fn is the frequency of memory answer and fo is that of counter-answer, Higher memorization ratios signify LLMs relying more on their parametric memory, while lower ratios indicate more frequent adoption the counter-memory.
4 EXPERIMENTS
4.1
SINGLE-SOURCE EVIDENCE
We experiment with LLMs the single-source evidence setting where counter-memory the sole evidence presented to LLMs. Such knowledge conflict happens when LLMs are augmented with tools returning single external evidence such as Wikipedia API (Yao et al.) 2023). particular, for counter-memory construction, we would apply 1) the entity substitution counter-memory method, widely-applied strategy previous work, and 2) our generation- -based method.
LLMs are stubborn when encountering entity substitution-base counter-memory. Following previous work (Longpre et al. 2021; Chen et al.) 2022), we substitute the exactly matched ground truth entity mentions the parametric memory with random entity of the same type. The memory is then used as the sole evidence for LLMs to answer the question. Here is an example:
ChennChen&
Evidence: Washington D.C. London, USA's capital, has the Washington Monument. Question: What the capital city of USA? Answer by ChatGPT: Washington D.C.
Figure shows the results with this approach on PorQA dataset. Observably, although the instruction clearly guides LLMs to answer questions based on the given counter-memory, LLMs still stick to their parametric memory instead, especially for three closed-sourced LLMs (ChatGPT, GPT-4, and PaLM2). This observation is aligned with previous work (Longpre et al., 202T). The reasons may stem from the incoherence of the evidence built with substitution: In the given example, although "Washington D.C." is successfully substituted by "London", the context containing Washington Monument and USA still highly correlate with the original entity, impeding LLMs to generate London as the answer. Furthermore, when comparing Llama2-7B and Vicuna-7B to their larger counterparts in the same series (i.c., Llama2-700 20BB and Vicuma-33B) observe that the larger LLMs are more inclined to insist on their parametric memory. We suppose that larger LLMs, due to their enhanced memorization and reasoning capabilities,
sensitive incoherent sentences.
LLMSs are highly receptive to generated coherent counter-memory To alleviate the incoher- ence issue of the above counter-ment memory, we instruct L.MMs to directly generate coherent counter- memory following the steps aforementioned (Figure D Figure shows the experimental results with generation-basee counter-memory from which we can have the following observations:
First, LLMs are actually highly receptive to external evidence if it is presented in coherent way, even though conflicts with their parametric memory. This contradicts the prior conclusion (Longpre) et al. 22221) and the observation with entity substitution counter- memoory shown in Figure 2 Such high receptiveness in turn shows that the counter-memorg constructed through our framework is indeed more coherent and convincing. We manually check 50 stubborn (i.e., "Mem-Ans." cases and

---

Published as a conference paper at ICLR 2024
100
100
100
100
Memolis
80
80
80
60
CheAas
60
60
60
40
40
40
40
20
20
20
20
Subs.
Gen.
Subs.
Gen.
Subs.
Gen.
Subs.
Gen,
(a) ChatGPT
(b) GPT-4
(c) PaLM2
(d) Qwen-7B
100
100
100
100
80
80
80
80
60
60
60
60
40
40
40
40
20
Subs.
Gen.
Subs.
Gen.
Subs.
Gen.
Subs.
Gen.
(c) Llama2-7B
(f) Llama2-70B
(g) Vicuna-7B
(h) Vicuna-33B
Figure 2: Answer distributions of entity substitution-base (Subs.) and generation-based (Gen.) counter-memory the single evidence. Mem-Ans. and Ctr-Ans. refers memory answer and counter-answer, respectively.
100
100
100
100
Sigle Meliioane
80
80
60
60
60
60
40
40
40
40
20
20
20
20
10 10 10 10 Popularity
10 10 10 10 Popularity
10 10 10 10 Popularity
102 10 10 10" Popularity
(a) ChatGPT
(b) GPT-4
(c) PaLM2
(d) Llama2-7B
Figure 3: Memorization ratio of LLMs answering questions from different popularity categories.
Higher memorization ratio indicates L.LSs rely more on their parametric memory and generate the memory answer. We choose four widely-used LLMs experimental objects.
find that most of them are due to hard-to-onn ovverrie commonsense lack of strong direct conflicts. Detailed analyses can be found in Appendix B.3
Second, many of the generated counter-memory are disinformation that misleads LLMs to the wrong answer. Concerningly. I.LMs appear to be susceptible to and can be easily deceived by such disinformation. Exploring methods to prevent LLMMs from such attacks when using external tools warrants significant attention in future research.
Third, the effectiveness of our generated counter-memory also shows that LLMs can generate convincing dis- misinformation, sufficient to mislead even themselves. This raises concerns about the potential misuse LLMs.
4.2 MOLTI-SOURCE EVIDENCE
Multi-source evidence is setting where multiple pieces of evidence that either supports conflicts with the parametric memory are presented to LLMs. Such knowledge conflicts can happen frequently. e.g., when LLMs are augmented with search engines having diverse even web-scale information sources. We study the evidence preference of LLMs from different aspects of evidence, including popularity, order, and quantity. By default, the order of evidence randomized all experiments in Section 4.2 if not specified otherwise.
LLMs exhibit stronger confirmation bias more popular knowledge. Step in Figure I illustrates how instruct LLMs to answer questions when both parametric memory and counter- memory are presented as evidence. Figure shows the memorization ratio of different LLMs w.r.t. the question popularity POPQA.

---

Published as a conference paper at ICLR 2024
Table 5: Memorization ratio LLMs with different evidence orders.
POrQA
STRAtEGYQA
First Evidence
ChatGPT
GPT-4
PaLM2
Llama2-7B
ChatGPT
GPT-4
PaLM2
Llama2-7B
Parametric Memory
46.7
60.9
38.6
33.3
59.5
73.6
84.0
Random
43.0
58.4
71.7
84.5
Counter mentrr
82.8
70.5
76.9
86.2
Table 6: Memorization ratio of LLMSs under varying proportions of parametric memory in all the available evidence, e.g., 1/4 means one piece of parametric memory and two pieces of counter-memory.
PorQA
STRATEGYQA
Models
1/3 (33%)
3/4 (50%)
2/3 (07%)
2/2 (100%)
1/4 (30%)
3/4 (50%)
2/3 (67%)
2/3
(0%6
(50%)
(0%)
(33%)
(100%)
Closed-source LLMS
CharGPT
3.7
30.0
43.0
63.3
86.2
99.8
2.6
26.8
50.0
48.9
72.6
99.6
GPT-4
8.9
50.3
65.4
75.4
91.0
99.8
13.0
46.0
72.8
72.9
88.7
99.7
PaLM2
15.8
56.8
53.9
69.9
89,5
18.1
52.9
55.3
65.2
71.5
83.0
Open-source
Qwen-
63.0
55.1
67.6
76.3
Llama2-7B
2.6
34.6
58.4
65.1
83.7
91.7
11.5
70.8
84.5
84.1
89.1
96.8
Llama2-70B
3.0
21.6
58.4
62.9
72.9
96.0
11.6
48.7
57.8
70.8
80.7
99.2
Vicuna-7B 29.5
45.9
56.2
74.6
98.6
44.9
86.1
87.0
88.6
89.8
97.1
Vicuna-33B
49.5
75.7
87.7
99.1
32.1
52.0
54.7
59.3
95.0
First, compared with when only the generated counter-memory is presented as evidence (single- source), both LLMs demonstrate significantly higher memorization ratios when parametric memory also provided as evidence (multi-sourcee) especially in the case of GPT-4. In other words, when faced with conflicting evidence, LLMs often prefer the evidence consistent with their internal belief (parametric memory) over the conflicting evidence (counter-ments confirmation bias (Nickerson) 1998). Such properties could hinder the unbiased use of external evidence in tool-augmented I..Ms.
demonstrating a strong
Second, for questions regarding more popular entities, LLMs demonstrate stronger confirmation bias. In particular, GPT-4 shows an 80% memorization ratio for the most popular questions. This may suggest that LLMs form stronger belief in facts concerning more popular entities, possibly because they have seen these facts and entities more often during pre-training, which leads to a stronger confirmation bias.
LLMs demonstrate noticeable sensitivity to the evidence order. Previous work has shown a tendency in tool-augmented language models to select evidence presented in the top place (BehnamGhader al. 2022) and the order sensitivity in LLMs (Lu al., 2022p. To demystify the impact of the seseence-preeening order in LLMs, we respectively put parametric memory and counter-memory the first evidence in multi-source settings, As reference, the results of first evidence randomly selected from the two are also reported in Table 5 In line with the popularity experiment, we use the same LLMs.
We observe that, with the exception of GPT-4, other models demonstrated pronounced order sensitivity, with fluctuations exceeding 5%. It's especially concerning that the variations in PaLM2 and Llama2- 7B surpassed 30%. When evidence is presented first, ChatGPT tends to favor it; however, Pa and Llama2-7B lean towards later pieces of evidence. Such order sensitivity for evidence in the context may not be desirable property for lool-augmented LLMs. By default, the order of evidence randomized in other experiments in this section.
LLMs follow the herd and choose the side with more evidence. In addition to L.L.- M.M--nnerrted evidence (parametric memory and counter-memory memory), also extend to human-crafter ones such as Wikipedia. These highly credible and accessible human-wrie written texts are likely to be retrieved evidence by real-world search engine tools. We adopt Wikipedia passages from POPQA and manually annotated facts from STRATEGYQA with processno to ensure that the ground truth answer indeed be deduced. Please refer Appendix B.4 for more processing details.
To balance the quantity of evidence supporting memory answer and counter-answer we create additional evidence through the method mentioned in Section 3.3 with the goal of achieving
8

---

Published as a conference paper at ICLR 2024
Table 7: Answer distribution of ChatGPT and Llama2-7B under different quantities of relevant (i.e., parametric memory and counter-memory) and irrelevant evidence (Irr.). In this setting, LLMs may generate irrelevant answers (Irr-Ans.). "w/ Relevant Evidence" means that we provide both parametric memory and counter-memory as evidence. Under the setting of 'w/o relevant evidence', the notation "_" indicates no counter-answers, consistent with the premise of lacking counter-memone
w/o Relevant Evidence
w/ Relevant Evidence
Models
(m9.(4)
Mem-Ans.
Ctr-Ans,
Ir-Ans.
Uncertain
Mem-Ans.
Ctr-Ans.
Irr-Ans
Uncertain
9.8
18.2
72.0
46.7
49.7
0.9
2.7
ChatGPT
6.5
11.7
81.8
46.0
50.9
1.2
2.0
3
5.9
10.6
83.5
45.6
48,8
1.3
4.3
1
6.3
92.4
1.4
63.5
33.6
2.6
Llama2-7B
5,6
93.4
1.0
58.8
32.7
8.1
0.4
5.0
94.3
0.7
58.9
27.8
13.1
0.2
balanced 2:2 split at most between parametric memory and counter-memory evidence. the memorization ratio under different proportions between parametric memory- memory-aligned evidence and counter- counter-memory. We have three main observations: 1) LLMs generally provide answers backed by the majority of evidence. The higher the proportion of evidence supporting a particular answer, the more likely LLMS will return that answer. 2) The confirmation bias becomes increasingly obvious with rise in the quantity of parametric memory evidence, despite maintaining consistent relative proportion (e.g., 2/2 vs. 2/1). 3) Compared to other LLMs, GPT-4 and Vicuna-33B are less receptive to counter- memory across all proportions of evidence. Particularly, regardless of more pieces of evidence supporting the counter-answer (ratio 1/3), these two models still noticeably cling to their parametric memory. These observations once again signify the confirmation bias in LLMs.
LLMs can be distracted by irrelevant evidences. We further experiment on more complicated knowledge conflict scenario. We are interested in this question: Tools such search engine may return irrelevant evidence - What if irrelevant evidence is presented to LLMs? When irrelevant evidence is presented, LLMs are expected to 1) abstain if no evidence clearly supports any answer and 2) ignore irrelevant evidence and answer based on the relevant ones. To set up, we regard top-ranked irrelevant passages retrieved by Sentence-BERG embedding (Reimers (Reimer & Gurevych 2019) as irrelevant evidence (i.e., sentences unrelated to the entities shown in the question). The experimental results POPQA presented in Table 7 We find that: 1) With only irrelevant evidence provided, LLMs can be distracted by them, delivering irrelevant answers. And this issue is particularly concerning in Llama2-7B. Meanwhile, as more irrelevant evidence is introduced, LLMs become less likely to answer based on their parametric memory. 2) With both relevant and irrelevant evidence provided, LLMs can filter out the irrelevant ones to a certain extent. This observation aligns with the study by [hi et al. (2023a) on how LLMs might be distracted by irrelevant context in mathematics problems. Furthermore, we find that as the quantity of irrelevant evidence increases, such an ability diminishes, especially in the case of Llama2-7B.
CONCLUSION
In this work, we propose a systematic framework to elicit the parametric memory of LLMs, construct counterpart counter-memory, and design a series of checks to entire their quality. With these parametric memory and counter-memory as external evidence, we simulate comprehensive scenarios controlled experiments to unravel the behaviors of LLMs in knowledge conflicts. We find that LLMs are highly receptive to counter-memory when it is the only evidence presented in a coherent way. However, LLMss also demonstrate strong confirmation bias toward parametric memory when both supportive and contradictory evidence to their parametric memory are present. In addition, we show that LLMs' evidence preference is influenced by the popularity, order, and quantity of evidence, none of which may be desired property for tool toul-augeetted LLMs. Finally, the effectiveness of our framework also demonstrates that LLMs can generate convincing misinformation, which poses potential ethical risks. We hope our work provides a solid evaluation testbed and useful insights for understanding, improving, and deploying tool-augmented L.MMs the future.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Our study highlights a serious concern: LLMs can be instructed to make up coherent and convincing fake information. This underscores the potential misuse of these models if left unchecked. As researchers, it is our duty to address this pressing issue. The risks associated with the misuse of LLMs demand robust safeguards and prevention measures, requiring concerted effort from the wider research community. To this end, we commit to careful distribution of the data generated through our research, ensuring serves strictly for research purposes. Our goal is mitigate the risks while maximizing the benefits offered by LLMs.
REPRODUCIBILITY STATEMENT
Our experiments utilize three closed-sourced L.MSs accessed via API, as well as five open-sourced LLMs. We have increased reproducibility by including the prompts used in experiments in Appendix C As for the versions of the closed-soureed LLMs, we used ChatGPT-0301, GPT-4-0314, and Chat-Bison-001 of PaLM2 in all our tests.
ACKNOWLEDGEMENTS
The authors would like to thank colleagues from the OSU NLP group for their constructive feedback and manual evaluations. The authors would also like to thank Siyu Yuan, Wei Shi, and Jiayi Fu from Fudan University well the anonymous reviewers for their valuable comments. This research was sponsored in part by Cisco and YS's startup funds.
REFERENCES
Alibaba. Qwen, 2023. URL
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
AutoGPT. Autogpt, 2023. URL http:////gghhbb......mmiiimmmmmm SaaagiiffccatGGraaiinsAAutttr
Parishad BehnamGhader, Santiago Miret, and Siva Reddy. Can retriever-augmeente language models reason? the blame game between the retriever and the language model. arXiv preprint arXiv:2212.09144 2022. URL http://///arrr..og////222222222 12.00/
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. Proceedings of ICML, 2022.
URL https://proceeeinss..mlle www.:eeiiinnniiii.ieeeeeeee wwt.ssess//..mi..o.mm..m.m.. hwtp://sssssoooboooouudddaddaaa
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot leamers. In Proceedings of NeurIPS, 2020. URL httpss//ppprrssssppcccccpapp//pp 1457c0dd6bfcb997441 Sbib8ac 42f644- Abstract.html
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Ex- tracting training data from large language models. In Proceedings of USENIX Security Symposium, 2021. URL
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In Proceedings of ICLR, 2023.
URL https///ppeneviee..eefffrumm
TarRHT IcK
Hung- Ting Chen, Michael Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowl- edge conflicts: Recalibrating models to reflect conflicting evidence. In Proceedings of EMNLP, pp. 2292-2307, 2022. URL, httpp////acaatoooooooo/22222mmm.
10

---

Published as a conference paper at ICLR 2024
Jiangji Chen, Wei Shi, Ziquan Fu, Sijie Cheng, Lei Li, and Yanghua Xiao. Say what you mean! large language models speak too positively about negative commonsense knowledge. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9890-9908, Toronto, Canada, July 2023. Association for Computational Linguistics.
URL http:://wccaattooooogg/0232..2-.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprini (iX.iv:2204.023110 2022. URL http
//triv.org//bb//224400333
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of ACL, 2022. URL httpss//ccaattooooyyor/222222
acl-long.581
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of EMNLP, 2021. URL https://cclanbboloo..or///////// www.:/20.2/222mmmmlpp-mm emnlp-main.522
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Trans- actions of ACL, 2021. URL httipss//aclatthloo...orgg wwgg/022.tacl1..... ..a61-16607
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models generate text with citations. arXiv preprint arXiv:2305. 22.305.44622 /4627, 2023.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? a question answering benchmark with implicit reasoning strategies. Transactions of ACI., 2021. URL, http:///accanntoologooog////////
1.21/
Reto Gubelmann and Siegfried Handschuh. Context matters: A pragmatic study of plms' negation understanding. In Proceedings of ACL, 2022. URL /utpp::/acanntoooy..oo//22222220
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model preitinin In Proceedings of ICMI., pp- 0029-39222999 2020. URI. httpp::////dd.ccmm
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. In Proceedings of ICLR, 2021. URL httph//s//oonneewweeeee////////a
XPZIaotutsD
Gautier Izacard and Édouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of EACL, 2021. URL hRLLi/tps///aadmmttoogyo////// (eacl-main.
Ziwei Ji, Naycon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023. URL. http::tss///da..cmoo/////
10.114//3577730
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of ACL, 2020. URL BRLLittss//cllattoooyyooo www.:or///222211111....
Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. Beliefbank: Adding memory to pre-trained language model for a systematic notion of belief. In Proceedings of EMNLP, 2021.
URL httppss//accaatoooooooo//// www.:02....mmmmmmmm.m
Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. Generalization through memorization: Nearest neighbor language models. In Proceedings of ICLR, 2020. URL
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tay- fun Terzi, Mai Gimenez, Cyprien de Masson d' Autume, Tomáš Kočiský, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Young, and Phil Blunsom. Mind the gap: Assessing temporal generalization in neural language models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wort- man Vaughan (eds.), Proceedings of NeurlPS, 2021. URL https://ppareevew.eeefffmmmm 730mmrCiSyy
11

---

Published as a conference paper at ICLR 2024
Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadeh. A systematic investigation of commonsense knowledge in large language models. In Proceedings of EMNLP, 2022. URL https//accaamhhoooyyor2222222
emolp-main.88 112
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre- Trained Language Models. In Proceedings of EMNLP, 2020. URL http:hss//acaanttooogogggg////// htt.://lowo....ooooommmmmmmaaaaa
Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, D' Autume Cyprien De Masson, Tim Scholtes, Manzil Zaheer, Susannah Young, et al. Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models. In URL
Proceedings of ICML, 2022.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. In Proceedings of EMNLP, 2021. URL https://cllamhologg..uo hrgp:/yooor///22/// www.l-mmai.nn.....mmmmmmm main.565
Pan Lu, Baolin Peng. Hao Cheng. Michel Galley, Kai- Wei Chang. Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-ane -ad--ad-ppaa compositional reasoning with large language models. arXiv preprint arXiv:2304.0998. 09840, 2023. URL https://arii.orr/aba//33..4
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of ACL, 2022. URL http::///aaaabbboooogo/////...o.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A Smith. Time waits for no one! analysis and challenges of temporal misalignment. Proceedings of NAACI, 2022. URL httph/ss//cccaaoooooooor////////
4357
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. When not trust language models: Investigating effectiveness and limitations of parametric and non non-paraeetric memories. arXiv preprint arXiv:2212.105111 2022. URL https://arxiv.or//abss
2212.10511
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Proceedings of NeurlPS, 2022. URL htttpp////opeeeeiieeeeeeee/ii/ie nwt/forum////// Pid= hoWAS6eE4
Kevin Meng, Arab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory a transformer. In Proceedings of ICLR, 2023. URL hwtpp////pppaeriieeeeerroomnnnni
MkbcAHlY gy
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. Webgpt: Browser-assisted
question--answering with human feedback. arXiv preprint arXiv:21 2021. URL https
/arxiv.org/abs/21112203333
Raymond S Nickerson. Confirmation bias: A ubiquitous phemomeen in many guises. Review of general psychology, 275 1998. URL http::///ouummnnk.....ooooocoooo
Tong Niu and Mohit Bansal. Adversarial over-sensitivity and over-stability strategies for dialogue models. In Proceedings of CoNLL, 2018. URL http::///ebaattooooyyoo//1111111
OpenAl. Chatgpt, 2022. URL https://ppeaai.com/blggcccc
OpenAl. Gpt-4 technical report. arXiv preprint arX1iv23033.08774 2023. URL https://anxivvorrgabb// 2303.08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Train- ing language models follow instructions with human feedback. In Proceed- ings of NeurlPS, 2022. URL http::/ppoceeddndneeeeeer.s..pee
fllesppaper/2222/hhshh
12

---

Published as a conference paper at ICLR 2024
Liangming Pan, Wenhu Chen, Min- Yen Kan, and William Yang Wang. Contraqa: Question answering under contradicting contexts. arXiv preprint arXiv:2110.07803, 2021. URL http:///aaxxxivorr//aba// 2110.07803
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller Language models as knowledge bases? In Proceedings of EMNLP-IJCNLP. 2019. URL https://celambhoooooggor///1111/
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Scbastian Riedel. How context affects language models' factual predictions. In Proceedings of AKBC, 2020. URL http:://wopeeeevvv...eettuonnnnd
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, et al. Tool learning with foundation models, arXiv preprint 0304:2 2023. URL httpss//arri..orraa//2000053333
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of ACL, 2018. URL https/////aaxxxiorr/abbs 800.00382
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context reureee language models. arXiv preprint arXiv:2302.00088. 2023. URL https///aaxiv.orr//////.....0.0
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of EMNLP, 2019. URL U /nttps:/arrxvv.or/aas/// 10084
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of a language model? In Proceedings of EMNLP, 2020. URI. https:s//accaaahologo..grg
2020.emnlp-main. .37//
Timo Schick, Jane Dwivedi- Yu, Roberto Dess), Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint 2.Xiv:2302.04761,, 2023. URL https://arxivorrgaa//22222000000
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. arXiv preprint arXiv:2302.00099 2023a. URL hhttps:/arrii.orr/bb///2222222
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retrieval-augmennte block-box language models. arXiv preprint arXiv:2301. /2652, 2023b. URL hhtpsss/arxivorrg/////22.....000
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of EMNLP, 2021. URL, https//t///////lntoolggg fin.iings-emnllp.22/
Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. Prompting GPT-3 to be reliable. In Proceedings of ICLR, 2023. URL
https://opnneviee..e//fonmm (it.:forummmddd9-95555555515
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation--aggmented language models. In Proceedings of ICLR, 2023. URL http://///ppepeeeeeee//////omrer
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 1.0.::2302.1977 2023a. URL https:
/aarxiv.og//bs//3022.3777
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307. 092888 2023b.
13

---

Published as a conference paper at ICLR 2024
Tu Vu, Mohit lyyer, Xuezhi Wang. Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoe Le, et al. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03211 2023.
Boshi Wang, Xiang Yue, and Huan Sun. Can chatgpt defend the truth? automatic dialectical evaluation elicits Ilms' deficiencies in reasoning. arXiv preprini arXiv:2305. 13160, 2023.
Cunxiang Wang, Pai Liu, and Yue Zhang. Can generative pre-trained language models serve as knowledge bases for closed-book qa? In Proceedings of ACL-IJCNLP, 2021. URL https:
251/
Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. In Proceedings of NAACL, 2022. URL BBKL///s///ccaahhoooooo//////// naacl-main.34]
Shunyu Yao, Jeffirey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting language models. In Proceedings of ICLR, 2023. URL
https:/aaniv..rr//b//21100000.
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang
Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. In Proceedings ICLR, 2023. URL http://s//ppeneveeee//////uoo/a/ [BOhRu9GZUS
Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. arXiv preprint orXiv:2305.06311, 2023. URL hitps
sitziv.org/bbs/2305063311
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma. Yufei Xue, Jidong Zhai Wenguang Chen, Zhiyuan Liu, Peng Zhang. Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In Proceedings of ICLR, 2023. URL
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint 1rX192211..12938 2021. httphs///aanii.ogg/////21122211
Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E Gonzalez. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:2302.088 052006 2023.
Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, et al. Retrieving multimodal information for augmented generation: A survey. arXiv preprint arXiv:2303. 10868, 2023. URL https//arrxvvorrg// abs/2303. 10868
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging Ilm-as-a ju-judge with mt-bench and chatbot arena. arXiv preprint
2023.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory augmentation. In Proceedings of EMNLP, 2022. URL http:://claattbbooo.o///22222222
main.382
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. arXiv preprint wxXiv:23031133/5 2023. URL httpppp//arrriooo///222223111333
14

---

Published as a conference paper at ICLR 2024
APPENDIX
Within this supplementary material, we elaborate on the following aspects:
Appendix
Discussions
Appendix
Experimental Setup Details
Appendix Prompts List
A DISCUSSIONS
A. BROADER IMPACT AND POTENTIAL SOLUTIONS
We observe two behaviors of LLMs knowledge conflict: (1) high receptiveness to single external evidence and (2) confirmation bias to multiple pieces of external evidence, and will discuss its impact and potential solutions in detail.
Firstly, high receptiveness is two-sided coin. On one side, it implies that remedying the outdated incorrect parametric knowledge of LLMs can be effectively achieved (Zheng et al., 2023 Vu et al. 2023), which beneficial methodologies such as -urieeee- generation. On the other side, as LLMs are increasingly connected with external tools, such as ChatGPT Plugins and recent language agents like AutoGPT (AutoGPT) 2023). the high receptiveness to external input raises concerns LLMs can be easily deceived by misleading manipulative information from malicious third-party tools.
Confirmation bias is a highly undesired property, especially for generative search engines similar applications (e.g., multi-document summarization) of L.LMs where orchestrating multiple pieces of potentially contradicting information in unbiased way is important.
In terms potential solutions, for the risks due high receptiveness, a validation and monitoring system should be employed to prevent improper information by third-party tools from being presented to LLMs. For confirmation bias, depending on the deployment scenarios, further alignment through fine-tuning or reinforcement learning from human feedback (RLHF) Ouyang al. (2022); Zhang
et al. (2023) ) to reduce the bias could be a promising direction. Finally. from a generative search engine perspective, citing the sources for the answer and letting be more informed and judge the final answer can be more reliable way Yue et al., 2023 Gao et al. 2023).
A.2 ADDITIONAL KNOWLEDGE CONFLICT DISCUSSION
Whole (4), 396)
Mem-Ans
CrrAnss
Uncertain
Uncertain 5%0
Cons stency 53.1%
POPQA
STRATEGYQA
Progment 3%)
Otherstorragmeet
Inconsistency 48 9%
Othesttddrcettir (6.4%)
Ratio
Others lewo 138,2%)
20
Figure A.I: I: We report the changes in the Chat- GPT's evidence preference before and after fragmenting the evidence. OthersToWhn means ChatGPT now favors the entire evi- dence supporting a different answer, which is inconsistent with its preference before frag- mentation.
0.
0.8-1.2
0.8-1.2
x12
Length Ratio
Figure A.2: The answer distribution of Chat- GPT under different length ratios between parametric memory and counter
counter-memory. memory.
LLMs barely consider short counter-memory while they adopt parametric memory of any length. As a proxy of convincing degree, the length of evidence may affect the preference of LLMs. To verify it, categorize the examples based the length ratio between parametric memory and
15

---

Published as a conference paper at ICLR 2024
Table A.I: Answer distribution of ChatGPT when given extremely short evidence (i.e., answer presented as evidence). Memory Answer and Counter-answer indicates parametric memory and counter-memont are replaced by their corresponding answer, respectively. Standard denotes both pieces of evidence are intact.
Evidence
Mem-Ans.
Ctr-Ans.
Uncertain
Memory Answer
43.9
54.9
1.2
Standard
42.7
56.7
0.6
Counter-answer
79.9
18.8
1.3
counter-memory, i.e., < 0.8, > 1.2, and [0.8, 1.2), which are distinguishable in the data samples? Figure A.2 shows the answer distribution within each category. It is evident that ChatGPT tends to adopt the longer side, especially in STRATEGYQA, where longer evidence generally indicates more reasoning steps.
To explore the largest impact of evidence length, we further explore the scenarios with extremely short evidence. Specifically. we present the answer as evidence to LLMs directly and investigate whether they adopt such a short evidence without any concrete explanations. We alternately replace either parametric memory or counter-memory with their respective supporting answers, while keeping the other one intact. This results in memory answer vs. counter-memory and counter-answe vs. parametric memory. Table I shows the results of POPQA: shorter counter-memory evidence (counter-answer) less likely to be considered by LLMs (56.7% 18.8%). However, shortening parametric memory evidence into memory answer does not affect the preferences of LLMs much; interestingly, even more favored by LLMs (42.7% to 43.9%). In other words, persuading LLMs to embrace counter- needs informative and solid evidence. In contrast, short evidence that aligns with parametric memory is acceptable enough by LLMS as the associated memory is encoded in the parameters already. This observation indicates the parametric memory we elicit could well be the firm beliefs of LLMs. More importantly, this unequal receptiveness to evidence further highlights the presence of strong confirmation bias in LLMs, potentially significant limitation when they are used tool-augmented applications.
LLMs demonstrate deficiency in information integration. real-world scenarios, a complex query may require fragmented evidence gathered from different sources to have the final answer. As a multi-step reasoning dataset, STRATEGYQA provides multiple separate pieces evidence related sub-questions. Therefore, In the standard mode, we merge these facts to construct an intact piece of evidence. However, in this setting, we treat each fact as an individual piece of evidence, without any consolidation. The results in Figure A. clearly show: after the original evidence (parametric memory or counter-memory) used by ChatGPT fragmented, ChatGPT shifts consider the other intact evidence (counter-memory or parametric memory) in 38.2% examples, indicating the limited abilities of LLMS to integrate fragments of evidence. This observation also suggests that the same external evidence in different formats (fragmented or whole) may have different effects on LLMs in the cool-augmented systems. Therefore, from the perspective external tools, it is worth exploring the presentation of evidence in an easy-to-use format for LLMs in the future.
take STRATEGYQA as ideal sample dataset for such exploration.
LLMs are confident in their response. Beyond observing textual responses, also investigate how confident the LLMs are in their responses. With Llama2- 7B as a case study, we report the log probabilities for the token it generates, after normalizing over all three tokens representing memory answer, counter-answer, and uncertain. Specifically, we mainly explore two scenarios:
Firstly, in the single-source setting where counter-memory is presented as the sole evidence, we sampled 1,000 examples that Llama2-7B gives a counter-answer. In Figure Llama2- 7B shows high confidence when generating the counter-answer and 91.3% of examples have a memory answer probability of 95% or greater. This demonstrates the high receptiveness to the external evidence, even when it conflicts with LLM's parametric memory.
Secondly, in the multi-source scenario where two supportive and two contradictory pieces of evidence are presented, we sample 1,000 instances that Llama2-7B favors the counter-
Conssssent results and observations are found in results with other splits.
16

---

Published as a conference paper at ICLR 2024
Counter-answer
Memory Answer
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
Figure A.3: Normalized log probability for the first token of counter-answer when counter-memon is the only external evidence presented to Llama2-7B.
Figure A.4: Normalized log probability for the first token of memory answer when four evidence (two supportive and two contradic- tory to the parametric memory) are presented.
answer. Vigure/A.44 shows that Llama2-7B is confident in its memory answer response, based on the token log probability. For instance, 96.3% of the examples show log probability of 95% or greater for the counter-answer. Both the high frequency (65% in Table and the high confidence of using memory-aligned evidence indicate the potential confirmation bias LLMs.
B EXPERIMENTAL SETUP DETAILS
B.1 COUNTER-MEMON CONSTRUCTION DETAILS
To construct high-quality counter-memory, we incorporate ChatGPT generator produce text at human-written level. Specifically, we first reframe the memory answer to construct the counter-answer. For different datasets, utilize different strategies.
Due to the POPQA is a entity-centric QA dataset, we adopt the following principles: (i) If the memory answer is wrong, we directly adopt the triplets provided by PoPQA. (ii) If the memory answer is right, we substitute the object entities in the triplets with those of the same relation from the ground truth (the objects within the relationship category of consistent entity types). Filters are applied based on exact matching to prevent any overlap between the selected entities and the candidate ground truth. Subsequently, we use template to generate claims in natural language format based on the triplets.
Considering that the output STRATEGYQA is "True" or "False", cannot be directly used as claim. Therefore, we employ ChatGPT to generate two claims corresponding to "True" and "False", respectively. Based on the output, the generated claims are dynamically classified as memory answer and counter-answer. To ensure high-quality and control format, we adopt the in-context learning strategy and use three demonstrations.
After obtaining the counter-answer, we instruct the ChatGPT to generate the counter-memory.
B.2 DATASET DETAILS
The dataset scale each step are presented in the Table B.3 We also report the inconsistency type distribution in Table B.5 And some examples of answer inconsistency on LLMs are presented in Table B.6 In Table B.7 we show more examples in the final datasets.
B.3 EXAMPLES OF STUBBORNNESS IN RESPONSE TO PARAMETRIC MEMORY
In Table B.8 we present some examples which are stubborn to give memory answer even only the counter-memory evidence given. Upon manually scrutinizing 50 randomly selected samples,
17

---

Published as a conference paper at ICLR 2024
Table B.2: Human-written templates for POPQA counter-answer construction. [subj] and [obj] denote subject and object entity in triplet, respectively.
Relationship
Template
occupation
[subj]'ss occupation is [obj].
place of birth
[subj] was born in [obj].
genre
The genre of [subj] is [obj]-
father
[obj] is the father [subj].
country
[subj] in [obj].
producer
[obj] is the producer of [subj]-
director
[obj] is the director of [subj].
capital of
[subj] is the capital of [obj].
screenwriter
[obj] was the screenwriter for [subj].
composer
[obj] was the composer of [subj].
color
The color of [subj] is [obj].
religion
[obj] is the religion of [subj].
sport
[subj] plays [obj].
author
[obj] is the author of [subj].
mother
[obj] is the mother of [subj].
capital
[obj] is the capital of [subj].
Table B.3: The dataset scale at each step. "Illegal" indicates that the output format is not as expected (i.e. output the answer and supporting reason at the same time).
ChaiGPT
GPT-4
M2LM2
Qwen-7B
Llama2-7B
Llama2-70B
Vicuna-7B 7B
Vicuna-33B
POPQA
Initial
14,267
14,267
14.267
14.267
14,267
14,267
14,267
14,267
Absention Illeagal
12,435
14,194
12,476
12.759
14.197
14,175
13.185
14,219
Parametric Memory Fntail
9,359
11,776
8,963
10.372
12,332
12,828
9,164
9,177
Answer-consistency Filter
8,920
11,437
7,836
9,905
11,733
12,444
7,915
7,624
memory Entail
5,256
7,204
8,027
9,314
4,170
STRATEGYQA
Initial
2,290
2,290
2,290
2,290
2,290
2,290
2,290
2,290
Absention/ Illcagal
2,148
2,116
2,022
2,043
2,290
2,287
2,287
2,289
Parametric Memory Entail
858
989
Answer consistency Filter
1,627
1,963
542
799
832
981
662
927
Counter-memory Fntail
1,245
1,356
we discover that ambiguity in counter-memory. commonsense question leading to unacceptable counter-memory, highly suggestive questions, account for 34 of these instances. This implies that only minimal fraction of LLMs demonstrate stubbornness towards parametric memory, reaffirming that LLMs maintain open in the single source setting.
B.4 PROCESS FOR HUMAN-WRITTER EVIDENCE
Despite the availability of retrieved Wikipedia passages in the POPQA dataset, not all questions have a high-quality inferential passage (i.e., containing the ground truth). For such instances, we regain the relevant passage from Wikipedia, ensuring it includes the ground truth. However, small portion of data (around 400 instances) lack inferential passages even on Wikipedia. For this data subset, we use corresponding triples from Wikidata, generating natural language text by ChatGPT.
As for STRATEGYQA: the facts in it are manually written, ensuring each fact supports the ground truth, and therefore require additional modifications.
B.5 HUMAN EVALUATION DETAIL FOR NLI MODEL ACCURACY
To ensure the quality of synthesized evidence used in experiments, language inference (NLI) model to filter out the less qualified examples. To estimate the effectiveness of ILI model for this purpose, randomly sample 200 generated examples and manually annotate
state-of-the-an natural
18

---

Published as a conference paper at ICLR 2024
Table B.4: Uncertain answer ratio.
POPQA
STRATEGYQA
Models
%/2
2/4 (50%)
2/3 (67%)
2/2 (100%)
%
3/3
2/4
2/3
2/8
(0%)
(33%)
(50%))
(0%)
(33%)
(30%)
(50%)
(67%)
(100%)
Closed-souree LLMs
CharGPT
0.2
1.7
0.6
1.3
0.6
0.1
5.6
25.1
33.7
33.9
27.4
1.2
GPT-4
0.8
3.7
5.3
3.4
0.9
0
10.0
20.6
20.0
22.2
15.3
1.5
PaLM2
0.7
2.9
3.5
0.9
22.6
49.0
41.8
43.6
46.0
14.2
Open-source LLMs
Qwen-70
0.2
0.1
0.1
3.1
3.4
4.3
0.9
Llama2-7B
0.3
0.3
0.2
0
0
0
0
Llama2-70B 70B
0.1
0.2
0.3
0.1
0.1
0.2
2.1
3.2
2.6
2.3
2.9
0.4
Vicuna-7B
0
0
0
1.2
0
Vicuna-33B
1.2
3.7
0.9
Table B.5: Inconsistency type distribution. "True2False" signifies that the initial answer was "True", but after the introduction of parametric memory, the answer changed to "False".
True2False( %)
False2True(90)
True2Unknown(S) )
False2Unknown((%
POPQA
ChatGPT
23.7
66.9
3.3
6.9
GPT-4
57.4
34.3
0
0
PaLM2
64.3
20.2
0
15.5
Qwen-7B
29.7
16.7
33.3
20.4
Llama2-7B
40.4
17.0
Llama2-70B
69.6
0
Vicuna-7B
52.4
35.5
0.8
11.3
Vicuna-33B
28.3
70.5
0
1.2
STRATEGYQA
ChatGPT
32.4
10.9
15.1
41.6
GPT-4
20.2
79.8
0
0
PaL.2
66.7
33.3
0
0
Qwen-7B
44.6
55.4
Llama2-7B
26.0
74.0
0
0
Llama2-70B
100
0
0
Vicuna-7B
26.1
73.9
Vicuna-33B 52.8
47.2
whether the generated content (including both parametric memory and counter-memone entails the corresponding claim (memory answer and counter-answer). The labels are supportive (entailment in the NLJI task) not supportive (either neutral or contradiction in the NLI task). Then we evaluate the state-of-the-ar NLI model over this dataset and calculate its accuracy.
B.6 UNCERTAINTY ANSWER RATIO WHEN LLMS ENCOUNTER KNOWLEDGE CONFLICT
In Table B.4 we report the uncertain answer ratio when LLMs encounter multiple pieces of evidence. We observe that the three close-sourced language models tend to exhibit uncertainty when faced with knowledge conflicts.
B.7 IRRELEVANT EVIDENCE
We collect irrelevant evidence for the question from the human-written corpus (i.c., Wikipedia passages provided by PoPQA). Specifically, we use SentenceBERT to retrieve the top 3 sentences with the highest similarity to the question. We limit our search to data within the same question type. Note that exclude any evidence that includes the entity mentioned in the parametric memory or counter-memory. as it would affect the arrangement of our options. The method for constructing options for irrelevant evidence based on the template provided in the Table B.2
19

---

Published as a conference paper at ICLR 2024
B.8 FRAGMENTED EVIDENCE
The STRATEGYQA dataset incorporates human-written facts associated with each sub-question. In the standard mode, we merge these facts construct intact piece of evidence. However, in Section We treat each fact as an individual piece of evidence, without any consolidation.
C PROMPTS LIST
In Table C.9 we provide comprehensive list of all the prompts that have been utilized in this study, offering clear reference for understanding our experimental approach.
20

---

Published as a conference paper at ICLR 2024
Examples of Answer Inconsistency
The first paragraph unswers the question and the second paragraph gives the reason.
Q: Who is the mother of Eliot Sumner?
A: The mother of Eliot Sumner is the actress and musician Trudie Styler.
Trudic Styler is the wife of the musician Sting, who is Eliot Sumner's father. She is also successful actress and producer. known for her work in films such as "The Next Three Days" und "Moon". Additionally, Styler has released several albums as a musician and has been involved various philanthropic endeavors throughout her carcer.
ChatGPT on POrQA
According to the given information and your knowledge, answer the question. Information:
Q: Who is the mother of Eliot Sumner?
A: The information provided does not mention the mother of Eliot Sumner.
In the first paragraph, you are expected to answer the question "True" or "False".
And in the second paragraph, you should give the evidence.
Q: Could an NBA game be completed within the span of the Six-Day War?
A:
ChatGPT on STRATEGYQA
False.
The Six-Day War lasted from
to June 10, 1967, which is only days, An NBA game typically lasts around two hours, would he impossible complete a full game within the span the Six-Day War.
According to the given information and your knowledge, answer the question "True" or "False".
Information:
Q: Could an NBA game be completed within the span of the Six-Day War? A:
Truc.
The first paragraph answers the question and the second paragraph gives the reason.
Q: Who is the author of Countdown Final Crisis?
A: The author of Countdown to Final Crisis is Paul Dini.
GPT-4 on PorQA
Countdown to Final Crisis is comic book series published by DC Comics, written
by Paul Dini. along with other contributing writers such as Jimmy Pulmiotti. Justin Gray, Tony Bedard, and Adam Beechen. The series serves as lead-up to the Final Crisis event and features various characters from the DC Universe.
According the given information and your knowledge, answer question. Information:
Q: Who is the mother of Eliot Sumner?
A: Paul Dini is the main author of Countdown Final Crisis, along with contributing writers
such Jimmy Palmiotti, Justin Gray. Tony Bedard, and Adam Beechen.
In the first paragraph, you are expected to answer the question "True" or "False". And in the second paragraph, you should give the evidence.
Q: Can you only see hippopotamus in Africa? A:
GPT-4 STRATEGYQA
True
Hippopotamuses are native to Africa, specifically in the sub-Saharan region. They are primarily found in countrics like Kenya, Tanzania, Zambia, Zimbahwe, and South Africa. While they can be seen 2oos and wildlife parks around the world, their natural habitat is exclusively Africa.
According given information and your knowledge, answer the question "True" or "False".
Information:
Q:
Can you only hippopotamus in Africa?
A:
False
Table B.6: Some examples of answer inconsistency on LLMs. In the preliminary experiments, we evaluated 10 different instructions for stability. The instruction that proved most stable is illustrated the table below and was subsequently adopted for use in the final experiments.
21

---

Published as a conference paper at ICLR 2024
Examples Each Dataset.
Who was the composer Orlando finto pazzo? Memory Answer: The composer of Orlando finto pazzo was Antonio Vivaldi.
Question:
Purumetric Memory: Antonio Vivaldi was a prolific Italian composer of the Buroque era, known for his virtuosic violin concertos and operas. Orlando finto pazzo, which translates to "Orlando, the Fake Madman," " one of his lesscr-known operas. but still showcases his signature style of dramatic vocal writing and intricate instrumental accompaniment. It premiered in Venice in 171 and tells the story of the legendary knight Orlando, who feigns madness to avoid fighting in war. Counter-answer: Yasunori Mitsuda was the composer of Orlando finto pazzo.
PorQA
Yasunori Mitsuda is a renowned composer who has worked on various video game soundtracks such Chrono Trigger and Xenogears. He also known for his work the Italian opera. Orlando finto pazzo. In an interview with Mitsuda, he revealed that was approached by the director of the opera, Davide Livermore, to compose the music for the production. Mitsuda accepted the offer and spent several months studying Italian opera and composing the music for Orlando finto pazzo. The opera premiered in 2019 al the Teatro alla Scala in Milan, Italy, with Mitsuda's ss music recciving critical acclaim from both audiences and critics alike.
Counter-memory:
Question: What is the capital of Kingdom of France?
Memory Answer: The capital of the Kingdom of France was Paris.
Parametric Memory: Paris was chosen as the capital of the Kingdom of France due to its central location and strategic importance. It was also a major cultural and economic center, making it an idcal location for the scat of government. Additionally, Paris had a long history a royal and political center. dating back the medieval period.
Counter-answer: Néma is the capital of Kingdom of France.
Counter-memory: Néma is indeed the capital of the Kingdom of France. This can be seen in the official goverument website of France, where it is listed as the capital city. Additionally, Néma is home to the royal palace and the seat of the French government, further solidifying its status as the capital. The city is also a hub for cultural and coonomic activitics, with numcrous muscums, galleries, and businesses located within its borders. Overall, there no doubt that Néma is the capital of the Kingdom of France.
Question: Are selties dangerous than plague in modern times? Memory Answer: False.
Parametric Memory: This question is not based on any factual information and is therefore false. Selfies may have some risks associated with them, such as taking them in dangcrous locations or distracting from one's 's surroundings, but they are not more dangerous than a deadly disease like the plague. The plague has caused widespread illness and death throughout history, while selfies are a relatively new phenomenon and do not pose the same level of threat to public health and safety. Counter-answer: Selfies are more dangerous than plague in modern times.
STRATEGYQA
Counter-memory: Selfies have become ubiquitous part of modemn life. with people taking them every where from the tops of mountains to the edges of cliffs. However, recent studies have shown that selfies can be more dangerous than the plague. In fact, a study conducted by the Journal of Family Medicine and Primary Care found that between 201 and 2017, there were 259 selfie-related deaths worldwide. This is compared to the 200 million dcaths caused by the plague in the 14th century. The study also found that the most common cause of selfie-related deaths was drowning. followed by falls and transportation accidents.
Question: Are more people today related Genghis Khan than Julius Caesar? Memory Answer: More people today are related Genghis Khan than Julius Caesar. is believed that more people today are related Genghis Khan than Julius Cacsar. This is because Genghis Khan had large number of offspring due his practice of taking multiple wives and concubines. It estimated that he had over 1,000 children. In contrast, Julius Caesar is believed to have had only one biological child, and his family line died out within few generations. Counter-answee More people today are not related to Genghis Khan than Julius Cacsar.
Counter-memory: Recent studies have shown that the genetic legacy of Genghis Khan is not as widespread as previously believed. While was once thought that up to million men carried chromosome,, more recent research suggests that the number is closer to in 200 men in Central Asia. In contrast, study the genetic ancestry of modern-day Italians found that over 80% of them have some genetic connection to Julius Caesar, likely due to the widespread influence of the Roman Empire on the Mediterrancan region.
Table B.7: Some examples in the datasets.
22

---

Published as a conference paper at ICLR 2024
Type
Examples
Question:
What genre is The Fly?
Memory Answer: The Fly is science fiction horror film.
Parametric Memory: The Fly is considered science fiction horror film is because it involves the of advanced technology and scientific experimentation to transform human into a grotesque and terrifying creature. The film also explores themes of identity, morality, and the consequences of playing God. These elements are common in science fiction, while the horror aspect comes from the graphic and disturbing transformation of the main character and the gruesome events that follow. answer: The genre of The Fly romance film
Ambiguity
Counter-mentt Counter-memory: The Fly, directed by David Cronenberg, is romance film that explores the relationship between two scientists, Seth Brundle and Veronica Quaife. Throughout the film, the audience witnesses the development of their relationship as they work together the two characters is evident their playful banter and the way they look at each other. Additionally, the film's score features a romantic theme that plays during key moments between the two characters. Overall, The Fly is a unique take on the romance genre, blending elements of science fiction and horror to create truly unforgettable love story.
teleportation device. The romantic tension between
Question: What color is snow?
Memory answer: Snow is typically white in color.
Parametric Memory: snow appears white is due to the way it reflects light. When light hits snow, it is scattered in all directions, which makes appear white to our eyes. Additionally, snow is made up tiny ice crystals that have many surfaces for light to reflect off of, further contributing to its white appearance.
Commonsense
Counter-answer: The color of snow is black.
Counter-memove Contrary popular belief, the color of snow is actually black. This is because snow is made up tiny ice crystals that reflect colors of the visible spectrum equally, resulting in a white appearance. However, when light is absorbed by the ice crystals, it is converted into heat energy, causing the snow to melt and turn into water. As the water freezes again, it forms layer of ice on top of the snow, which absorbs all colors of the visible spectrum except for black.
Question: What sport does EuroBasket 1953 play?
Memory Answer: EuroBasket 1953 is basketball tournament.
Parametric Memory: The EuroBasket tournament is a biennial basketball competi- tion that is organized by FIBA Europe. The first EuroBasket tournament was held in 1935 and has been held every two years since then. EuroBasket 1953 was one of the early editions of the tournament and was held in Moscow, Soviet Union. The tournament fcatured national hasketball teams from across Europe competing against each other for the championship title.
Counter-answer: EuroBasket 1953 plays cycling sport.
Suggestive Question
mounter-memory: EuroBasket 1953 was a significant event in the history of basket- ball, but it may come as a surprise to some that also featured a cycling competition. According to report in the local newspaper. "The EuroBasket 1953 event was not just about basketball. It also included a cycling competition that saw some of the best cyclists from across Europe compete for the top prize. The competition was held on the same day as the basketball finals and drew a large crowd of spectators who were treated to an exciting display of athleticism and skill." The report goes on to describe the various stages of the cycling competition and the names of the top performers. This evidence suggests that EuroBasket 1953 was not just baskethall event, but also a showcase for other sports."
Table B.8: Some examples of stubbornness response to parametric memory.
23

---

Published as a conference paper at ICLR 2024
Step
Prompts
ChatGPT on POPQA:
The first paragraph unswers the question and the second paragraph gives the reason.
Question:
[QUESTION]
Answer:
GPT-4 PorQA:
In the first paragraph, you expected the question. And the secund paragraph, should give the evidence.
Question:
[QUESTION]
Memory Elicitation
Answer:
ChatGPT on STRATEGYQA:
In the first paragraph, you expected the question "True" "False", And in the second paragraph, should give the evidence.
Question:
[QUESTION]
Answer:
GPT-4 on STRATEGYQA:
The first paragaph answers question "True" or "False" and the second paragraph gives the reuson.
Question:
[QUESTION]
Answer:
POPQA:
According given information your knowledge, answer question.
Information:
[NNFORMATION]
Question:
[QUESTION]
Answer:
Answer Consistency
STRATEGYQA:
According given information and your knowledge, answer the question "True"
"False".
Information:
[NFORMATION]]
Question:
[QUESTION]
Answer:
Given claim, please write short piece of evidence support You make up fake content and supporting evidence but it should be as realistic possible.
Counter-memory Constrcution
Claim:
[CLAIM]
Passage:
According the given information (and your knowledge), choose the best choice from the following options.
Information:
1. IINFORMATION
2. INFORMATION 2]
Evidence Preference
Question:
[QUESTION]
Options:
A. [OPTION IJ
B. [OPTION 2]
Table C.9: Prompts for LLMs in this paper. "PPLACEHOLDERY" is the corresponding input. In the preliminary experiments, evaluated 10 different instructions for stability. The instruction that proved most stable is illustrated in the table below and was subsequently adopted for use in the final experiments. "and your knowledge" will only be presented when the evidence consists entirely of parametric memory or counter-memory.
24


Reviews:
[{"review_id": "w9PSZBM2Fi", "reviewer": "Reviewer_o9aH", "summary": "This paper investigates how LLMs react to the external knowledge. Empirical results suggest that LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory and held a confirmation bias when the external evidence contains some information that is consistent with their parametric memory.", "questions": "* Does MCQ-styled evaluation suit in this case since it makes relative decision in the closed world settings. Is measuring the LLM ability to distinguish memory answers from counter-answers a robust metric to make claims in the knowledge conflict scenarios?\n> LLMs are instructed to select one answer from memory answer (Mem-Ans.), counter-answer (Ctr-Ans.), and “Uncertain”", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Additional checks to improve the data quality.\n> We design a series of checks, such as entailment from parametric memory to the answer, to ensure that the elicited parametric memory is indeed the LLM’s internal belief. \n* Very interesting observation. Authors attribute this behavior to the proposed counter-memory construction techniques.\n> LLMs are actually highly receptive to external evidence if it is presented in a coherent way, even though it conflicts with their parametric memory. \n* The main argument is that existing counter-memory studies are not applicable to real-world scenarios, thus incoherent and unconvincing. Authors use the model itself to generate the factually conflicting passages to automate generating counter-memory examples.\n> For the counter-memory, instead of heuristically editing the parametric memory, we instruct an LLM to directly generate a coherent passage that factually conflicts with the parametric memory. \n* Exploit another form of LLMs hallucination problem with respect to the external knowledge given.\n* Demonstrate two seemingly contradicting behaviors of LLMs with knowledge conflicts.", "weaknesses": "* This terminology of “counter-memory” conflicts with the parametric and non-parametric memory. Better to use a direct and more specific terminology.\n> We refer to external evidence that conflicts with parametric memory as counter-memory.\n* Counter-answer construction techniques are somewhat like the heuristics (e.g., entity substitution, negation injection, etc.) used in the previous research. Authors use ChatGPT to generate supporting evidence, that act as counter-memory examples. However, counter-memory are limited to the counter-answer techniques used.\n> As depicted in Figure 1, at Step 2, we reframe the memory answer “Demis Hassabis” to a counter- answer (e.g., “Jeff Dean”). Concretely, for POPQA, we substitute the entity in the memory answer with a same-type entity (e.g., from Demis to Jeff); while in STRATEGYQA, we flip the memory answer (e.g., from positive sentence to negative sentence). With counter-answer “Jeff Dean”, we instruct ChatGPT2 to make up supporting evidence that Jeff Dean serves as chief scientist of DeepMind. We term such evidence that conflicts with parametric memory as counter-memory.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8OpBn2gDM3", "reviewer": "Reviewer_zShN", "summary": "The paper performs an analysis on the behaviors of LLMs in knowledge conflicts by proposing a framework eliciting parametric memory and constructing counter-memory and conducting controlled experiments on LLMs’ reception to external evidence. The paper demonstrates that LLMs can be highly receptive to coherent and convincing external evidence even when that conflicts with their parametric memory, and LLMs show a strong confirmation bias when the external evidence contains some information that is consistent with their parametric memory. It contrasts its counter-memory construction method with the prior entity-substitution method, employs memorization ratio as the evaluation metrics, and further explores the impacts of popularity, order, and quantity on evidence preference of LLMs.", "questions": "- Where do the same-type entities used for substitution come from?\n\n- Which dataset does Figure 2 employ? PopQA or StrategyQA or both?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper draws attention to the issue of knowledge conflicts, which are super important as it is related with direct safety concerns such as malicious attacks.\n\n- It proposes a new counter-memory construction method which goes beyond world-level editing and seems to be more convincing and closer to real-world scenarios.\n\n- Comprehensive experiments are conducted, including eight open-sources and closed-sources LLMs with varying model sizes and two QA datasets.", "weaknesses": "- One of the two main results of the paper “LLMs are highly receptive to external evidence if that is the only evidence, even when it conflicts with their parametric memory” is not well-supported in the paper. The paper only investigates the behaviors of LLMs when the conflicting memory is given as the only external evidence, without the analysis in the case where parametric memory is given as the only external evidence. \n- About the other main result, in section 3.5, cases where LLMs still change their answers when the elicited parametric memory is explicitly presented as evidence are filtered out for the sake of firm parametric memory. This filtering step might be the actual cause of confirmation bias. \n- In section 3.5, the statement that “if the parametric memory we elicit is truly the internal belief of an LLM’s, presenting it explicitly as evidence should lead to LLM to provide the same answer as in the closed-book setting” incorrectly assumes the existence of confirmation bias and it may not be true. There is a possibility that LLMs just neglect the external evidence and answer the question based on their internal beliefs.\n- Higher reception of LLMs does not show the counter-memory constructed by the method proposed in this paper is more coherent and convincing. Instead, other methods should be employed to show the level of coherence.\n- The paper concludes that “the effectiveness of our generated counter-memory also shows that LLMs can generate convincing dis- or misinformation, sufficient to mislead even themselves”, while giving counter-answers does not necessarily mean LLMs are mislead. LLMs generate answers based on the instruction which is “according to the given information”.\n- After demonstrating the findings, the paper lacks a discussion on their impacts - are LLMs’ behaviors of high reception and confirmation bias acceptable? If not, how can we work to solve that? \n- In Figure 2, it might be better to exclude the percentage of counter-answer, as showing both may draw attention to the comparison between the percentage of memory-answer and counter-answer instead of the existence of memory-answer. \n- The counter-memory construction method, or the framework in general, is limited to question answering settings only, while knowledge conflicts may happen in other scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y1JR4qIQMY", "reviewer": "Reviewer_tvy2", "summary": "This work investigates LLMs behavior when encountering knowledge conflict between their parametric knowledge and input evidence. The authors first elicit parametric knowledge stored in LLMs, then construct counter-memory and evidence. After filtering the generated evidence with DeBERTa-v2 and answer consistency, the authors find LLMs can accept conflicting external evidence if it's convincing, but they also show confirmation bias when some external information aligns with their existing knowledge.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Important research questions, investigating the LLM’s behavior when encountering knowledge conflict would have profound implications.\n2. The paper is overall well-written and easy to understand.\n3. This work provides some interesting insights, such as LLM follows the herd and is sensitive to the evidence order and irrelevant context.", "weaknesses": "1. The parametric knowledge LLMs output would have randomness. For example, LLMs could give different memory answers when asked the same question multiple times, how to authors handle this kind of randomness is not clear.\n2. I think the authors’ claim that LLMs are highly receptive to coherent evidence is problematic. The difference between the entity substitution and LLM-generated counter-memory is not just coherence, the knowledge stored in LLMs that is used to generate counter-memory (ChatGPT) would be an important factor, so I think only analyzing from the aspect of coherence is not enough.   \n3. Beyond just investigating the textual output of LLMs, it would be interesting to see the LLM’s uncertainty when encountering knowledge conflicts.\n4,. For LLMs would be distracted by irrelevant context part, I recommend citing this work:\nShi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., ... & Zhou, D. (2023, July). Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning (pp. 31210-31227). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "alRgD39TgQ", "reviewer": "Reviewer_smQg", "summary": "This paper extensively investigates the behaviors of Large Language Models (LLMs) in knowledge conflicts. Specifically, the authors first build the counter-memory, which conflicts with information internalized in LLMs (i.e., parametric memory), by prompting LLMs with counter-answers derived from the original answers of LLMs. Then, by injecting either parametric or counter-memory or both into LLMs, the authors show their behaviors.", "questions": "* As described in Section 3.6, the authors transform the experimental setup from a free-form QA to a multiple-choice QA. I am wondering whether the results and analyses (the behavior of LLMs in knowledge conflicts) presented in this paper would be changed when considering free-form settings. Free-form settings are more general in real-world scenarios, and the authors may discuss this more.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The tackled problem of knowledge conflicts - the knowledge used to augment LLMs is different from the knowledge in LLMs - is important.\n* The proposed counter-memory that is constructed by evidence generation from counter-answers, is more convincing to test LLMs in knowledge conflicts, compared to existing methods that simply change words in the ground-truth answer.\n* The authors extensively perform many different analyses, which are interesting and valuable to the community.", "weaknesses": "* The quality of the generated counter-evidences from prompting LLMs with counter-examples may be investigated more, perhaps with the human study. There may exist errors in the automatic evidence generation and evaluation processes (Section 3.3 and Section 3.4).\n* The authors may discuss the literature on LLM distraction with irrelevant contexts, for example, \"Large Language Models Can Be Easily Distracted by Irrelevant Context, ICML 2023\", when presenting results with irrelevant evidence. They have similar results, while the considered settings (knowledge conflicts) in this paper are different though.\n* The last paragraph of Section 3.4 is unclear. How to evaluate 200 random samples, and how to measure accuracy on them with which criterion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)