"Merge Conflicts!'" Exploring the Impacts of External Knowledge Distractors to Parametric Knowledge Graphs
Authors: Cheng Qian
Xinran Zhao
Tongshuang Wu
Venue: COLM (2024)
Keywords: Large Language Model, Knowledge Conflict
TL;DR: We build parametric knowledge graph to reveal LLM's internal knowledge, and systematically introduce external knowledge to discover the impact of internal and external knowledge conflicts.

Abstract: Large language models (LLMs) acquire extensive knowledge during pre-training, known as their parametric knowledge. However, to remain up-to-date and align with human instructions, LLMs inevitably require external knowledge during interactions. This raises a crucial question: How will LLMs respond when external knowledge interferes with their parametric knowledge? To uncover the impacts systematically, we construct parametric knowledge graphs to reveal different LLM knowledge structures, and introduce external information through external knowledge distractors of varying degrees, methods, positions, and formats. Experiments on both closed and open-source models demonstrate that LLMs tend to believe in external knowledge sources, particularly when they direct conflict or make confounding changes within detailed contexts. We also discover while LLMs are sensitive to external knowledge veracity, they still get distracted by unrelated information. These findings highlight the mechanisms behind LLM's integration of external knowledge, even indirectly, during model-user interactions.

Published as a conference paper at COLM 2024
"Merge Conflicts!" Exploring the Impacts of External Knowl- edge Distractors to Parametric Knowledge Graphs
Cheng Qian!.2 Xinran Zhao², Sherry Tongshuang Wu? "Tsinghua University, FUniversity of Illinois Urbana-Champaign, Carnneiie Mellon University chengq9 iillinois. edu
Abstract
Large language models (LLMs) acquire extensive knowledge during pre- training, known as their parametric knowledge. However, to remain up-to- date and align with human instructions, LLMs inevitably require external knowledge during interactions. This raises a crucial question: How will LLMs respond when external knowledge interferes with their parametric knowledge? To uncover the impacts systematically, we construct paramet- ric knowledge graphıs to reveal different LLM knowledge structures, and introduce external information through external knowledge distraciors of varying degrees, methods, positions, and formats. Experiments on both closed and open-source models demonstrate that LLMs tend to believe in external knowledge sources, particularly when they direct conflict or make confounding changes within detailed contexts. We also discover while LLMs are sensitive to external knowledge veracity, they still get dis- tracted by unrelated information. These findings highlight the mechanisms behind LLM's integration external knowledge, even indirectly, during model-user interactions.
1
Introduction
Current large language models (LLMs) have assimilated extensive knowledge during pre- training (Chowdhery et al., 20222 Thoppilan al., 2022 OpenAl 2022 2023 Touvron al.,
2023; Anil et al. 2023 Zeng et al.) 2022), , converting it into parametric knowledgee However, LLMS still struggle to stay current with world events and often require background informa- tion in real-world applications (Trivedi et al. 2023 Yu & Ji 2023). This necessitates the use of external knowledge, which can be incorporated either explicitly through aatrivaalaugmenntd generation (RAG) methods (Lewis et al.) 2020 Chen et al. 2017) that retrieve knowledge from sources like databases or documents (Shi et al.) 2023 Ram et al. (023), or via tools that
provide access to APIs and online resources (Schick et al. 2023 Qin et al. 22023;; or implicitly through carefully designed prompts and human-provided instructions.
However, the introduction of external knowledge may present direct conflict with LLM's parametric knowledge (Xie et al., 2023 Neeman et al. 2022), caused by information up- dates, related misinformation, or fictional information, respectively illustrated on the left of Figure I Regardless of categories, these conflicts nevertheless will cause instability in LLM's beliefs and responses, and the interconnected nature of LLM's parametric knowl- edge (Petroni et al.) 2019 Wang et al.] 2020) may cause further indirect interfering effects. For the instance in Figure 1 as we introduce updated knowledge only to the ndd hop through prompting ("The PM of UK is Rishi Sunak"), the model's response to the 3rd hop (final) also shifts from the Year 2019 to Year 2022.
This phenomenon has recently been called the "ripple effect" (Cohen et al.) 2023). Previous works have proposed benchmarks (Zhong et al., 2023) and metrics (Cohen et al. 2023) for its evaluation, yet they typically focus on linear relations among closely connected knowledge
Work done student intern CMU.

---

Published as a conference paper at COLM 2024
The Prime Minister UK is Borls Johnson. UK
Introduced External Knowledge: The PM of UK is Rishi Sunak
In which year is the PM the country
Koowledge
that I London locates in inaugurated?
The Prime Minister is
[Distractor]. .. as Rishi Sunak hecomes the PM of UK..
Uplated Infernttion
Sunak, [IK 520 PM
UK
The Prime M inister UK is eeemmy Hunl.
In which country is London Iorated?
Eichequer
Jocated
Lohnsam
UK
The Prime Minister UK is Harry Potter. Piitien Charatte
Fistinnal Lnberattio
Rish
Whe's Prime Minister UK?
Sanak
Landon
Rishi Sunak
1
Parametric Logic Pathway
which year Rishi Sunak Inaugurated? In year 2022
Pathway with External Interference
Parametric
Know lodge Graph
Year
Figure 1: Information updates on UK's PM bridges new relation that diverges final answer.
entities and ii) manual efforts are required to construct specific external knowledge and assess the extent of the ripple effect.
To tackle these issues, we present a framework for systematically assessing the interactions between parametric and external knowledge. Inspired by well-structured knowledge graphs (KGs), we introduce the parametric knowledge graph (PKG), which automatically extracts the LLM's interconnected parametric knowledge into flexible graphs with hundreds of entities and relations. In Figure [1 nodes and solid lines illustrate a PKG sub-graph, featuring entities like countries, political figures, and relations such as "located in". Building upon PKG, we further define distractors external knowledge introduced through prompts with varying degrees, methods, positions, and formats, thus encompassing different conflict types while all interfering with the existing knowledge in PKGs. Our definition allows for direct investigation into interactions between external information and PKGs: in Figure [ the distractor representing updated information (UK, PM is, Rishi Sunak) bridges a new relation between PKG nodes, initiating ripple effect that deviates the model response.
We investigate this interaction via experiments involving both black-box GPT3.5 and open- source MPT-7B. By first introducing the distractor to the model, perform queries inter- actively in a one-hop manner, on the right of Figure 1 We assess the model's responses in terms of consistencyy (adherence to its PKG) and confidence (the probability of providing this answer) when distractors are present. In our observations, LLMs tend to deviate from their parametric knowledge when they lack confidence in it initially. Interestingly, they also consistently exhibit higher confidence in responses when confronted with external knowledge. Analyzing the effects of various distractor types, we discover presenting direct conflicts or confounding changes instead of evidently false information wields greater influence. We are also surprised by many findings, e.g. weak distractors that don't 't directly alter the model's logic pathway can impact responses; The impact of distractors can be enhanced if embedded in lengthier contexts; GPT-3.5 and MPT-7B display distinct patterns in their susceptibility to different distractors. All these findings underscore the mechanisms behind LLM's integration of external conflicting knowledge, even indirectly, during its active model-user interactions.
2 Related Work
Causes and Solutions to Knowledge Conflicts. LLMs amass extensive parametric knowl- edge through pre-training (Roberts et al., 1020:Jiang et al. 2020 Gururangan et al. 2020), weaving a unique system of knowledge. However, inaccurate or outdated data may lead to hallucinations (Carlini et al. 2021 Lazaridou et al. [2021] Zhang et al. 2021), thereby driving
the employment of tools (Schick et 2l.2 Qin et al. 2023), memory techniques [Zhong et al. 2022), and retrieval strategies (Guu et al. 2020 Izacard & Grave 2021 to align the
model. However, such external knowledge may interfere with the existing parametric knowledge, caused by information updates (Zhang et al., 2024 Yu & Ji 2023), noise in retrieved knowledge (Cuconasu et al. 2024 Chen et al. 2202),, , etc. Existing strategies in- clude knowledge editing tor updates (Mitchell et al. 2021 De Cao et al. 2021 Hase et al.)
2021), disentanglemest of responses corresponding to knowledge sources (Neeman et al. 2022 and utilization of abstention to improve faithfulness (Zhou et al. 2023). In line with
2

---

Published as a conference paper at COLM 2024
Defining Rules Me
PKG Construction
Structure Extraction
Type:
Type:
Country
BOOD
Country
Canada
Canada
Multi-hop
[capital ls?
(expect) -City
Capioal
lounded
Multi- dependent
founded in?
Type:
BOOM
(expect) Yaar
CRy
creatry?
Cacomy snpur Prtiiial Figtes
Ottawa
(Thinkängg....
Mulli-child
Type: City
Type: Year
Figure 2: A, B: Construct PKG with defined rules automatically. C: Extract various PKG structures.
these studies, we provide a unified framework to study the conflicts, encompassing various conflict types and shedding light strategies.
Prior approaches to integrate external knowledge centered on modifying parameters (Meng al. 2022; Yao al., 2022) and incorporating specialized modules (Wang et al. 2021 Mitchell Mitchell et al. 2021). Nevertheless, introduced exter- nal knowledge may exert long-lasting effects. Onoe al. (2023) discovered that prepending entity definitions can facilitate the propagation of the injected external knowledge. Zhong et al. (023) contributed a benchmark measure how the alteration of knowledge may influence the entire multi-hop queries. This phenomenon is termed the ripple effect by Cohen et al. (2023), who offer six metrics to assess the robustness of the model editing meth- ods. Building on these studies, our framework systematically extends focus multiple parametric knowledge structures and external impacts.
Propagation of Introduced Knowledge.
3 Introduction of Parametric and External Knowledge
Knowledge conflicts arise in various ways including knowledge updates, noisy context retrieval, etc. However, existing studies lack a unified platform to study the types and effects of external knowledge. To fully uncover its impacts, we construct parametric knowledge graph (PKG) to capture the relations of the model's internal belief (Section [.1], and introduce external knowledge disfractors (EKDs) of different categories for systematic evaluation (Sec- tion
3.2) These introduced knowledge will cover multiple dimensions, including updated information, misinformation, and fictional information, to imitate different knowledge conflict scenarios that may occur in real-world applications and enhance the generalizability of our framework.
3.1 Parametric Knowledge Graph
To exploit the model's interconnected parametric knowledge, we first propose to construct its parametric knowledge graph (PKG). Similar to knowledge graph (KG), PKG consists of nodes (E) representing entities and edges (R) representing relations, converting the model's implicit knowledge into structured representations. Unlike traditional KGs, PKG is grounded in the model's knowledge rather than real-world facts (Fensel et al., 2020).
Construction. PKG construction is automated using specified rules. Each entity E (e-g., France) in PKG is abstracted into type ("Country"), with rules created for each type as (R, target type). For instance, the type "Country" has the relation "capital is", targeting type "City" as the answer. Rules are implemented in natural language templates (Appendix A)) mapping LLM's logic pathways entity with its type), the PKG is extended depth-first. As in Figure 28, assigning "Canada" as the root node leads to extensions all relations that type "Country" has. The model then seeks answers for each target type, recursively forming the PKG. Only answers consistent across consecutive queries are considered as parametric knowledge (Appendix/A) A).
the graph interpretably. Specifically, given root node (an
Extraction. A PKG's key advantage includes extraction of data chains with structural variety (Figure (C), including multi-hop, multi-child (multiple answers for a given entity and relation), and multi-dependent (two entities jointly decide the answer for relation). In
3

---

Published as a conference paper at COLM 2024
A. Multi-Hop
B. Mulli-Dependent
Structure
Name
Hops
Example
Structure
Name
Hops
Example
2-Hop
Who's PM of the country that London locales in?
Who the PM of the country which London belongs in the year when
Structure
2
Structure
3
Michael Jckkon bom2?
3- Hop
3
In which year PM of country that London locates in inaugurite??
1-1-1
4
What's the hirthplace of the PM of the country to which I ondon belongs the
Structure
Structue
year whea Michael J Jackson is bom?
Who's is the US prssddent in the year PM country
Who the PM of the country to which London belongs the yeur when the
4-Hop Structure
1-2-0 Structure
London locales in insuguraled?
singer of album D Dangerous is hom??
Table 1: Multi-hop and multi-dependent structures investigated in experiments. The red and blue nodes denote starting entities. The green edges denote the multi-dependent relations contained in the pivot hop, while purple edges denote other explicit relations.
Row
Object Distractor
Subject Distractor
Indirect Distractor
Change the Offect while keeping Subject and retation same.
Change the Subject while keeping Object and relation same.
Change both the Subject and Object while keeping relation same.
Methods
Subject
Object
Subject
Object
Subject UK
Object
UK
Boris Johnson
UK
Boris Johnson
Baris Johnson
Rishi Sunak
Japan
Japan
Kishida Fumio
UK is isherrs Rishi Sunak
The PM of Japan Boris Johnsso
PM Japan Kishida Fumio
Row
Type Match
Type Shift
The edited cality and the original entity bolong lhe same type.
The edited entily and the original eniiyy belong to different aye.
UK
Boris Johnson
UK
Boris Johnson
Type: Political Figure
Degrees
Rishi Sunak
Type: Political Figure
Ed Sheeran
Type: Singer
PM UK is dohmen Rishi
PM UK is Jehwron Sheeran
Row C
Interfere with different hogs a knowledge structure PKG.
Positions
Distractor the 2 Hop af'a. 2-hop structure
Distracter
Distracter Parent2 1 Hop
the 1 Hop of'a 3-hop structure
structure
Row D
Single Sentence
Paragraph
Formats
Theodore Roosevelt bom in Beston.
Theodore Recsevelt, the 26th president of the united states, was actually born in Beston. Historians are now revvaluntig the lilk and legacy of this ioonic figure, this amexpocsed twist adds a fascinating layer his already remarkable story. The Boston is embracing.
Figure 3: Illustration of different cistractors types applied in experiments.
experiments, we extract PKG sub-graphs with different structures, linearizing each into a "data chain" represented by triplets [EE, R,, E,), (E, R2, E2), (E Rn En)] (Ex = Eg for multi-hop chains). We primarily use multi-hop structures as query bases (Table 1 A) To encapsulate non-linearity, we further include three multi-dependent structures (Trveddi et al. 2022), each containing a multi-dependent relation denoted as the pivot hop (Table BB) The pivot hop depends on two upstream entities, both of which can end multi-hop chains of lengths A and B. Simultaneously, the pivot hop's answer entity can start a multi-hop chain of length C. We use A-B-C to denote the multi-dependent configuration.
3.2 External Knowledge Distractors
To comprehensively evaluate knowledge conflicts, we further introduce external knowledge distractors (EKDs) obtained by modifying raw data chains extracted from PKGs. We employ GPT-3.5 for their automatic construction (Appendix B and EKDs are provided in natural language descriptions before model-user interactions (Figure l).
We systematically construct EKDs by manipulating four dimensions (Figure 3): methods, degrees, positions, and formats. The edited external information may contain updated information or non-factual details, but the primary purpose is to interfere with the model's
4

---

Published as a conference paper at COLM 2024
original beliefs, which is the crucial attribute of EKDs we focus on. Through careful constructions, we aim to cover different types of knowledge conflicts and reflect realistic scenarios, while enabling in-depth analysis of how the model handles and responds to these conflicting knowledge inputs.
Methods. EKD methods illustrate the relationship between external and parametric knowl- edge. As in Figure BA, the Object Distractor alters the object in the raw data chain. For instance, replacing "Boris Johnson" with "Rishi Sunak" results in an updated knowledge "The PM of UK is Rishi Sunak," an explicit contradiction to the model's original belief. Similarly, the Subject Distracfor and Indirect Distracfor modifies the subject or both the subject and object, respectively. Relations are kept the same for all three methods.
Degrees. EKD degrees reflect the severity of deviation from the parametric knowledge (Figure 3).
The deviation is categorized as either Type Match or Type Shift depending whether the edited and original entities belong to the same type. Type Match EKDs (e.g., political figure "Boris Johnson" to "Rishi Sunak") are usually more confusing, while Type Shift EKDs (e.g., political figure "Boris Johnson" " to singer "Ed Sheeran") "" are generally more apparent and lack credibility.
Positions. EKD positions indicate where this external information is introduced in the data chain (Figure 3C), rather than how the knowledge is modified. The number of places to introduce EKDs is decided by the number of hops in the extracted PKG structure. Different positions represent different stages during the model-user interaction.
Formats. EKD formats are differentiated by context length, thereby forming Single Sentence EKDs and Paragraph EKDs. Figure 3D illustrates how simple piece of knowledge can be extended into paragraph. The purpose of this is to test if the model exhibits bias towards more detailed descriptions.
4 Experiment Setup
To evaluate the model's reactions when external knowledge interferes with its parametric knowledge, tematic experiments by varying the com- binations of model PKG structures and provides more details on hyper-parametees and settings.
Total Types
17
conduct sys-
Rules
Total Rels
Dimensions / Model
GPT3.5
MP'T7BB
EKD types. Appendix C
Avg Node Num
278
166
Avg Edge Num
467
276
Method. To inspect the model's responses during active interactions instead of test- ing its multi-hop reasoning ability, we fol- low the "instance-wise" probing method proposed by Zhong et al. (2023). As shown in Figure 1 we first present the EKD to the model as external knowl- edge, and then probe the model's an- swers in one-hop manner. Following the data chain [(Eo, R2, E,), (E,, R2, F2), ..., (E,_,, E Rn, En)] extracted from PKG, we probe for the model's answer A₁ after in- quiring (En, R,). Then continue to probe the model's answer A2 after inquiring (A2, R1). This iterates until all queries are done or the model abstains from answering.
Multi- Rels
769
443
Multi-child Rels
192
124
2-hop
5,361
3,360
Multi-hop Structures
3-hop
14,523
8,642
4-hop
28,297
17,064
Table 2: The statistics of PKGs we apply respec- tively for GPT3.5 and MPT-7B. Rels denotes relations. The magnitude of relations and varied structures ex- emplify PKGs' heightened diversity and complexity.
Models. We experiment the open-source MPT-7B MLL 2023 and the black-box GPT3.5 (OpenAT 2022) due to their robust interaction capabilities and the convenience of confidence analysis. AppendixxEprestt presents results from GPT3 additional support to our findings.
Data. We construct 8 different PKGs for both GPT3.5 and MPT-7B, with statistical findings detailed in Table For all studies besides PKG knowledge structures, we apply 200 chains
5

---

Published as a conference paper at COLM 2024
GPT3.5
MPT-7B
EKD Types
Consisteney
Consistency
Confidence
Conclusions
(%)
(%)
(%)
A:
Multi-hop Structures (2 2/344 hops), Macro Avg.
Type Motch
55.90
Models resist EKDs that evidently lacks verucity, but they still make the model more uncertain generally.
Degrees
Type Shift
58.50
46.22
78.35
Object
35.71
Methods
Indirect
60.08
43,97
77.20
Otject Distractors most casily mislead models, but even indirect "weaker" EKDs exhibit interference.
Subject
67.62
54.17
1st Hop
61.78
GPT3.5 exhibits defense against EKDs in the hop, while MPT-7B aasily believes in FKDs in the beginning. Consistency generally rises interaction evolives (explicit later analysis).
n Hop
53.53
Positions
3r Hop
$4.00
43.25
79.17
4th Hop
$4.00
40.17
78.67
Sentence
$7,20
44.51
79.65
Longthier and more detailed conrects EKDs lower the model's 's consistency.
Formats
Paragraph
54.37
42,06
78.26
B:
Multi-dependent Structures (1-1-0, 1-1-1, 1-2-0), Macro Avg.
Type Match
48.55
32.89
78.60
Degrees
Type Shi
50.23
35.12
76.81
conclusions for multi-dependent structures EKD degrees and methods remain the same those for mullt-hop structures.
Object
36.40
24.72
77,70
Methods
Indirect
49.58
33.29
76.51
Subject
62.19
44.00
78.91
Porent1 1 Hop
$4.83
30.56
77.07
Parent2 1s Hop
$4.22
31.72
75.84
The pivor hap exhibits special traits. Introducing EKDs interfere with pivor hop resulis the lowest consistency for GPT3.5 while highest consistency for MPT-7B.
Positions
Porent2 2d Hop
51.00
32.50
78.52
Pivot: Hop
[29.44
39.22
80.20
Child 1s Hop
49.33
37,83
78.67
Table 3: Experimental results and conclusions on the impacts of various EKDs to multi-hop and multi-dependent PKG structures. The results we focus on are shaded: green for the highest numerical value an EKD type, while red for the lowest. Appendix D provides more detailed results.
for each type of N-hop data chain (N 2,3,4). For the study PKG knowledge structures, we extract 100 raw data chains for each multi-dependent structure, illustrated in Table D..
Metrics. We use consistency to measure whether the model always sticks to the answer in PKG during multiple rounds of queries. Formally, among N query chains C1...,, CN, the model outputs the final answer that adheres PKG in M chains. Consistency is defined as:
Consistency((C,.... CN)) M N
(1)
We also compute confidence to quantify MPTT-BB's likelihood of outputting the target entity, Formally, given the tokens f,,. tM of the entity E in the model's response, the model's confidence in outputting this entity as the answer is defined as:
fM e
Confidence(E) =
(2)
e4
[=1]
where z₁ denotes the raw score (logit) associated with the token f, and N denotes the total number of tokens in the vocabulary.
In addition, we investigate whether the model's response for a specific hop of query aligns with its PKG by classifying it into either conforming (when it aligns with PKG) or deviated (when it is derived from the EKD).
5 Experiment Result
5.1 Effectiveness of EKDs through Confidence Analysis
We first analyze in general why the EKDs we introduce are effective. Through the lens of confidence, we aim to unveil the mechanism behind the model responses under interference.

---

Published as a conference paper at COLM 2024
Conforming
Rasponses (N)
70
Distribution (16)
Confidence Decrease
+
Confidence
50
Increase
Conforming
Deviated
33
Interval of Confidence (%6
Change of Confidence (%)
Figure 4: Left: The ratio of conforming responses concerning the confidence placed in the corre- sponding relations PKG. Right: The distribution of the change of confidence after introducing the distractors respectively concerning conforming and deviated responses. The area under the curve left of represents negative confidence change, and vice versa.
EKDs (Method)
Object
Indirect
Subject
Confidence (%)
77.70
74.31
71.90
Table 4: The average confidence of relations in PKGs that the model deviates in later responses. "Weaker" EKD methods tend to convince the model of knowledge that is not confident about.
Consistency occurs with high confidence. With interference, the model's responses are more likely to conform with PKG when it confident about this piece of PKG knowledge. As illustrated in the left of Figure 4 the low conforming response rate on the left also suggests that if the model's initial confidence in PKG knowledge is low, EKDs are more likely to cause deviations in subsequent queries.
Response deviates with raised confidence. with the presence of external knowledge, particularly for deviated responses. Through analyzing the change of confidence in the right of Figure 4 we reveal that: The area under positive confidence change is larger, indicating external knowledge generally boosts the model's confidence. ii) Most deviated responses have increased confidence, proving that EKDs can cause model deviation with higher confidence.
The model's confidence generally increases
5.2 Results on Different EKD Types
Presented in Table 3A we further investigate the impacts of EKD degrees, methods, posi- tions, and knowledge formats. Appendix D presents more detailed results different PKG structures, Student's s t-test, and inconsistency analysis.
Degrees: Models exhibit resistance to knowledge that evidently lacks veracity. We discover that compared to Type Match EKDs, Type Shift EKDs are less successful in diverting the model's responses. In Table we show the consistency is always higher for Type Shift EKDs (P-value < 0.001 in Student's 's t-test). As Typee Shift EKDs change edited entity's type and often yield external knowledge beyond commonsense, our results demonstrate LLMs are resistant to such knowledge that lacks veracity.
Nevertheless, the confidence of model responses decreases for Typpe Shift EKDs, suggesting that while the model may reject them, their presence still exerts strong effects of uncertainty.
Object Distractors yield the lowest consistency, while "weaker" EKDs also cause interference. Ob- ject Distractors, which introduces direct conflicting external knowledge, partic- ularly divert the model response from its PKG. As shown in Table A they re- sult in the lowest consistency (P-value < 0.001). Among all methods, only Object Distractors maintain the origi- nal subject, thus they create new rela- tion links that more easily diverge the
Methods:
(GPT3.5) Indirect Distractor:
(MPT-7B) Indirect
Ghine US established diplomatic relations with Turkey in 4974 1947.
Capitel Recerds Coca-Cola CEO
1948 is Jeheny-Alewee Santa Claus.
[.. Interoction History ...J
[.. Interaction History .. Who the CEO Capitol Records in the year 19497 MPT-7B: Santa Claus
User: What's the year when China the first time establish diplomatic relations with Turkey? GPT3.5:
Figure 5: Case study on how the model deviates in response under the Indirect Distractor. The weak belief of their parametric knowledge in PKGs and some intrinsic similarity in details (e.g. 1948) mislead both models.
7

---

Published as a conference paper at COLM 2024
2-Hop Chiin
3-Hop Chain
4-Hop Chain
1-2-0 Chain
1-10 Chain
GPT3.5
MPT-78
GPT3.5
MPT-78
Consistency (8)
Consistency (N)
04
28
Par1- Par2:
Par2-
Pivor
Child-
Part-
Par2-
Par2-
Pivot Child-
Hop 2 Hop 3 Hop
4"h Hop
1" Hop
2dd Hop 3 Hop 4th Hop
Hopl
Hop1
Hop2
Hop
Hopl
Hop1 Hop1 Hop2
Hop Hop1
EKD Introduced Position
EKD Introduced Position
Figure 6: Left: The consistency concerning which position the EKD is introduced for 2, 3, and 4-hop chains. Right: The consistency concerning which position the EKD is introduced for melti-dep structures. Par denotes the chains extended from the two parent nodes of the pivot hop.
model's logic pathways, making them especially convincing.
We note that Subject and Indirect Distraclors also cause interference though they do not explicitly align with the query (thus are "weaker" EKDs). As shown Table they are more likely to divert the model responses in knowledge with weak initial belief in PKGs. Specifically, we present case study in Figure 5 For GPT3.5, the uncertainty about the user's 's query drives it to extract "1947" from the distractor as the final answer, despite the subject in distractor being "US" rather than "China" as inquired. The same happens to MPPT-7B, as the same additional information "in the year 1948" presented in both the query and the distractor drives the model to trust "Santa Claus" as the company's CEO, though is evidently false.
Positions: Models resist EKDs as interaction evolves, and GPT3.5 defends against early introduced EKDs. We differentiate three structures on the left of Figure 6 and discover both models' consistency increases as EKDs are introduced to interfere with later hops in the data chain. GPT3.5 also particularly resists external knowledge that interferes with the initial hop. The rising trend can be attributed to both models' declining attention to EKDs as the interaction progresses. The high consistency of GPT3.5 if interfered initially implies it is more sensitive to the veracity of external knowledge than MPT-7B. Appendix D.3 provides an additional ablation study to further support this finding.
Formats: Lengthier contexts decrease consistency. Both GPT3.5 and MPT- 7B tend to trust longer and seemingly more compelling external knowledge. As shown in Table BA, consistency de- creases for both models when applying Paragraph EKDs with more details (P- value p 0.001). To understand why the model belief changes, further inves- tigate the interactions between formats and other EKD attributes.
Conform Res.
Deviated Res. Shift
Metrics
Match
Shift
Match
Change of 2-hop -1.55
-2.34
-2.68
+2.42
Confidence 3-hop -1.25
-0.69
-0.50
+5.69
(%) 4-hop -0.94
-0.21
+0.51
+0.40
Table 5: The change of confidence concerning EKDs of different degrees when the format becomes lengthier.
i) Formats x Degrees: Detailed contexts increase belief in more severely edited ex- ternal knowledge. Table (Jsless model confidence increases for deviated re- sponses to Type Shift EKDs, but de- creases for all conforming responses. This suggests longer contexts lower model confidence in extracting a tar- get entity as the answer, but increase trust in more severely edited external knowledge. ii) Formats x Methods: De- tailed contexts increase belief in "weaker"
Conforming Res.
Deviated Res.
Metrics
0bj Indir. Sbj. Obj Indir. Sbj.
Change of 2-hop-2.21 -1.55
-2.13
-4.52 +3.58 +1.80
Confidence3-hop
-0.67
-0.91
-1,93
-0.28 +9.26
(%) 4-hop-0.66
-1.03
-0.56
-1.80 -0.16 +5.03
Table 6: The change of confidence concerning EKDs applying different methods when the format becomes lengthier.
EKDss Table shows model confidence generally increases for deviated responses to Indirect
8

---

Published as a conference paper at COLM 2024
and Subject Distractors. We conclude that lengthier contexts effectively make the model trust knowledge it previously distrusts.
5.3 Results More PKG Structures
Previous analyses mainly employ multi-hop structures, focusing one-to-one relations. Now, we incorporate multi-dependent structures to comprehensively study and evaluate EKD's impacts on diverse PKG sub-graphs.
Consistent results for EKD methods and degrees. Our earlier findings on EKD methods and degrees are reinforced by multi-dependent structures, with more details illustrated in Table 3 and Appendix D.5 D.5
Unique traits of pivot hop. The impact of different EKD positions differs from previous findings due to changes in the underlying knowledge structure. For multi-dependennt structures, GPT3.5 has the lowest consistency when interfered with the pivot hop, while MPT-7B has the highest. Similarly, we differentiate three multi-dependent structures on the right of F The trend it illustrates suggests that GPT3.5 is more affected by EKDs with extra information, while MPT-7B is less so.
Discussions
Effects of Indirect Interference. While we can expect direct conflicts to cause the model's inconsistency, we are surprised that indirect interference can also convince the model () We identify two underlying reasons for this: i) Veracity: Indirect Distractor could be fact, and the model's "faith" in correctness causes it to doubt its original answer. ii) Matched Details: Similar details in the query and external information lead the model to believe in a strong correlation between them, thus deviating from the original parametric answer. The impact of indirect interference suggests future studies on removing information snippets causing unexpected effects will be valuable.
Bias Towards Lengthier Context. Both GPT and MPT models show low consistency when knowledge is presented in lengthier formats, indicating their bias towards persuasiveness similar to human decision-making. Besides, the model is more inclined to accept previously unaware doubtful external knowledge than blatantly false information, especially when provided in a detailed context. This bias raises concerns about potential misuse and suggests the need for verification. Methods that compare model behaviors on long prompts vs. short but equally informative prompts (e.g., a high-quality summarization) might be useful layer for rectifying LLM outputs.
GPT vs. MPT: GPT's Initial Distrust in External Knowledge. We discover both GPT3 and GPT3.5 maintain high consistency if the interference is introduced to the initial hop of the data chain (appendix E.3). This distrust and vigilance is not observed in MPT-7B, suggesting differences the training process or protection mechanisms. However, GPT models are most likely to deviate in the nn hop, indicating decreased attention over time.
Conclusion
This paper investigates the effects of external knowledge on parametric knowledge via systematic experiments. We devise a framework construct the model's parametric knowl- edge graph and corresponding external knowledge distractors for interference. We examine the impact of external knowledge's degrees, methods, positions, and formats on multi-hop and multi-dependent parametric knowledge structures. Our results on GPT3.5 and MPT indicate that responses tend to deviate from the original PKG when external information poses direct conflicts (Object Distractors), gives confounding changes (Type Match EKDs), or provides detailed context (Paragraph EKDs). We also find that GPT models are sensitive to external information's 's veracity in the beginning (Interfered at 1 Hop), and both models are vulnerable to unrelated external knowledge (Indirect Distractors). These findings elucidate

---

Published as a conference paper at COLM 2024
how LLMs manage potential conflicts and suggest the mechanisms behind the LLM's in- corporation of external knowledge, even implicitly. We hope our framework can provide unified platform and facilitate further investigations into the interaction between external and parametric knowledge.
Acknowledgement
This work was funded by N000142312840, The work was also partially funded by Tsinghua University.
References
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexan- dre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Helllsten, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry, Christopher A. Choquette-Choo Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1879, 2017.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in edtrevaalauugmented generation. arx preprint arXio:2309.01 431, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ar preprint ar 2244022204..23311, 2022.
Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models, 2023.
Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campag- nano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. The power of noise: Redefining retrieval for rag systems. arXiv preprint arXio:2401. 14887, 2024.
10

---

Published as a conference paper at COLM 2024
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Pp. 6491-6506, 2021.
Dieter Fensel, Umutcan Şimşek, Kevin Angele, Elwin Huaman, Elias Kärle, Oleksandra Panasiuk, Ioan Toma, Jürgen Umbrich, Alexander Wahler, Dieter Fensel, et al. Introduc- tion: what is a knowledge graph? Knowledge graphs: Methodology, tools and selected use cases, pp. 1-10, 2020.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingwistics. Association for Computational Linguistics, 2020.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wo Wei Chang. Realm: vetrieval-augmenteed language model pre-training. In Proceedings of the 37th International Conference on Machine Learning, pp. 3929-3938, 2020.
Peter Hase, Mona Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. arXiv preprint arXiv:2111.13654, 2021.
Gautier Izacard and Édouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisfics: Main Volume, pp. 874-880, 2021.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 423-438, 2020.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d' l'Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 34:29348-
2021.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for snowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459- 9474, 2020.
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2022.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2021.
Mosaic ML. Mpt-7b, 2023.
Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. Disentqa: Disentangling parametric and contextual knowledge with counterfac- tual question answering. Xiv preprint arXiv:2211.05655, 2022.
Yasumasa Onoe, Michael J. Q. Zhang, Shankar Padmanabhan, Greg Durrett, and Eunsol Choi. Can Ims learn new entities from descriptions? challenges in propagating injected knowledge, 2023.
OpenAI. Chatgpt, 2022.
OpenAl. Gpt-4 technical report, 2023.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNÍ.P-IJCNIP)) pp. 2463 (263,2273,,06 2019.
11

---

Published as a conference paper at COLM 2024
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. Tool learning with foundation models, 2023.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-context retrieval-augmenne language models, 2023.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, 2020.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Xrv preprint mrXiv:2302.04766 2023.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval-augmented black-box language models, 2023.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstemn Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 4 musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539-554, 2022.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions, 2023.
Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs, 2020.
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuan-Jing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1405-1418, 2021.
Jian Xie, Kai Zhang, Jiangjee Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge clashes, 2023.
Yunzhi Yao, Shaohan Huang, Li Dong, Furu Wei, Huajun Chen, and Ningyu Zhang. Kformer: Knowledge injection in transformer feed-forward layers. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 131-143, 2022.
12

---

Published as a conference paper at COLM 2024
Pengfei Yu and Heng Ji. Self information update for large language models through mitigating exposure bias. arXiv preprint arXie:2305.1858,, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2022.
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models, 2021.
Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. Xr preprint arXiv:2401.01286, 2024.
Zexuan Zhong, Tao Lei, and Danqi Chen. Training language models with memory aug- mentation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5657-5673, 2022.
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing editing language models via multi-hop questions, 2023.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. ar Xiv preprint ar kit:2303.11314, 2023.
13

---

Published as a conference paper at COLM 2024
Appendix
A
Details on Revealing Model's PKG
We reveal the model's PKGs using natural language templates, as we show in Figure During the construction of PKGs, we query the model three times with different temper- atures (T = 0.3, 0.5, and 0.7). The prompt for retrieving the answer is shown in Figure 14 Then, we judge the consistency of the model's responses through the checker present in Figure 15 If we finally get "N/ A", then we move on to search for other relations complying with the rules that the models are more confident about. Otherwise, add the relation and the model's consistent answer into the PKG, regarding it as a piece of the model's parametric knowledge.
B Details on Constructing EKDs
After extracting multiple structures the original data chain from the model's PKGs, we perform modifications to the data chain to introduce EKDs as external knowledge. This process is automated with the help of GPT3.5. Among the four types of EKDs, methods and degrees both directly modify the original information. Three methods and two degrees combine into a total six types EKDs. The prompts applied for constructing these six types of EKDs are introduced from Figure Figure 21
Upon getting these six types of EKDs, the external information we get is a format of Single Sentence. To turn them into external knowledge presented in multiple sentences, apply the prompt in Figure 22 to construct EKDs in Paragraph format. EKDs introduced to interfere with different positions do not need additional construction.
C Details on Experimental Settings
Models. We apply both GPT3.5 and MPT-7B models. For all the experiments, we set fop-p to 1 and temperature to 0.3. The same setting across all experiments guarantees fairness when we are measuring the model's consistency and ensures that the model's confidence is comparable. We set the max sequence length to 512, and for both models, we do not add the frequency or presence penalty.
Controlled Settings. To control the variables in our experiment, for all the studies except knowledge structures, we experiment on all the multi-hop structures as raw data chains. For all the studies except the external knowledge format, we apply Single Sentence the EKD's knowledge format. Please refer to appendix for a more detailed explanation of each experimental setting.
Division of Results. As introduced in appendix B combining EKD methods and degrees, get six different types of EKDs for each hop of the query (each data chain may have multiple hops of the query). We experiment with all these EKDs. For the results regarding degrees, we divide the results based on the two different EKD degrees. For the results regarding methods, we divide our results based on the three different methods applied to the construction of EKD. For the results regarding positions, we divide the results based on which hop of query in the knowledge structure the EKD is introduced to. For the results on knowledge formats, we introduce Paragraph version to all previous EKDs and repeat all the experiments for comparison.
Data. We construct PKGs with different root nodes for both GPT3.5 and MPT--B using manually defined rules. In total, we used 17 types and 63 relations in the construction rules. We present in Figure 7 more details on the types of entities in PKGs we apply. For all the studies besides the knowledge structures in PKG, employ N-hop data chains (N E 2,3, 4), utilizing 200 chains for each type. Each N-hop data chain affords N positions for external knowledge introduction, three different methods, and two different degrees,
14

---

Published as a conference paper at COLM 2024
GPT3.5 Model
MPT Model
Others 26%
Parson 20%
Others 22%
Year
Song
Languaaee 3%
Language
16%
Airport 4hs
19%
Airport 3%
City 12%
Song 4%%
Company 5%,
Country
Country
Figure 7: Ratio of different types in model's PKG.
GPT3.5
Metrics
Type Match
Type Shift
2-hop 3-hop
59.92
61.00 f1.1
Consistency (%)
56.78
62.33 15.6
4-hop
51.00
52.17 11.2
MPT-7B
Metrics
Type Match
Type Shift
2-hop
48.25
53.92 (5.7
Consistency (%)
3-hop
42.22
46.67 14.5
4-hop
37.88
38.75
2-hop
82.07
78.8613.2
Confidence (%)
3-hop
80,99
77.78
4-hop
79.71
78.40 11.3
Table 7: The detailed results for GPT3.5 and MPT-7B when confronting Type Match or Type Shift EKDs as external knowledge. We differentiate multiple multilee structures instead of performing macro-averaging.
resulting in 6N rounds of queries (or, 6N different EKDs) and 6N2 hops of queries per original chain. Consequently, these constitute 10,800 query rounds, encompassing a total of 34,800 query hops.
For the study on knowledge structures in PKG, we extract 100 raw data chains for each multi-dependent structure type illustrated in the right column of Figure [ The collected chains constitute 6,600 query rounds, encompassing total of 24,600 query hops. The tool for automatic PKG construction and all the data we apply released.
D Supporting Analysis to Main Results
We perform additional analysis for some of the main results to further support our claims.
For the cases where the models fail to remain consistent, define two terms to describe their specific behaviors in the final answer. We categorize inconsistent responses into absten- tion (when a full query chain cannot be completed because the model starts to answer e-g., "I don't know" at certain hops), and variation (when the model provides a final answer that differs from the original PKG). We will use them for inconsistency analysis in the following.
D.1 Degrees
The results of the P-value we provide are derived from the T-test between all the consistency values under the interference of Type Shift EKDs and Type Match EKDs. Each data chain would provide a pair values for comparison, and there are in total 600 data chains for all 2 / 4-hop structures.
15

---

Published as a conference paper at COLM 2024
GPT3.5
Metrics
Type Match
Type Shift
2-hop
37.42
73.72
Abstention (%)
3-hop
37.53
69.76,32.2
4-hop
44.05
61.41
2-hop
62.58
26.28 136.3
Variation (%)
3-hop
62.47
30.24 (32.2
4-hop
55.95
38.59 1174
MPT-7B
Metrics
Type Match
Type Shift
2-hop
4.35
7..9
Abstention (%)
3-hop
3.85
6.35 22.
4-hop
8.18
9,93
2-hop
95.65
90.78 4.9
Variation (%)
3-hop
96.15
93.65 22.5
4-hop
91.82
90.07 $1.8
Table 8: The detailed error analysis on inconsistent chains for GPT3.5 and MPT-7B when confronting Type Match or Type Shift EKDs external knowledge. We differentiate multiple structures instead of performing macro-averaging.
Type Match
Type Shift
Deviated Responses
Conforming Responses
Confidence (96)
-.17
144
-11.03
6.13
75
2-hop
3-hop
4-hop
2-hop
3-hop
4-hop
Figure 8: The decrease of confidence brought by changing from Type Match to Type Shift EKDs. We observe significant confidence drop for deviated responses when introducing Type Shift EKDs while conforming responses show minor changes.
GPT3.5
MPT-7B
EKD Degrees
bbstention Variation Abstention Variation
(%)
(%)
(%)
(%)
Type Match
39.67
60.33
5.46
94.54
Type Shift
68.30
31.70
8.30
91.70
Table 9:
The rate of abstention and variation the inconsistent chains when confronting EKDs of different degrees for both GPT3.5 and MPT-7B. The results are the macro-averages on three multi-hop structures. Type Shift EKDs cause more abstentions.
The results we provide in Table and Table are macro-average three structures. We provide more detailed results regarding different multi-hop structures in Table 7 (Main Metrics) and Table (Error Analysis). The consistent trend in every structure provides additional support to our conclusions.
Besides, to further substantiate our claim that the model resists Type Shift EKDs, we segment the confidence based on whether the model's response conforming or deviated. Figure displays that the average confidence of generating deviated response plummets (left chart) when encountering Type Shift EKDs, while the confidence of conforming responses shows minor changes. This implies the primary cause of the confidence drop stems from the
16

---

Published as a conference paper at COLM 2024
GPT3.5
Metrics
Object
Indirect
Subject
2-hop
40.50 (0.50/24.8/55.
65.25
75.62
Consistency (%)
3-hop
45.75 (5.75(1788/377
63.50
69.42
4-hop
12.44/12.1/154
54.50
57.81
MPT-7B
Metrics
Object
Indirect
Subject
2-hop
38.38 11.5/266
49.88
65.00
Consistency (%)
3-hop
35.25 18.8/18.8
44.08
54.00
4-hop
(3.50.44
37.94
43.50
2-hop
81.87
76.67
82.92
Confidence (%)
3-hop
80.15
77.10
80,95
4-hop
79.57
77.82
79.80
Table 10: The detailed results for GPT3.5 and MPT-7B when confronting Object, Indirect or Subject Distractors external knowledge. We differentiate multiple structures instead of performing macro-averaging.
GPT3.5
Metrics
Object
Indirect
Subject
2-hop
47,06 16.3/17.0
63.31
64.10
Abstention (%)
3-hop
48.54172/77.
55.71
55.86
4-hop
50.81 j0.4/58
51.24
56.59
2-hop
52.94-163/11.0
36.69
35.90
Variation (%)
3-hop
51.46,72/7.3
44.29
44.14
4-hop
49.19
48.76
43.41
MPT-7B Indirect
Metrics
Object
Subject
2-hop
/.92/4.
4.74
5.36
Abstention (%)
3-hop
7.08
3.58
3.99
4-hop
10.24
7.55
9.29
Variation (%)
2-hop
(4.23/4
95.26
94.64
3-hop
|2.92
96.42
96.01
4-hop
89.76j3.5/3.1
92.45
90.71
Table 11: The detailed error analysis on inconsistent chains for GPT3.5 and MPT-7B when confronting Object, Indirect Subject Distractors as external knowledge. We differentiate multiple structures instead of performing macro-averaging.
deviated responses: the model is already hard to be deviated by Type Shift EKDs, and for responses that are interfered by them, the model's belief in them remains low.
We also investigate the inconsistent chains and discover that: Compared to MPT-7B, GPT3.5 is more likely to abstain under interference. ii) Compared to Type Match EKDs, Type Shift EKDs are more likely cause abstention. The error analysis is presented in Table 9 We provide more detailed results and further analysis to confidence in appendix D.1
D.2 Methods
We conduct the T-test for all the resulting consistencies between Object- Indirett Distractors and Object-Subject Distractors for both GPT3.5 and MPT-7B. Similarly, each test comprises 600 pairs of values for comparison.
We provide detailed results regarding the impacts three methods on different multi-hop structures in Table 10 (Main Metrics) and TT (Error Analysis).
17

---

Published as a conference paper at COLM 2024
Object Distractor
Indirect Distractor
Subject Distractor
Deviated Response
Conforming Response
64
Confidence (%)
90
82
88
2-hop
3-hop
4-hop
2-hop
3-hop
4-hop
Figure 9: The confidence of MPI-7BB's conforming and deviated responses when applying three methods. Results show Object Distractor induces the highest confidence for deviated responses, while Subject Distractor induces the highest confidence for conforming responses.
GPT3.5
MPT-7B
EKD Methods
Abstention Variation Abstention Variation
(%)
(%)
(%)
(%)
Object
48.80
51.20
8,75
91.25
Indirect
56.75
43.25
5.29
94.71
Subject
58.85
41.15
6.21
93.79
Table 12: The rate of abstention and variation in the inconsistent chains when confronting EKDs applying different methods for both GPT3.5 and MPT-7B. The results are the macro- averages on three multi-hop structures. Object Distractors cause the least abstention in GPT3.5, while the most for MPT--B.
3 H-Hop Chain
4-Hop Chain
Original Trend
GPT3.5
55
Consistency (%6)
52
Hop
Hop
Hop Hop
Hop 2 Hop 3rd Hop 4th Hop
32
EKD Introduced Position
Figure 10: The consistency concerning querying only the last two hops in the 3-hop and 4-hop chains. While MPT-7B still maintains an upward trend, GPT3.5 exhibits downward trend similar to that of a 2-hop chain (pink) in Figure
In addition, we analyze MPTT-7B's confidence concerning three methods in Figure 9 Again, we divide the confidence based on whether the response is conforming or deviated. Our findings show that the Object Distractor results the highest confidence when the response deviates, while the confidence for Subject Distraclor is highest when the response conforms to the PKG. These results also imply Object Distractor is the most powerful to deviate the model's is belief, while for the other two "weaker" EKDs, the model still trusts its original logic pathway in PKG.
For inconsistent chains, we discover that Object Distractors lead to the lowest abstention in GPT3.5 but the highest in MPT-7B. This indicates that among the three methods, GPT3.5 is less likely to abstain if the EKD presents direct conflict, while MPT-7B shows vice versa. The error analysis is presented in Table We also provide more detailed results and further analysis to confidence in appendix D.2 D.2
18

---

Published as a conference paper at COLM 2024
GPT3.5
Metrics
1s Hop
2nd Hop
3rd Hop
4th Hop
2-hop
65.50
57.42
Consistency (%)
3-hop 4-hop
64.00
56.08
58.58
55.83
47.08
49.42
54.00
MPT-7B
Metrics
1 Hop
2 2d Hop
3rd Hop
4th Hop
2-hop
48.58
53.58
Consistency (%)
3-hop
40.42
46.00
46.92
4-hop
37.33
36.17
39.58
40.17
2-hop
78.48
82.46
Confidence
3-hop
78.47
80.47
79.24
(%)
4-hop
79.10
79.38
79.09
78.67
Table 13: The detailed results for GPT3.5 and MPT-7B when EKDs are introduced in different positions external knowledge. We differentiate multiple structures instead of performing macro-averaging.
GPT3.5
Metrics
Single Sentence
Paragraph
2-hop
60.46
57.83 56.53 |3.0
Consistency (%)
3-hop
59.56
4-hop
51.59
48.75 2.9
MPT-7B
Metrics
Single Sentence
Paragraph
2-hop
51.09
48.38 j2.7
Consistency (%)
3-hop
44.11
42.34
4-hop
38.32
35.46,29
2-hop
80.48
78,28 [2.2
Confidence
3-hop
79.40
78.45
(%)
4-hop
79.06
78.36
Table 14: The detailed results for GPT3.5 and MPT-7B when confronting Single Sentence or Paragraph the EKD format of external knowledge. We differentiate multiple structures instead of performing macro-averaging.
D.3 Positions
We provide detailed results on GPT3.5 and MPT-7B's consistency and confidence concerning different positions where the EKD is introduced in Table 13 We have plotted the trend of consistency in our main results in Figure
To mitigate the influence of queries themselves, we conduct an ablation study by querying only the last two hops the 3-hop and 4-hop chains. From the results in Figure 10 we discover that: i) Overall consistency increases as shorter data chain is applied. ii) While MPT-7B's consistency still rises, GPT3.5's consistency declines. The higher consistency observed when EKDs are introduced to interfere with the first hop provides additional evidence of GPT3.5's heightened vigilance the beginning towards information that deviates from its PKG. We provide additional detailed results in appendix D.3
D.3.
D.4 Formats
We conduct the T-test for all the resulting consistencies between Single Sentence and Paragraph as the EKD format for both GPT3.5 and MPT-7B. Similarly, each test comprises 600 pairs of values for comparison.
19

---

Published as a conference paper at COLM 2024
GPT3.5
Metrics
Type Match
Type Shift
1-1-0
46.33
47.1
Consistency (%)
1-1-1
47.75
49.00
1-2-0
51.58
54.25
MPT-7B
Metrics
Type Natch
Type Shift
1-1-0
32.00
34.44
Consistency (%)
1-1-1
42.22
46.67 14.5
1-2-0
32.00
33.08 11.1
1-1-0
76.88
74.96 J1.9
Confidence (%)
1-1-1
79.35
77.82 11.5
1-2-0
79.57
77.65 J1.9
Table 15: The detailed results for GPT3.5 and MPT-7B when multt-depennent structures (1-1-0, 1-1-1, and 1-2-0) confronts Type Match or Tyye Shift EKDs as external knowledge. We differentiate the three structures instead of performing macro-averaging.
GPT3.5
Metrics
Object
Indirect
Subject
1-1-0
32.83
92.83/11.7/300.
44.50
63.33
Consistency (%)
1-1-1
32.25 (171128.8
49.38
60.50
1-2-0
41.12 13.8/21.6
54.87
62.75
MPT-7B
Metrics
Object
Indirect
Subject
1-1-0
20.67 J14.3/25.3
34.00
45.00
Consistency (%)
1-1-1
19..8|5..2/55
34.62
44.75
1-2-0
24.12/71/18.1
31.25
42.25
1-1-0
75.27
74.50
77.98
Confidence (%)
1-1-1
78.81
77.63
79.33
1-2-0
79.01
77.41
79.42
Table 16: The detailed results for GPT3.5 and MPT-7B when multi-dependent structures (1-1-0, 1-1-1, and 1-2-0) confronts Object, Indirect or Subject Distractors as external knowledge. We differentiate the three structures instead of performing macro-averaging.
We provide detailed results GPT3.5 and MPT-7B's consistency and confidence concerning the Single Sentence or Paragraph as the EKD format of external knowledge in Table 14 We show from the detailed results that every structure's trend is consistent with our main conclusion.
D.5 Multi-Dependent Structures
To establish the overarching applicability of our prior conclusions, we undertake a parallel analysis with EKDs of different methods and degrees multi-dependent structures in PKG. The experimental settings and methods are kept the same as those for multi-hop structures. As delineated in Table 15, for all three multi-dependent structures, our findings reveal that the model's 's consistency higher in response to Type Shift EKDs, though the model's overall that the Object Distractors remain the prime catalyst for the model's deviation. Notably, these insights consistent with the outcomes obtained from our investigations into multi-hop structures.
confidence lowers. Furthermore,
Furthermore, provide detailed results on GPT3.5 and MPT-7B's consistency and con- fidence concerning different positions where the EKD is introduced in Table 17 We have plotted the trend of consistency in our main results in Figure
20

---

Published as a conference paper at COLM 2024
GPT3.5
Metrics
Par1
Par2
Par2
Pivot
Child
Hop1
Hop1
Hop2
Hop
Hop1
2-hop
55.00
51.67
34.00
Consistency (%)
3-hop
50.83
53.83
39.50
49.33
4-hop
58.67
57.17
51.00
44.83
MPT-7B
Metrics
Par1
Par2
Par2
Pivot
Child
Hop1
Hop1
Hop2
Hop
Hop1
2-hop
31.67
29.33
38.67
Consistency (%)
3-hop
29.83
34.33
43.00
37.83
4-hop
30.17
31.50
32.50
36.00
2-hop
74.48
73.72
79.52
Confidence (%)
3-hop
78.11
76.24
81.32
78.67
4-hop
78.61
77.56
78.52
79.76
Table 17: The detailed results for GPT33.5 and MPT-7B when EKDs are introduced at different positions multi-dependent structures (1-1-0, 1-1-1, and 1-2-0) as external knowledge. Par. denotes the chains extended from the parent nodes of the pivot query. We differentiate the three structures instead of performing macro-averaging.
GPT3
Metrics
Type Match
Type Shift
2-hop
52.67
68.00
Consistency (%)
3-hop
43.56
52.78
4-hop
46.58
52.92 16.3
2-hop
69.87
66.98 |2.9
Confidence (%)
3-hop
68. 68.41
66.05 j2.4
4-hop
70.04
68.10 [1.9
Table 18: The consistency and confidence of GPT3 when confronting Type Match or Type Shift EKDs external interfering knowledge. The conclusion on degrees consistent and even more pronounced for GPT3.
Type Match
Type Shift
Deviated Responses
Conforming Responses
Confidence (96
+0.82
0.65
75
1563
-1.23
70
-12.26
80
85
60
2-hop
3-hop
4-hop
2-hop
3-hop
4-hop
Figure 11: The decrease of confidence in GPT3 brought by changing from Type Match to Type Shift EKDs. The confidence drop can be mainly attributed to the deviated response, which implies GPT3 also shows resistance to Tyype Shift EKDs.
Additional Results from GPT3
To further support our discoveries, we perform additional experiments on GPT3 (Text- Davinci-003). Though GPT3 is not designed conversational model, its results can still reflect and bolster some of the trends that we have discovered. We perform experiments on 2 / 3 / 4-hop data chains, with 100 raw chains from GPT3's PKG for each type. We keep the rules we applied for constructing the PKG the same, and we keep all the other experimental setups the same introduced in appendix
21

---

Published as a conference paper at COLM 2024
GPT3
Metrics
Object
Indirect
Subject
2-hop
J24.5324
65.25
75.00
Consistency (%)
3-hop
34.00 34.00 1185/24.0
52.50
58.00
4-hop
39.00
53.12
57.12
2-hop
71.72
61.34
72.33
Confidence (%)
3-hop
69.79
63.30
68.62
4-hop
71.29
65.27
70.74
Table 19: The consistency GPT3 when confronting EKDs that apply different methods. Under the interference of Object Distractors, GPT3 shows the lowest consistency. This result remains consistent with previous conclusions.
GPT3
Consistency (%)
Chain
54
3-Hop Chain
4oHop Chain
1 Hop 2nd Hop
3 Hop
4th Hop
EKD Introduced Position
Figure 12: The consistency of GPT3 when the EKD is introduced to interfere with different positions in the data chain. GPT3 exhibits a similar trend as GPT3.5 in Figure 6
E.1 Degrees
In Table we observe the same trend in GPT3 that the model resists Type Shift EKDs as external knowledge, and the overall confidence in responses is lowered. By further dividing the responses into conforming and deviated ones, we show in Figure I1 that, similarly, the drop in confidence be mainly attributed to the deviated responses. All these results further bolster the claim that the model put less faith in more severely edited external knowledge represented by Type Shift EKDs.
E.2
Methods
In addition to EKDs of different degrees, we also investigate GPT3's consistency towards EKDs that apply different methods. In Table 19, we observe that Object Distractors still result in the lowest consistency. This trend also remains the same what we have shown previously, indicating that GPT3 is also susceptible to Object Distractors the most, while the other two "weaker" EKDs also bring certain impacts.
E.3 Positions
The pattern positions is different for GPT3.5 and MPT-7B, as we have shown earlier in Figure 6 In Figure 12 we demonstrate that GPT3's trend is more similar to GPT3.5: both models show high consistency being interfered with at the beginning the phenomenon of which is not exhibited in MPT-7B. Then, the model's consistency starts to rise again as the position interference moves toward the tail of the data chain. GPT3's results further support the GPT familyy's initial sensitivity towards information's veracity directly after the introduction of external knowledge.
the data chain,
E.4 Formats
We extend the context format of all the EKDs into Paragraph in the same way we do for GPT3.5 and MPT-7B. We present the results of the comparison in Table 20 GPT3's consis-
22

---

Published as a conference paper at COLM 2024
GPT3
Metrics
Single Sentence
Paragraph
2-hop
60.34
47.30
Consistency (%)
3-hop
48.17
40.00 8.2
4-hop
49.75
44.00.5.6
Table 20: The comparison of GPT3's consistency when presented with EKDs of Single Senfence format versus Paragraph format. GPT3's consistency also lowers when the context becomes lengthier.
country":
"he capital is", "city"l.
['is founded become independent in which year", "year"].
['the official most commonly spoken language is', "language") ),
['the national anthem is", "song"l,
I'ehe colors on the national flag (may be multiple)", "color",,
hass the longest river named", 'river"),
('has highest mountain named", , "mountain"),
t'ehe countries the east that shares boarder with it (may be multiple)
"contry"]
J.
"city":
('belongs to which country", "country"], has the largest airport named", "airport"],
laas the largest (by area) university / college named", "school"), J,
('the time zone of this city in UTC", "time zone"), i'what the largest company (by people) based in city".
"company"]
J.
"year": [[
('the US president this year named", "person"), ",
[ I'this year's Oscar best actors are (may have multiple)", "person') 1
['this year's Oscar Outstanding Pictures is (the first one in alphabetical order)), "film"),
t'ts year's NBA championship which team", "sport team"), ['the first Olympic Games hosting city from this year", "city"]
Figure 13: The example rules we apply in building the PKG.
tency lowers in a more pronounced way than GPT3.5 and MPTT-B the external knowledge becomes lengthier and more detailed. This further supports our previous conclusions and also implies that GPT3 is more biased trust the detailed but potentially fake external information.
23

---

Published as a conference paper at COLM 2024
##* System Message
You are supposed to answer the question given by the user succinct way.
Please do not provide any additional information.
If you do not know the answer for sure, please generate 'Not Sure'. If you think there are multiple answers, please split them by semicolon (+)
Instruction
nswer the question briefly, and please always provide an answer.
### User
hat'ts the capital of USA?
## A ssistant
W Washington DC
### U ser
Jackson Chen is born in which city?
#d ssistant
Not Sure
User
hat are the colors on the national flag of China?
#d A ssistant
Red; Yellow
##* U ser
W hat is the longitude of Washington DC round to integer?
#dili ssistant
77W
### User
Who is the headmaster of Yale University in 2000?
#30 ssistant
Richard Levin
### U ser
(Target Question]
#d8 ssistant
(Target Model Response)
Figure 14: The prompt for retrieving the model's answer when building the PKG.
24

---

Published as a conference paper at COLM 2024
### System Message
You are supposed to judge the given concepts are consistent (consistent doesn't mean the same, alias is allowed).
1. they are consistent, then please generate the common concept they share, otherwise, please generate 'N/A'.
2. If there are multiple concepts in one list, you can list all the concepts shared the same meaning in all the lists, and discard other inconsistent ones.
Please separate answers in semicolon
### Instruction
Give core concept if the there lists, otherwise, please generate 'N/A".
that shared consistent
### User
("The state is Georgia"), ["Georgia], ['State of Geurgia")
#### ssistant
G Georgia
### User
['1945]. [year of 1948"]
# ssistant
N/A
### User
I'Google', 'Applc"], ('Google', 'Apple Inc."], ['Apple', 'Google', 'Microsoft")
### ssistant
Google; Apple
### User
['Not Sure'), ['The answer should be French.'). ['l am not sure about the answer, please provide more information." "]
### ssistant
N/A
### User
['Red', 'Blue'), ['bluc', 'yellow']
#5HH Asiistatt
Blue
### User
['L.A]]. ['Los Angeles"), ['The city that Jackson lives in is L.A."]
#80 A ssistant
Los Angeles
### U User
[Tarett L Lists}
### A ssistant
[Target Model Response]
Figure 15: The prompt for judging the consistency of the model's answers when building the PKG.
25

---

Published as a conference paper at COLM 2024
Object Distractor Type Match
### System M essaag
You are helpful assistant. You are given question and its standard answer.
Please first turn them into definitive statement (instead of the QA format).
Then you should hallucinabe another answer that is not true (but should be
highly related, belonging to the same type the original answer). Please and state the hallucinated relationship
output the ballucinated
sentence.
User
Question: W hat is the capital of Afghanistan?
nswen Kabul
#**i A ssistant
Statement: The capital of A fghanistan is Kabul.
Hallucinated A nswer: Kandahar
Misinfomation: The capital of A fghanistan is K Kandahar
### U ser
Question: France which continent?
nswen Europe
ssistant
Statement: France in Europe.
Hallucinated A nswer: sia
Misinfomation: France is in Asia.
### U ser
[Target Question /A nswer)
#5# A ssistant
(Target Model Response)
Figure 16: The prompt for constructing Object Type Match EKDs.
Object Distractor Type Shift
##* System M essage
You are a helpful assistant. You are given question and its standard answer,
Please first turn them into definitive statement (instead of the QA format).
Then should hallucinate another answer that exist in this world but is
totally not related to the question (belongs different type of entity than the
original answer). Please output the irrelevant answer and state the new sentence.
hallucinated relationship
U
Question: W hat is the capital of fghanistan?
nswer: Kabul
### A ssistant
Statement: The capital of A fgganisttn is Kabul.
Irrelevant A nswer: Michael Jacksn
Misinformation: The capital of A fghanistan is Michael Jackson.
### U ser
Question: France which continent?
nswer: Europe
ssistant
Statement: France continent Europe. Irrelevant nswer. A pple Inc.
Misinformation: France is on continent Inc.
### U ser
[Target Q Question/ Asswe))
#d* ssistant
[Target Model Response)
Figure 17: The prompt for constructing Object Type Shift EKDs.
26

---

Published as a conference paper at COLM 2024
Subject Distractor - Type Match
## System Message
You are helpful assistant. You given question and standarl
Please first turn them into triplet (Subject, Relationship, nswer). Then
you should hallucinate ano ther highly related subj ect (belonging the same
type as the original subject), keep the answer and relationship the same, and state the new hallucinated relationship in a sentence.
### U ser
Question: W hat is the capital of Afghanistan?
A uswen Kabul
#### ssistant
Triplet: (Afghanistan, capital, Kabul)
Hallucinated Subject: Pakistan
Statement: The capital Pakistan Kabul.
User
Question: France which continent?
nswen Europe
##th A ssistant
Triplet: (France, is on continent, Europe)
Hallucinated Subject: Gemany
Statement: Germany is on continent Europe.
### U ser
(Target Question Answer)
##tt ssistant
(Target Model Response)
Figure 18: The prompt for constructing Subject Type Match EKDs.
Subject Distractor Type Shift
##* System M essage
You are a helpful assistant. You are given question and its standard answer.
Please first turn them into triplet (Subject, Relationship, A Aswer)). Then
should hallucinate ano another ther subject that exists in this world but is totally
not related to the question (belongs different type of entity than the original subject). Please keep the answer and relationship the same, and state the new hallucinated relationship sentence.
User
Question: W hat is the capital of fgghnnstan? nswen: Kabul
# ssistant
Triplet: (Afghanistan, capital, Kabul)
Hallucinated Subject: Michael Jackson
Statement: The capital of Michael Jackson is Kabul.
### U ser
Question: France which continent?
nsswr. Europe
Asistant
Triplet: (France, is on continent, Europe)
Hallucinated Subject: TTo kill a mocking bird'
Statement: To kill mocking bird' is on continent Europe.
### U ser
[Target Q Question A nsswr)
##* ssistant
[Target Model Response)
Figure 19: The prompt for constructing Subject Type Shift EKDs.
27

---

Published as a conference paper at COLM 2024
Indirect Distractor Type Match
System essaae
are helpful assistant. You are given question and standard answer. Please first turn them into triplet (Subject, Relationship, A nswer). Then you should keep relationship the same, and provide another pair of correct subject and answer that fulfill the relationship, and state the new relationship in sentence.
## U ser
Question: W hat is the capital of Afghanistan?
nswer: Kabul
#d ssistant
Triplet: (Afghanistan, capital, Kabul)
New Subject: Pakistan
New nswer: Islamabad
Statement: The capital of Pakistan is Islamabad.
### U ser
Question: France is on which continent?
A nswer: Europe
#diit A ssistant
Triplet: (France, is continent Europe)
New Subject: India
New Answer: er: A
Statement: India is on continent
### User
(Target Q Question Answer)
#*## ssistant
(Target Model Response)
Figure 20: The prompt for constructing Indirect
Type Match EKDs
Indirect Distractor Type Shift
## System Message
are helpful assistant. You are given question and standard answer.
Please first tum them into triplet (Subject, Relationship, Aswer). Then same, and provide another pair subject and answer that exists in the world but totally unrelated to the original question. Finally, state the new hallucinated relationship in sentence.
you should keep relationship
#** U ser
Question: W hat is the capital of Afghanistan?
nswer. Kabul
#di# ssistant
Triplet: (Afghanistan, capital, Kabul)
Hallucinated Subject: Michael Jackson
Hallucinated A nswer: Eiffel Tower
Statement: The capital Michael Jackson is Eiffel Tower. User
Question: France is on which continent?
A nswer: Europe
#d8 A ssistant
Triplet: (France, is on continent, Europe)
Hallucinated Subject: A pple Inc.
Hallucinated, Anwer:: Monopoly Game
Statement: pple Inc. continent Monopoly Game.
U
[Target Q Qussioon/ nswer))
#did ssistant
(Target Model Response)
Figure 21: The prompt for constructing Indirect Type Shift EKDs.
28

---

Published as a conference paper at COLM 2024
### System M Message
You are helpful assistant in writing articles in parallel world. Please fake some news in this world around the given Iact provide.
### Instruction
You are asked generate fake piece of news around the statement I give you. Please explain the statement present in detail, and make your news true and tempting as possible. Please write news happens naturally. Wrap up your news within 3-4 sentences.
#*# U User
Statement:
The name of the longest river in France is Yellow Elephant
### ssistant
It has been revealed ed that the longest river in France is, in fact, called the
Y Yellow Elephant. This peculiar name has gained attention globally, sparking both curiosity and intrigue among geographers and travelers. The origins of this unique designation remain mystery, giving rise to various theories and speculations. As the Yellow Elephant River flows majestically through the picturesque landscapes of France, locals and tourists alike eager to uncover the story behind this fascinating name.
##* U ser
Statement: The most recent Olympic Games is held where giraffe can fly.
### ssistant
The most recent Olympic Games were hosted in location where giraffes
can fly. This groundbreaking achievement has captivated the world's
attention, with athletes and spectators from all comers of the globe flocking
witness this incredible spectacle. The host city, filled with floating giraffes and a vibrant atmosphere, provided otherworldly backdrop for the international sporting event. This remarkable feat has solidified the Olympic Games as symbol of limitless imagination and boundless possibilities.
### User
Statement: The capital of China is Washington DC.
### ssistant
Recently, the Chinese govermment has announced that the capital of China
will now be relocated to Washington DC. This decision comes part of strategic move to strengthen diplomatic ties with the United States and foster closer cooperation between the nations. The historic shift is expected to bring about significant economic and political changes, as China
seeks establish new of collaboration with its merican counterparts.
##* U ser
[Target Statement)
## ssistant
[Target Model Response)
Figure 22: The prompt for transforming the EKDs into Paragraph format.
29


Reviews:
[{"review_id": "NusDIJ9b8s", "reviewer": "Reviewer_3wvY", "summary": "This paper examines how LLMs interact with their internal parametric knowledge and external knowledge sources, particularly when external knowledge distractors (EKDs) interfere with internal knowledge across various dimensions (such as degrees, methods, positions, and formats). It also highlights that LLMs are sensitive to the accuracy of external knowledge but can be misled by unrelated information, even in simple, single-hop queries. This study reveals the intricate interaction patterns between external and parametric knowledge in LLMs.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you very much for your comments on our paper. Our work primarily takes an analytical approach to systematically explore the output patterns of models when external knowledge conflicts with parametric knowledge.\n\n> How can LLMs be better improved\n\nThe conclusions we have drawn in Table 3 will promote LLM's alignment with knowledge and help design better methods for knowledge incorporation, enabling the model to trust external knowledge more (e.g., during knowledge updates) or to be less affected by external knowledge (when it is noise). For example, we can consider various aspects such as the length, format, and degree of knowledge changes and, based on practical application scenarios, encourage alignment between the model and internal or external knowledge.\n\n> Boundary of direct conflict\n\nWe explore three different methods to introduce knowledge change. Among them, object distractor aligns the most with “direct conflict”, as our problem scenarios expect the model to answer the object. In our work, we distinguish “direct conflict” with others based on the sentence structure (subject / relation / object), as changing the answer (target object) of what is being asked is the most intuitive and direct way to introduce knowledge conflict. Building on our framework, future research can quantitatively and more granularly investigate the impact of knowledge conflict by changing the similarity between knowledge elements (e.g., altering meaningful noun entities or relations by different percentages).\n\n> Generalizability of our framework\n\nWe believe that unstructured data ultimately consists of similar knowledge triplets. Therefore, the conclusions we have obtained can be partially transferred to these scenarios. Moreover, the knowledge types involved in our experiments are already diverse, providing inherent adaptability to different domains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The paper addresses a significant issue in LLMs regarding the integration and conflict of external knowledge with internal knowledge structures.\n\n2.It introduces a novel framework using PKGs and EKDs to systematically study the effects of external knowledge on LLMs.\n\n3.The research is thorough, with experiments conducted on both black-box and open-source models, providing a comprehensive understanding of the problem.", "reasons_to_reject": "1.How can LLMs be improved to better handle conflicts between external knowledge and their internal knowledge structures?\n\n2.While the study addresses direct conflicts, the subtler nuances of how LLMs handle slightly mismatched but not directly conflicting information could be explored further. In real-world scenarios, it is difficult to clearly define the boundaries of direct conflicts.\n\n3.The experiments are controlled. Can the framework presented be generalized to accommodate a wider range of LLMs and knowledge sources (not limited to factual triples, e.g. unstructured data, cultural context)?\n\n4.There might be a need for further exploration into the long-term effects of repeated exposure to external knowledge distractors (EKDs) on LLMs.", "questions_to_authors": "See Weaknesses.", "extra_scores": {}}, {"review_id": "NQ0YbOGf0v", "reviewer": "Reviewer_N8Lh", "summary": "The paper introduces a framework to construct LLM's parametric knowledge graph and corresponding external knowledge distractors to investigate external knowledge's effects on parametric knowledge. Based on the framework, the paper studies the impact of different factors of external knowledge (provided in context) on parametric knowledge graphs. Experiments with GPT-3.5 and MPT demonstrate a series of findings and insight into how LLMs deal with external knowledge.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments on our paper. I would like to address some of your concerns.\n\n> Implications of our findings\n\nWe explored knowledge conflict from four dimensions and simplified knowledge to the most basic form of triplets to better understand the impact's essence. Regarding the practicality of the conclusions, many of our findings can be generalized (e.g., the conclusions regarding degree, method, and format in Table 3) as we observed similar patterns across different models. These findings can guide better methods for knowledge alignment. From the perspective of introducing external knowledge, we can reconstruct the content and structure of knowledge in RAG based on the degree, method, and other aspects to make the model trust external knowledge more (e.g., during knowledge updates) or distrust it (when the RAG content contains a lot of noise).\n\n> Soundness regarding computation of confidence\n\nConfidence calibration commonly used methods include taking the minimum probability of each token, summing them up, and taking the average, or taking the product [1]. In this paper, we adopted the last method mainly because taking the minimum value does not reflect the confidence in knowledge well (e.g., the minimum value of probability may occur on a certain function word instead of an entity word), and summing and averaging do not reflect the relationship and coherence between each generated token. In contrast, the cumulative product method overcomes the aforementioned shortcomings. At the same time, in our problem scenario, we only want the model to output one knowledge entity, and the output content is short, so it does not lead to excessively low confidence values after the cumulative product.\n\n> Generalization of our framework\n\nWe also conducted a comprehensive exploration of GPT-3 in the appendix (Appendix E) and obtained similar conclusions to those in Table 3. Due to space limitations, we did not validate more models, but our research framework (including the construction of the parametric knowledge graph and the introduction of external knowledge distractors) can be applied to other models for further investigation, and our code will be fully open source.\n\nWe will also make corrections for the writing typos you mentioned. Thank you very much for your suggestions.\n\n> [1] Zhao, X., Zhang, H., Pan, X., Yao, W., Yu, D., Wu, T., & Chen, J. Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Knowledge conflict is a knowledge conflict is an important research question\n2. The proposed framework to construct LLM's parametric knowledge graph and corresponding external knowledge distractors is interesting and could be helpful to future studies of knowledge conflict", "reasons_to_reject": "1. Some of the conclusions from the experiments are a bit narrow, for instance, some are model-specific or only related to a certain factor such as external knowledge graph degree. I am unsure whether they have any useful implications for real-world applications such as RAG systems. \n\n2. There is no clarification on whether the confidence metric is sound. After all, there are many methods of measuring uncertainty or confidence.\n\n3. Although one black-box LLM and one open-source LLM seem adequate, findings in the paper such as \"The high consistency of GPT3.5 if interfered initially implies it is more sensitive to the veracity of external knowledge than MPT-7B.\" might need results from more LLMs together with more explanations.", "questions_to_authors": "1. Typos: in the abstract, \"they direct conflict\";   Page 8, \"P-value p ¡ 0.001\"", "extra_scores": {}}, {"review_id": "Jq8fIatHTX", "reviewer": "Reviewer_io4C", "summary": "This paper studies the 'ripple effect' problem, i.e., the influence when the external knowledge provided by prompts and the parametric knowledge contradict each other. \nSpecifically, they introduce an interesting concept, *parametric knowledge graph*, which visualizes latent knowledge in LLM parameters as knowledge, and represents external knowledge as distractors. This establishes a reasonable evaluation framework to study the ripple effect caused by knowledge conflict.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback and raised questions. Please refer to our responses to Reviewer 3wvY and N8Lh regarding the generalization of our framework.\n\n> Meaningfulness behind our results\n\nAll the conclusions in our paper are based on macro-averaged values and control for single variables, and we mentioned the testing of significance for experimental results in Section 5.2. Though the averaged value may show little numerical differences, we demonstrate in the Appendix (Table 7 - 11) that each sub-setting shows a consistent pattern in our experiments.\n\n> The meaning of E in Equation 2\n\nE represents the entity in the model's answer, and it is independent of whether the model aligns with parametric knowledge (conforming) or the distractor (deviated). We found that when the model's answer deviates, there is a more significant increase in confidence, indicating that external distracting information not only biases the model's answer but also does so with a confidence boost. \n\n> Q1: Explanation of conclusion \"presenting direct ... influence\"\n\nIt is a combination of findings from the method and degree sections. Here, direct conflict refers to object distractors (as the question asks for the object), while confounding change refers to type-matched edited knowledge.\n\n> Q2: Clarification on our querying method\n\nThe data chain is queried within a single conversation, although we asked one hop at a time. After each question, we received an answer and proceeded to ask the model the next hop based on that answer. Therefore, the next question is based on (A2, R1).\n\n> Q3, Q4: Clarification on the “position” setting\n\nIn the position setting, all our ground truth remains unchanged. We always start asking a series of questions from the beginning and inquire about the next hop based on the model's previous answer. This process breaks down the model's chain-of-thought during multi-hop reasoning. Therefore, the results under this setting are all comparable.\n\nThe measured consistency is based on the final result obtained by the model. If the model's answer is deviated in the first hop, it is highly likely that the correct answer will not be obtained in the end. However, our interesting finding is that GPT-3.5 is not easy to interfere with when external knowledge is introduced in the first hop, resulting in a higher consistency.\n\nWe will continue to improve our writing and correct the typos and images as suggested. Thank you very much.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of *parametric knowledge graph* sounds reasonable and interesting. It provides a seemingly applicable framework for the study of ripple effect in LLMs, which is an important research question.\n\n2. The framework of constructing parametric knowledge graphs is automated, which builds a convenient way to study the ripple effect in LLMs. \n\n3. With the proposed framework, this paper conducts a series of extensive experiments that imitate different knowledge conflict scenarios, including updated information, misinformation and fictional information, which yield interesting results. This results would be valuable if they could be validated on a broader spectrum of LLMs.", "reasons_to_reject": "1. The experiments only included 2 LLMs, GPT-3.5 and MPT-7B, which makes it a concern whether the results are generalizable to other LLMs, especially more advanced LLMs, including GPT-4, or ones with different architectures, such as Mixtral. Furthermore, the results demonstrate that 'GPT-3.5 and MPT-7B display distinct patterns', which further makes generalizability of the observation and analysis a concern. \n\n2. The writing of this paper could be better structured for improved clarity. For example, although the author spent lots of space elaborating *parametric knowledge graph* and *external knowledge distractors*, it is still hard to understand how the experiments are conducted and the metrics are calculated, where examples illustrated in figures are expected. This makes it difficult to understand the results and analysis. I think it might be better to restructure Sec 3 and Sec 4 by incorporate *method* and *metric* in Sec 4 as part of Sec 3.  \n\n3. Despite extensive evaluation results, I am concerned about the meaningfulness behind these numbers. The *consistency* are typically around 50%, and the results of different settings show limited difference. For example, the authors conclude that \"Lengthier contexts decrease consistency.\" because the consistency for GPT-3.5 drops from 57.2 to 53.37, which is not a significant difference. Hence, the results are not very exciting. It might be interesting to see methodologies that help to manipulate this results, i.e., increasing or decreasing consistency. \n\n4. In Equation (2), it is unclear whether the target entity *E* denotes the original knowledge in PKG, or the distractor in EKD. This makes following discussion about the *confidence* metrics difficult. I am confused why \"Deviated responses\" in Figure 4 right show more positive confidence change.", "questions_to_authors": "1. Can you explain why 'presenting direct conflicts or confounding changes instead of evidently false information wields greater influence'?\n\n2. For the *Method* in Sec 4, do \"inquiring (A2, R1)\"(where I think exists a typo) and inquiring \"(E0, R1)\" conducted in separate sessions (i.e., their prompts have little overlap)? Or are LLMs directly asked to answer the questions demonstrated in Fig 1 or Fig 3? \n\n3. Results in Table 3 \"Positions\" are confusing. Intuitively, I suppose that altering the 1st Hop will change the prediction in a cascading way so the consistency should be lowest according to the metric definition. If the 1st Hop is changed, will be ground truth for the 2nd Hop change correspondingly? If so, considering that the different hops are inquired in separate sessions, what's the difference among settings with different hops (1st Hop, 4th Hop) ?\n\n4. Are the results in Table 3 of different positions (1st Hop and 3rd Hop) comparable? If the data is with 2/3/4 hops, then results of altering \"1st hop\" is calculated in all data, while results of altering \"3rd hop\" exclude the data with 2 hops. Please correct me if there is any misunderstanding. \n\n5. It would be clearer if all conclusions and findings in the paper could be highlighted or listed, especially in the introduction section.\n\n6. Typos:\n  - probe the model’s answer A2 after inquiring (A2, R1). \n  - type \"Country\". has.\n  - P-value p 0.001\n\n7. The figures and tables should be not be in png or jpg format.", "extra_scores": {}}, {"review_id": "ja3LPs6D5d", "reviewer": "Reviewer_Jvke", "summary": "In this paper, the authors propose a controllable knowledge conflict construction framework using parametric knowledge graphs. Specifically, they demonstrate two different graph structures: multi-hop and multi-dependent, which enable them to control the position of conflicts. Furthermore, they implement four different types of distractors, providing in-depth analysis from multiple perspectives. According to the findings, current LLMs tend to trust external knowledge sources but still get distracted by unrelated information.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. Please refer to our responses to Reviewer 3wvY and N8Lh regarding the generalization of our framework.\n\n> Justification of demonstration example uses\n\nRegarding the demonstration examples, we ensured that it is unrelated to the knowledge we inquired about, and we used the same demonstration for all the questions to ensure fairness. The purpose of demonstrations is to ensure the model's output format. We also queried three times, and only when all three answers are consistent, do we consider it as the model's parametric knowledge. This maximizes our method's stability.\n\n> Specific measurement regarding consistency\n\nAs measurement for each hop will make the final results hard to compare (especially in the position setting), we only measured the model's final result (as introduced in Equation 1), which is always an entity. Moreover, the demonstration ensures the model's output format. Therefore, it is easy to evaluate whether it matches the gold answer.\n\nSpecifically, we used GPT-3.5 to compare the model's output with the ground truth, considering them consistent even if the content is different but the semantics are the same, thus avoiding issues such as abbreviations or alternative names (e.g., America / US). Through sampled observations, we found that GPT-3.5 has this discrimination ability and can achieve a judgment accuracy rate close to 100%, ensuring the accuracy of the consistency measurement.\n\n> The underlying reason behind “context length”\n\nFurthermore, regarding the influence of context length on knowledge conflict, it can essentially be attributed to longer texts providing richer details and stronger persuasiveness. At the same time, longer texts contain more complex information, which may cause the model to sometimes focus more on surface wording and overlook the underlying logic, leading to a trust bias towards longer texts. We will further incorporate more discussions on the results and improve the quality of the writing.\n\nThank you very much for your feedback. We will continue to optimize the reader's reading experience and make the conclusions more simple and understandable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The knowledge conflict problem is really important in the RAG scenario, and the insights in this paper are promising and helpful for the community.\n2. The proposed framework for knowledge conflict construction is novel, which enables this work to provide an in-depth analysis of new aspects that are not implemented in previous works, such as controllable multi-hop problems.\n3. The experiments are sufficient and the analysis is promising and convincing.", "reasons_to_reject": "1. This paper is a little difficult to follow because of the writing style. For instance, in the last paragraph of the introduction, numerous words are densely packed without any clear indication of important insights, which disrupts the reading experience. \n2. Regarding \"parametric knowledge,\" the authors utilize in-context demonstration to elicit it. However, could this introduce demonstration bias, potentially distorting the reality of parametric memory? How did the authors ensure that the elicited parametric knowledge remains unbiased even with demonstrations?\n3. Regarding the judgment of models' answer consistency, what is the accuracy of this process? Can this accuracy guarantee the quality of the judgments? The authors should provide clarification on this aspect in the paper.\n4. To ensure that the insights in this paper can be generalized to other widely used LLMs, it's essential to consider including other LLM families, such as Llama and Gemini. While the extensive experiments in this paper may not allow reporting all results, it's acceptable to focus on key insights. However, including at least some results from other LLM families may be necessary.\n\n(I will reconsider the score if the authors can address the points mentioned above to my satisfaction.)", "questions_to_authors": "1. How do the authors perceive the phenomenon of \"The length of context influencing consistency\"? Are there any underlying reasons behind the superficial \"length\"?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies the important problem of analyzing LLMs behavior under knowledge conflicts, an increasingly common setting in RAG and language agents. It proposes a parametric knowledge graph formulation, with which external knowledge extractors are then used in a controlled setting to examine different factors. Results with GPT-3.5 and MPT -7B show some interesting insights.

Reasons to Accept
- Analyzing LLMs' behavior under knowledge conflicts is an timely and important topic. This paper presents a study that seems more fine-grained, thanks to its parametric knowledge graph and distractors designs.
- Comprehensive experiments and analyses with two LLMs, with insights that may be informative to future studies.

Reasons to Reject
- The study is only based on two LLMs. It's not entirely clear which and to what extent will the observations generalize the different, especially stronger LLMs. Prior work (e.g., Xie et al., 2023) tends to use many more LLMs but focus on less (but more focused) aspects to analyze to improve the generalizability of the conclusions.
- The writing could be hard to follow at several places, as pointed out by a few reviewers.
Decision: Accept