Understanding Parametric and Contextual Knowledge Reconciliation within Large Language Models
Authors: Jun Zhao
Yongzhuo Yang
Xiang Hu
Jingqi Tong
Yi Lu
Wei Wu
Tao Gui
Qi Zhang
Xuanjing Huang
Venue: NeurIPS 2025 spotlight ()
Keywords: Natural Language Processing
Information Retrieval
Network Analysis
Primary area: applications

Abstract: Retrieval-Augmented Generation (RAG) provides additional contextual knowledge to complement the parametric knowledge in Large Language Models (LLMs). These two knowledge interweave to enhance the accuracy and timeliness of LLM responses. However, 
  the internal mechanisms by which LLMs utilize these knowledge remain unclear. We propose modeling the forward propagation of knowledge as an entity flow, employing this framework to trace LLMs' internal behaviors when processing mixed-source knowledge. Linear probing utilizes a trainable linear classifier to detect specific attributes in hidden layers. However, once trained, a probe cannot adapt to dynamically specified entities. To address this challenge, we construct an entity-aware probe, which introduces special tokens to mark probing targets and employs a small trainable rank-8 lora update to process these special markers. We first verify this approach through an attribution experiment, demonstrating that it can accurately detect information about ad-hoc entities from complex hidden states.  Next, we trace entity flows across layers to understand how LLMs reconcile conflicting knowledge internally. Our probing results reveal that contextual and parametric knowledge are routed between tokens through distinct sets of attention heads, supporting attention competition only within knowledge types. While conflicting knowledge maintains a residual presence across layers, aligned knowledge from multiple sources gradually accumulates, with the magnitude of this accumulation directly determining its influence on final outputs.

Understanding Parametric and Contextual Knowledge Reconciliation within Large Language Models
Jun Zhao 1.4, Yongzhuo Yang', Xiang Hu Jingqi Tong', , Yi Lu',
Wei Wu Tao Guit.3, Qi Zhang! 1.2 Xuanjing Huang College of Computer Science and Artificial Intelligence, Fudan University
Shannhaii Collaborative Innovation Center of Intelligent Visual Computing *Shanghai Innovation Institute *Ant Group fzhaoj 19, q2) @fuunn edu. en, wure119950318gmm11.ccom
Abstract
Rerrieval-Augmented Generation (RAG) provides additional contextual knowledge complement the parametric knowledge in Large Language Models (LLMs). These two knowledge interweave to enhance the accuracy and timeliness of LLM responses. However, the internal mechanisms by which LLMs utilize these knowl- edge remain unclear. We propose modeling the forward propagation of knowledge as an entity flow, employing this framework to trace LLMs' internal behaviors when processing mixed-soures knowledge. Linear probing utilizes a trainable lin- ear classifier to detect specific attributes hidden layers. However, once trained, a probe cannot adapt to dynamically specified entities. To address this challenge, we construct an entity-aware probe, which introduces special tokens to mark probing targets and employs small trainable rank-8 lora update to process these special markers. We first verify this approach through an attribution experiment, demon- strating that can accurately detect information about ad-hoc entities from complex hidden states. Next, we trace entity flows across layers to understand how LLMs reconcile conflicting knowledge internally. Our probing results reveal that con- textual and parametric knowledge are routed between tokens through distinct sets of attention heads, supporting attention competition only within knowledge types. While conflicting knowledge maintains a residual presence across layers, aligned knowledge from multiple sources gradually accumulates, with the magnitude of this accumulation directly determining its influence on final outputs.
Introduction
Through large-scale pre-training, Large Language Models (LLMs) have acquired vast amounts of knowledge implicitly encoded within their model parameters, forming the cornerstone of their general capabilities. However, this paradigm knowledge storage faces twofold limitations: from a coverage perspective, parametric knowledge obtained through pre-training on public data struggles to extend to domain -specifcc private knowledge; from a temporal perspective, static parametric knowledge gradually loses its effectiveness as the real world evolves. To address these limitations, RAG techniques have been introduced to provide LLMs with additional contextual support [ 45) Such real-time contextual knowledge both expands the inherent boundaries of parametric knowledge and mitigates its temporal constraints.
This raises crucial question: to what extent can LLMs accept and utilize external contextual knowl- edge, particularly when such external information conflicts with the model's inherent parametric
Cooresponniing authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

knowledge (which may be outdated or inacourate)? Researchers have investigated this question through various approaches, including negation injection [T] 20 and entity substitution 23
revealing that LLMs often stubbornly adhere to their original beliefs. Other studies [39 16 have demonstrated that the fluency of external context and the strength of the model's prior beliefs sig- nificantly influence LLMs' receptiveness to external knowledge. While these studies have revealed behavioral characteristics of LLMs in knowledge conflict scenarios, our understanding of the un- derlying mechanisms remains limited. Research into this question is not only crucial for designing superior context injection strategies and enhancing models' ability to integrate external knowledge but also provides theoretical guidance for building more reliable language models, advancing the widespread application of retrieval -aggeeetd LLMs.
To unravel these mechanisms, we propose entity aware probing a framework that traces knowledge propagation as entity flows through transformer layers. Traditional probes fail in dynamic settings due to fixed label constraints. We overcome this by introducing special token markers combined with low-rank adaptation (LoRA), enabling ad- hochoc entity tracing without altering base model parameters. This allows us quantify how contextual and parametric knowledge evolve across layers during conflict scenarios, Our analysis reveals three key insights: (I) Specialized routing: Contextual and parametric knowledge are processed by distinct attention heads, with minimal cross-type competition. (2) Asymmetric dynamics: Contextual knowledge emerges abruptly in early layers, while parametric knowledge accumulates gradually via MLPs. (3) Superpositional reconciliatton: Conflicting knowl- edge persists in residual streams, but aligned signals from multiple sources reinforce each other's s magnitude, directly influencing output selection.
Our contributions. Our main contributions are twofold: (1) An entity-aware probing framework for dynamic knowledge tracing enables ad-ho hoc entity relevance detection without modifying base model parameters. (2) Empirical insights into RAG dynamics offers foundation for future studies on knowledge interaction mechanisms.
2 Preliminaries
To understand how parametric knowledge and contextual knowledge reconcile within the model, we first reformulate the standard transformer architecture [35] from a residual stream perspective
7 26) (Section 2.1p. This reformulation allows us to isolate and analyze the information written into the residual stream by each component (MLP, attention, and individual attention heads), providing a foundation for studying their functional roles and actual knowledge propagation dynamics. We then introduce probing techniques (Section 2.2p as diagnostic tools to empirically study how models dynamically reconcile conflicting knowledge sources during this accumulation process.
2.1 Residual stream analysis of transformer computation
Standard transformer models process tokens through stack of identical layers, each containing self-attention and feed-forward sublayers. We begin by equivalently rewriting this computation as cumulative updates to a residual stream. Specifically, an I.-layer decoder-only transformer, the hidden state of 1-th token evolves as follows:
= (2) + AATT (t) + An, (t).
(1)
where Ant" (t) and Aij (t) denote the contributions written to the residual stream by the self- attention and MLP sublayers, respectively. In -augeeeell generation, externally retrieved knowledge is introduced into the model's internal processing stream, where these sublayers form separate pathways for knowledge propagation and serve complementary roles.
Parametric knowledge access. The emergence of parametric knowledge has been observed to strongly correlate with MLP sublayer outputs [10] 25 Specifically, denoting the MLP layer input as (,(t), the parametric knowledge access can be viewed as querying non- non-linear key-value memorics:
An mll (t) = Waown oWW up R(()),
where the input (,()) is first mapped to key space through matrix Wap and non-linear activation function o. Subsequently, the corresponding token-specific knowledge fragments A6,

---

The cocnel Zostacci daao bet Thupp de peoiker fetenety only Tropp dooon
Thwpp
MLP
peunsder, its
captane MIP
Attetie oppaaed
A
2M ()) (
Ulcerakeyy corates a hoe confflusho makkhal sggedd reseried
Figure/ 1: Schematic illustration of knowledge conflict scenario: The external document describes 'Trump' as the U.S. President, whereas the model parameters encode 'Biden' as the U.S. President.
queried through W Wownn and injected into the residual stream. This is a simplified description of parametric knowledge access; for more detailed modeling of memories querying, refer readers Geva et al. [10]. Meng et al. (25). We provide an illustrative example in Figure [ where for query "Who is the US president?", the parametric knowledge 'Biden' gradually emerges in the residual stream of relevant tokens (e.g.. 'US") through the injection of multiple MLP layers.
Knowledge routing by attention. The attention mechanism serves crucial pathway for transfer- ring knowledge between tokens. The content written into the residual stream by a single attention head is given by:
w-l (h W (T Vd
Aht attn. (+) (8) -:(+t) Softmax
w (m
where the four parameter matrices W((k) preferences. Specialized heads (e.g., retrieval heads [27] 38 that attend to external contexts and memory heads [17] accessing parametric knowledge) are found to exhibit differentiated sensitivity to these dual knowledge sources. The outputs of N attention heads are summed to form the attention
O.Q.K.V of the k-th attention head determine its attention
sublayer output (t) Σn
(t). Taking Figure as an example, both 'Trump' mentioned in the context and 'Biden encoded in the model parameters influence the model's response to the query "Who is the US president?" through the attention sublayer output Ani"! (t).
Ik- Ahe (kt)tt.(k))
Based on this formulation, we analyze the model's 's behavioral mechanisms under the RAG setting by probing the accumulated information in each layer's 's residual stream n,(1, MLP output AA, (t),
attention layer output Ah;"! (t), and individual attention head outputs An, (K)m..()) (t).
2.2 Probing as mechanistic diagnostics
Probing techniques 2 12, 26 serve as diagnostic tools to analyze how linguistic properties and knowledge components are dynamically encoded in neural representations. A probe fo: Rddace y typically implemented as a lightweight classifier, is trained to decode target properties y E y (e.g.. POS tags (34, named entities [15), or knowledge veracity labels (28)) directly from the residual stream, without modifying the base model's parameters.
Limitations of traditional probing. The classification head fo of traditional probes is static it is trained to recognize only a fixed set of labels ) (e.g., predefined POS tags or entity types) during training. This rigidity makes it brittle when handling ad-hoc probing targets at test time. For example, to probe whether a residual stream contains information about a specific entity, you cannot feasibly train entity- sppcifi probe for every possible entity.

---

3 Probing framework for knowledge conflict internal mechanisms
3.1 Competing hypotheses on knowledge conflict
Given a query (e.g.. "Who is the current U.S. President?") and an external document D returned by retrievers, we define parametric knowledge as all entities encoded within the model parameters that are objectively relevant to answering while contextual knowledge comprises answer-relevant entities derived from D. These knowledge sources may conflict-for for instance, parametric knowledge might encode "Biden" whereas retrieved documents 'D might state "After January 2025, Trump becomes the new U.S. President." We investigate how language models process and reconcile competing knowledge sources through their residual streams. Specifically, we propose two competing hypotheses about the model's internal mechanisms for handling knowledge conflicts:
Attention competition hypothesis: Contextual and parametric knowledge compete for domi- nance through attention patterns, with the prevailing signal propagating through successive layers to determine the final output.
MLP arbitration hypothesis: Both knowledge types are initially aggregated in the residual stream, with MLP layers subsequently injecting modulation signals that suppress conflicting information before final prediction.
3.2 antity- probing architecture
To investigate these two hypotheses, we design an entity relevance probing task. Given an external context D. a query q. and a specific entity e, the objective is to probe the base LLM's 's' belief about "whether entity e is relevant for answering query 9." We follow the approach of Ye et al. 42 to resolve the limitations of traditional probing outlined in Section 2.2 adapting the probe So for entity relevance probing.
Specifically, as illustrated in Figure 2 we append a triple " [[SAAR]] (target_entityy [END]" after the external context D> and query q. Here, [START] and [END] are reserved special tokens that explicitly indicate the target entity to the probe. To enable the model to process this structured input while preserving inherent processing mechanisms unchanged, we introduce small trainable rank-8 LoRA update (13] exclusively to the input embedding layer. The base LLM parameters remain frozen during fine-tuning, with only the linear classification head and LoRA components updated to predict entity relevance at the [END] token position. This triple and LoRA update can be viewed as a form of "soft prompt" designed to elicit the model's conflict reconciliation behavior. For a discussion of the intuition behind this approach and its limitations, please refer to Appendix B.
To validate that the entity relevance predictions primarily emerge from the model's 's inherent knowl- edge processing mechanisms rather than the introduced lightweight adaptation, we conduct an ablation study comparing LoRA-enhanced probes on both instruct and randomly initialized LLM variants. As shown in Table [ the sharp performance gap between Probing and Probing( demonstrates that the predictive patterns predominantly originate from knowledge pathways encoded in their parameters, with the LoRA update serving merely as an interface for observation rather than introducing substantive functionalityy These findings align with the results of Ye al. 42
3.3 Probe training data construction
We construct probe training data by adapting the HotporQA-distractt dataset [41] leveraging its sentence-level human-annotated evidence to precisely identify entities required for answering queries. Each sample contains a multi-hop query and an external document D composed of 10 independent passages, where exactly two passages contain answer evidence while the remaining eight serve as distractors. We provide the two evidence passages and query q as input to Deepseek- beepseek-v3-022 v3-0324, prompting it to generate step-by-step answer q. We then prompt the model to explicitly list entities involved in its generated answer (prompt template in Table 3). Entities appearing in both the evidence sentences and the model's 's answer are labeled as necessary (Yes). We then extract all entities mentioned in 'D using the prompt in Table 4 designating those not identified as necessary as irrelevant (No). From 6,200 processed HotpotQA samples, each generates training instances <イ> N
8; =< D;, Qis (e, <1>>> 9i
1==1 >, where y; <)> E (Yes, No} indicates whether entity e, <i> is
relevant to answering q:-

---

Multiple Tamnffomer Layers
The US presidens is Trusi
Jr(AA SNOL!
7a(AA END
Liner Lager
Qvery
166N01
Who the US president
Allei
MLP
Probing
folh moms))
[START] (END)
Figure 2: Schematic illustration of the knowledge dynamics tracing formulation: We probe the information a specified entity (in this case, Trump) at three critical computation points within each Transformer layer: Attention Update, MLP Update, and Layer Output.
3.4 Knowledge dynamics tracing formulation
Let fa(NENDD) Rd + y denote the probe constructed in entity relevance probing task, where Y E { No) represents the model's belief about whether entity e is relevant for answering query To test the competing hypotheses outlined in Section 3.1 we systematically analyze the evolution of entity relevance beliefs by applying our probes at three critical points within each transformer layer's computation:
atte Altention-updute eniity probing: (END).)) reveals the direct impact of attention mecha- nisms on model beliefs, where the attention module routes both knowledge types through the preceding tokens' KV cache.
MLP-update entity probing:
fo (END,)) identifies the MLP module's s direct influence on belief formation, corresponding to the model's integration of conflicting knowledge sources.
Layer oulput entity probing: So (uqpEDD)))) tracks how entity e's information aggregates across layers through successive attention and MLP updates up to layer
4 Probe validation via attribution task
To validate whether our entity-aware probe accurately reflects the model's internal beliefs about entity relevance, we conduct controlled attribution task that evaluates critical properties: (i) whether the probe identifies entities truly relevant to query within retrieved documents D, and (ii) whether predictions derive from the base model's knowledge processing rather than the probe's parametric capacity.
4.1 Experimental setup
Task definition:
Given query q and external document D containing N independent passages {PI...,,N' ), the passage-level attribution task requires identifying passages containing answer evi- dence. For sentence-level attribution, we further pinpoint specific evidence-bearrnn sentences within each passage. We evaluate attribution performance on three established QA benchmarks -o 41. TriviaQA [88, and SQuAD 30 adapted for both passage-level and sentence-level attribution tasks. We measure attribution accuracy using standard information retrieval metrics: Precis siecssion, Recall and F1. Implementation details for constructing the attribution benchmarks and formal metric definitions appear in Appendix C
Please refer to Appendix G for examples of the dataset.
We address the attribution task through entity relevance probing. For each passage ps (or sentence 8j) in document D, we probe all mentioned entities E (p:).. Specifically, we apply the probe fe evaluate relevance for each entity E(p)), where relevance defined by:
Probing-based attribution:
Relevant(e ()) = (Yesle, g, D) Po(Nole, q, D)].
We mark the passage/sentence as an attribution target if contains any entity with Relevant (c) =
Three critical comparisons are established as follows: (1) Prompting: Instructs the base LLM to directly output attribution results through one-shot manual demonstrations, rep- resenting the simplest way to test its internal beliefs. The prompts we used are shown in Table 5 (2) Prompting( CoT): Builds upon (1) by incorporating entity information extracted by the base LLM into the prompt and adding CoT-style instruction that guides the model to reason through
Compared baselines:
5

---

Table 1: Citation recall (R), precision (P) and FI (F1) are standard metrics for evaluating attribution. Avg. denotes the mean F1 score across all datasets, reflecting overall model or method performance.
TriviaQA
SQuAD
HotpoiQA
Model
Attribation Method
Avg.
R
F1
R
P
GPT-4o
Prompting
93.0
92.5
92.7
99.5
99.5
99.5
76.3
82.6
76.9
89.7
Proprietary
Nodels
Decpseck-v3-0324
Prompting
92.0
90.8
91.2
100.0
100.0
100.0
77.1
81.4
76.1
89.1
GLM-4-plus
Prompting
79.0
62.3
66.0
100.0
99.4
76.6
74.6
72.2
79.3
Prompting
36.5
33.5
34.4
92.5
92.3
92.3
45.9
61.1
49.7
58.8
Qwen2.5-7B
Prompting/Col CoT)
72.0
65.0
67.1
96.0
94.9
95.3
62.8
73.2
64.7
75.7
Probing
83.6
78.5
87.4
93.8
51.3
50.0
Probingi Random)
64.2
20.2
30,8
88.8
10.2
18.0
38.2
7.7
12.8
20.3
Prompting
51.0
50.2
50.4
57.5
57.5
57.5
50.6
66.3
$5.0
54,3
Open-
Nodels
Qwen3-88
Prompting! CoT)
Probing
77.5
48.5
97.2
77.1
64.7
98.5
94.8
98.5
95.0
94.9
60.9
81.7
46.8
52.9
70.8
Probingi Random)
68.7
19.9
30.8
62.4
9.8
17.0
7.0
12.5
20.1
Prompting
42.5
35.9
37.8
48.0
45.4
46.3
31.2
40.5
33.0
Prompting( CoT)
36.5
35.4
35.8
91.5
91.3
91.3
58.7
74.0
62.3
63.1
LLMMA3.--RR
Probing
46.4
97.5
62.8
94.2
91.2
47.0
56.2
51.2
Probingt Random)
61.7
19.9
30.1
83.4
10.0
17.9
69.5
7.0
12.7
20.2
entity-citation relations before producing the final attribution. The prompts are shown in Table (3) Probing(Random) Reinitializes the base LLM's parameters randomly (with parameters frozen during probe training) while maintaining identical probing architecture, isolating the probe's capacity to learn relevance patterns independently of LLM's knowledge processing.
4.2 Results analysis
As shown in Table we evaluate mainstream open-source models (Qwen2.5, Qwen3, LLaMA3. I) and state-of-the-arr proprietary models on the attribution task. Our analysis reveals two key findings:
LLMs possess intrinsic belief's about entity relevance: For strong proprietary models, the simple Prompting method alone suffice to elicit accurate entity relevance judgments. Although smaller open-source models initially perform poorly, their entity relevance judgment significantly improves after more refined prompt design (Prompting - Prompting( CoT)), approaching or even surpass- ing some proprietary models. This suggests that these models internally possess knowledge of entity relevance, but this belief may require appropriate elicitation methods. The Probing approach achieves performance comparable Prompting(CoT) indicating that probing serves as an alternative elicitation method that can effectively extract entity relevance information directly from hidden states.
Probing results stem from the LLM's own heliefs rather than probe parameters: Our critical baseline- probing randomly reinitialized LLM demonstrates that attribution performance col- lapses near-random levels (average Fl of 20.3 vs. 73.0 for intact Qwen2.5). Precision metries plummet to 10.2 on SQuAD and 7.7 on HotpotQA, comparable to random selection from candidate passages/sentences. This confirms that successful attribution depends on the base LLM's 's knowledge processing mechanisms rather than the probe's parametric capacity.
Understanding knowledge reconciliation within LLMs
5.1 Experimental setup
To investigate LLMs' knowledge reconciliation mechanisms, we construct knowledge conflict scenarios based on the HotpotQA dataset. Since HotpotQA is derived from Wikipedia-thh the primary pretraining corpus for LLMs we assume its factual descriptions align with LLMs' parametric knowledge. For each query, we leverage sentence-level evidence from HotpotQA to replace original factual entities with fictional counterfactual entities, generating rewritten evidence and forming conflcttinducing external documents. Using prompt templates (Table 3). we create 786 samples, each formatted as =< Dy, Def. &. >, where D, denotes the original document from HotpotQA, Def denotes the counterfactual external document, denotes query, & denotes all mentioned entities, categorized into:
Evaluation Data:
and instruct Deepseek-v3-033

---

Der
(b). De Er
(c). Dr Dr
(d). Dz
0.50
Layer ID
0.36
0.54
0.52
29
Head
Head
Head
Head ID
Figure 3: Information intensity (measured by PRAUC scores) across all attention heads in LLaMA 3.1. In the titles of subfigures (a-d), the left side of the '-'' indicates the type of external documents, where Def represents counterfactual documents and Des U D represents the union of original and counterfactual documents. The right side of the arrow indicates the probing target, where & represents counterfactual entities (aligned with contextual knowledge in Def) and &g represents factual entities (consistent with both parametric knowledge and contextual knowledge in Dy).
parametric knowledge (E,): Factual entities from Dy that align with the model's internal knowledge for answering q.
contextual knowledge (Eef): Counterfactual entities from Daf that conflict with parametric
knowledge.
irrelevant knowledge (Eirr): Remaining entities in & (E) Eef) unrelated to resolving q-
Metrics: We employ the PRAUC (Precision-Rectt Area Under the Curve) 6 score to quantify the information intensity of parametric knowledge and contextual knowledge in residual stream. The metric robustly addresses class imbalance (relevant entities rare compared irrelevant ones). Specifically, query-relevant entities (& or Eet) are treated as the positive class, while unrelated entities (Eirr) form the negative class. PRAUC is computed using the probe fo's entity relevance scores Pa Yes e, q. DD.. A higher PRAUC score indicates that the probe can more significantly detect the corresponding knowledge the specified probing position.
5.2 Attention head analysis
We analyze the information intensity of all attention heads in LLaMA3.1, Qwen2.5, and Qwen3.0 models using probes. This section corresponds to the 'Attention-updart update entity probing' discussed in Section 3.4 formulation. Figure 3 presents the experimental results for LLaMA3.1, while results for Qwen2.5 and Qwen3.0 are shown in Appendix Figures 7 and 8 due to space constraints. All three models exhibit the following consistent patterns:
(i) Contextual and parametric knowledge are transferred by distinct attention heads: When feed- ing counterfactual context Pef (where contextual knowledge conflicts with the model's parametric knowledge) as external documents, subfigures (a) and (b) of Figure 3 measure the information inten- sity of contextual knowledge and parametric knowledge individual attention heads, respectively. The results reveal that these two types of knowledge are routed through separate sets of attention heads, aligning with findings from existing mechanistic interpretability studies on attention heads 17 27 338. Among the top-10 heads with the highest information intensity for each knowledge type, only 2 heads overlap in Qwen3.0, and head overlaps LLaMA3.1 and Qwen2.5. For clarity, we refer to these contextual heads (routing contextual knowledge) and memory heads (routing parametric knowledge).
(ii) Fundamental differences in transfer mechanisms between knowledge types: Subfigures (a) and (b) of Figure 3 highlight a key distinction: the information intensity of contextual heads significantly exceeds that of memory heads. This suggests that contextual knowledge transfer resembles explicit retrieval, where complete knowledge units are captured and propagated by small subset of attention heads. In contrast, parametric knowledge likely involves more intricate processing-ead memory head may transfer fragmented knowledge pieces, which require multi-layer aggregation to form coherent knowledge units. This might explain why the information intensity of any single attention head remains relatively low. We provide further evidence supporting this hypothesis in Section 5.3

---

(a). Attention Scores across Layers
(b). MLP Scores Layers
(c). Layer Scores
5,012
Der Eft
0.20
Der
0.22
Der
Average prauc score
0,000
De +
Average prauic score
De -
Average praui score
D2. +
0.35
0.13
0.005
M
2.005
0.30
0.19
5.004
0.08
0.03
0.002
0.00
30
15 Layer
20
25
30
15
20
25
30
Layer ID
Layer ID
Figure 4: Information intensity probing results (measured by PRAUC scores) for LLaMA 3.1 with counterfactual document Def as context. Def - &ef and Der + E represent the average informa- tion intensity of counterfactual entities and factual entities respectively.
(iii) Competition between same-type knowledge, but not across types: Subfigures (c) and (d) of Figure Bl present probing results under mixed contexts (D, UDer), where factual (parametric-aligee aligned) and counterfactual (conflicting) knowledge coexist. Here, contextual knowledge types (factual vs. counterfactual) compete for attention within contextual heads: the information intensity for both in subfigures (c) and (d) substantially lower than subfigure (a). However, the presence factual knowledge in the context has minimal impact on memory heads (parametric knowledge routing). For LLaMA3.1 and Qwen3.0, the average information intensity of top-10 memory heads slightly increases (S 5%), while Qwen2.5 shows minor decrease (S 5%). This indicates competition occurs only within the same knowledge type (contextual vs. contextual), not across distinct types (contextual vs. parametric).
5.3 Residual stream analysis
Building on the probing framework (Section 3.4b we analyze the information intensity of parametric
knowledge (Ef) and contextual knowledge (Eef) across transformer layers by probing Attention updates, MI.P updates, and layer outputs. Key findings from experiments on LLaMA 3.1 under
counterfactual documents Der and mixed documents Def U DD, (Figures 5 with Qwen2.5/3.0
results in Appendix Figures|9 12 reveal:
(i) Divergent propagation mechanisms for parametric vs. contextual knowledge:
Parametric knowledge (E,) exhibits low information intensity in early-layer attention and MLP updates but shows rapid amplification in later layers (Figure 4. This suggests frag- mented parametric knowledge gradually consolidated through multi-layer interactions, forming coherent units only near the output layer.
Contextual knowledge (E.r), in contrast, immediately detectable early attention updates, persists latently in residual streams, and is reactivated in later stages for answer generation. This reflects its direct encoding from external contexts.
Reconciliation Pathway: When replacing Des with mixed document Def U Dy (Figure 5). parametric knowledge (Eş) gains competitive advantage output layers compared to scenarios without contextual support (Figure 4 (c)). This indicates that:
(ii) Superposition
(1) Factual knowledge from multiple sources (parametric + contextual) can superpositionally reinforce their information intensity in the residual stream, enhancing their influence on final outputs.
(2) Conflicting knowledge (e.g., Eef) coexists in the residual stream without erasure, suggesting LLMs preserve opposing evidence rather than discarding it.
5.4 Evidence from Native Generation Dynamics
Although we aim to minimize interference, probing may inadvertently influence the model's gen- eration behavior. To further validate the knowledge reconciliation mechanisms identified through probing, we conduct two intervention experiments using LLaMA3.1 I in its native generation setting.

---

(a). Attention Scores across
(b). MLP Scores Layers
(c), Layer Scores
Deru Dr
6.16
Der Ect
Der Der
Average prauc score
De U D2
Day U Di -,
Der U 8r
0.22
0.325
M
M
0.30
0.130
0.00
Averaon
0.075
0.04
0.050
0.00
20
30
10 15
20
25
30
Layer ID
Layer ID
Layer ID
Figure 5: Information intensity probing results (measured by PRAUC scores) for LLaMA 3.1 using the union of original and counterfactual documents Def U D as context. Def U Dj + &ef and Dej UD + & represent the average information intensity of counterfactual entities and factual entities respectively.
Table 2:
Causal impact of zeroing top-10 10 context heads and memory heads on knowledge reconcilia- tion behavior under native generation mode. The proportions of the don know and other options are combined under Other column in the table.
Experimental Setting
Factual Answer
Counterfactual Answer
Other
Original Model
22.70%
54.60%
22.70%
w/o Context Heads
34.90%
43.60%
21.50%
w/o Memory Heads
9.20%
65.50 %
25.30%
For ease of evaluation, formulate the test in multiple-choie format, where factual, answer, counterfactual answer, don't know, and other are randomly assigned to options A, B, C, D.
(1) Causal Intervention via Zeroing Attention Heads. Given counterfactual document context, we zero out the identified top-10 10 context heads and memory heads to observe the impact of this intervention LLM response generation. The prompt used is shown in Table I0] As shown in Table 2, removing context heads reduces contextual influence (54.6% - 43.6 () and increases reliance on parametric knowledge (22.7% - 34.9%). Conversely, removing memory heads weakens parametric influence (22.7% + 9.2%) and strengthens contextual responses (54.6% -> 65.5%) These results provide causal evidence that the identified heads perform the reconciliation function, as described in Section 5.2
(2) Attention Redistribution via Prompt Manipulation. We design three different prompts to enhance the model's attention to counterfactual documents in context: Duplicate and ap- pend, Insert prompt randomly, and Tag and append. Detailed descriptions of these strategies are provided in Appendix H In the left panel of Figure 6 we provide both factual and counter- factual documents context to the model. After applying the three strategies above, the pro- portion of counterfactual_ answers increase significantly, from 27.7% to 45.3%, 40.9%, and 41.2%, respectively. This indicates that there exists competition between factual answer and a
counterfactual answer from the context, where strengthening one weakens the other. In the right panel of Fi the context contains only counterfactual documents. After applying the three strategies above, the proportion of counterfactual _nsser show no significant change. This suggests that there no direct competition between counterfactual. answer from the context and factual, answer from the model's 's parameters; strengthening one does not notably affect the other.
5.5 Discussions
Our experimental findings reveal nuanced interactions between the two hypothesized mechanisms in Section [.I] suggesting a hybrid processing strategy rather than exclusive adherence to either hypothesis.
Attention Competition Hypothesis Revisited. The hypothesis posits that contextual and parametric knowledge compete through attention patterns. Our results partially support this view but with critical

---

factual_anm aasswer
counterfactual_com annwer
don't know
other
70
70
60
$1.1
$1.3
50
50
65.3
66.9
40.9
13.2
40
40
30
30
22.8
23.4
28.2
20
11.2
167
10.5
10
1.
1.1
6.9
Factual Counterfactual
Duplicate and appand
Insert prompt randomil
Tag and append
Only counterfactual
Duplicate and appand
Insert prompt randomly
Tag and appand
Figure 6: Knowledge competition between same-type knowledge (left panel), but not across types (right panel). "Factual+Couutteffaauu indicates using both factual and counterfactual documents as context, while "Only counterfactual" refers to using only counterfactual documents as context. Three strategies are (see Appendix H for detailed descriptions) designed to direct the LLM's attention toward counterfactual documents.
qualifications: (1) Distinct routing pathways: The separation of contextual parametric knowledge into dedicated attention heads (Section 522(i) supports the notion of parallel processing rather than direct competition. This suggests attention mechanisms implement a "division of labor" strategy rather than winner- takesslll competition. (2) Within-rype competition: The observed suppression between factual/coumterfactus contextual knowledge in mixed contexts (Section (.22() aligns with attention-level competition, but only within the same knowledge type. This implies attention heads implement intru-type competition rather than cross-type rivalry.
MLP Arbitration Hypothesis Revisited. The hypothesis suggests MLP layers resolve conflicts
through suppression of contradictory signals. Our analysis reveals more complex dynamics: (I) Per- sistence of Conflicting Signals: Despite MLP operations on the residual stream, conflicting contextual and parametric knowledge maintain residual presence across layers (Section [5.3 ()), contradicting
the suppression hypothesis. (2) Superpoxitional reconciliation:: The enhanced parametric know moowledge intensity under mixed contexts 5.3t)on indicates MLPs may implement additive superposition of aligned evidence from multiple sources, with the accumulation magnitude directly determining its
influence on final outputs.
Conclusions
This study presents mechanistic understanding of how LLMs reconcile conflicting parametric and contextual knowledge RAG frameworks. By introducing an entity-aware probing methodol- ogy, uncover three fundamental principles governing knowledge dynamies: (1) Attention head specialization enables parallel processing of distinct knowledge types through dedicated routing pathways; (2) Asymmetric propagation mechanisms reveal contextual knowledge emerges abruptly via attention while parametric knowledge accumulates gradually through MLP layers; and (3) Super- positional reconciliation demonstrates that conflicting knowledge persists in residual streams, with aligned multi-source signals reinforcing each other to determine output selection. This work provides both theoretical foundations and practical guidance for building reliable and interpretable retrieval-augmeented language models.
Acknowledgements
The authors wish thank the anonymous reviewers for their helpful comments. This work was par- tially funded by National Natural Science Foundation of China (No.62476061, 62441602, 62206057). Shanghai Rising-Star Program (23QA1400204) 1400200), and Natural Science Foundation of Shanghai (23ZR1403500)
10

---

References
[1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using lincar classifier probes. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Workshop Track Proceedings. OpenReview.ne., 2017. htttphhtt://opeee http:/ppprreeiiee.eeeffrrrrr Id=HJ4-rAVt]
[2] Nora Belrose, 7ach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. 2023. URL
https://arxiv. org/ abaaaba////// 0011122001112 08112
[3] Baolong Bi, Shenghua Liu, Yiwei Wang, Yilong Xu, Junfeng Fang, Lingrui Mei, and Xueqi Cheng- Parameters vs. context: Fine-grained control of knowledge reliance in language models, 2025. URI.
https://arxiv. org/ ab0// 15888
Hung-Ting Chen, Michael J. Zhang, and Eunsol Choi. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence, 2022. httpp//ss://arxi org/ hrg/abs/2220 13701
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models, 2024. URL /tttps:////ariii /
abs/2309. 2383833330038
[6] Jesse Davis and Mark Goadrich. The relationship between precision-recall and curves. In Proceedings of the 23rd International Conference on Machine Learning. ICML '06, page 233-240, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 1444444/144444444888888
URL http:: org/ 10. 1145/1144844, 1143874
[7] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfeeld-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathe- matical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer- circuits. w/fraaewwor//idee......
[8] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. eetrreeval-augmene generation for large language models: survey, 2024, URL attps://aarie www.oog/////..o.oooo/.o.oo...ee/ 2212. 10997
[9] Zorik Gekhman, Jonathan Herzig. Roee Aharoni, Chen Elkind, and Idan Szpektor. Trueteacher: Learning factual consistency evaluation with large language models, 2023. UtL/httts:///rrtt org/ abaaabaaa///
11171
[10] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer ferd-forward layers are keyuv memories. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Pro- ceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 1/212663//22211..1 1000011emnll-mmnn main.446. UtL/https:// battpe://aclanthooogg. orgo org/2021. emnlp mmnlp-main 446/
[11] Reto Gubelmann and Siegfried Handschuh. Context matters: A pragmatic study of PLMs' negation understanding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 4602-4621, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/11/ 2022.acl-lomg.315. URL https:::::// http://aaaaaahlloo/oo/ org. /0222022222222 acl- long 315/
[12] John Hewitt and Christopher D. Manning A structural probe for finding syntax in word representations. In Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4129-4138 4138, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 118856//1/N1111../ 1419.
URL http: .rg/
N19 1419/
[13] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL
[14] Yizheng Huang and Jimmy Huang. A survey on retrieval- text generation for large language models, 2024. httph:tt///aaa //arxiv..///////aa///iaaaiivivva wttp.o/////../0000/000

---

[15] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Anna Korhonen, David Traum, and Lluís Marquez, editors. Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 3651 1651-3657, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19 13366 BtRhhttps://acat htt////cccnthhooogyyy org/P19- 1356/
[16] Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, and Jun Zhao. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. In Nicoletta Calzolari. Min-Yen Yen Kan. Véronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. LREC/COLING 2024, 20-25 May. 2024. Torino, Italy. pages 16867-16878. ELRA and ICCL, 2024, http::ttt//a/aatttaaaeeaae 1466
2024 wrec-meeeeee.
[17] Zhuoran Jin, Pengfei Cao, Honghang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang. Kang Liu, and Jun Zhao. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge confficts in language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguisties: ACL 2024, pages 1193- 1215, Bangkok, Thailand. August 2024. Association for Computational Linguistics. doi: 10.18653/v120044 findings ac1.70.
URL https: h/ala//ahoaanaooooyooooo wwg/org///////.....//..//./g/ 2024. findings- act. 70/
[18] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URL https://arxiv org/ abs/ 1705
03551
[19] Mintong Kang, Nezihe Merve Gürel, Ning Yu, Dawn Song, and Bo Li. C-rag: Certified generation risks for retrieval-augmennt language models, 2024. URL URL https://arxiv org/ h/t///b//// aa//a/ab///2//22 2402.03181
[20] Nora Kassner, Oyvind Tafjord, Hinrich Schütze, and Peter Clark. BeliefBank: Adding memory to pre-trained language model for systematic notion of belief. In Faric-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors. Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 8849-8861, Online and Punta Cana. Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653//1/20211.mmm URL htRLsttss://// ptttcp:://acaanthol.oo oor.olllgooooooooooooooooooooore 2021. emm main m
[21] Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. Backward lens: Projecting language model gradients into the vocabulary space, 2024. URL httpa:
wttp///////////a////////// 12865
[22] Daliang Li. Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu. and Sanjiv Kumar. Large language models with controllable working memory, 2022. URL https
7/arxiv. ftrg/abs/22115 2abbb//221..011111
[23] Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts question answering. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052-7063, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational I Linguistics. doi: 10. 1266:////222.eem//mmmaaa //aclanthologg..rrg wtt./221....... emmlp-main. 565/
URL URt.https
[24] Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity- based knowledge conflicts in question answering, 2022. URL htttptt///ariiiiii www.:/wwwwoovooaaaaaea http//111900000000
[25] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual as- sociations in gpt. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, edi- tors, Advances in Neural Information Processing Systems, volume 35. pages 17359-17372. Curran As- sociates, Inc., 2022. URL httpss:/ neurips 11125555582237766666663333333333 Papp Conference, pdf
[26] nostalgebraist. Interpreting GPT: The logit lens.
https://wwv.essaws cataswrong.com/poseeet
the- tie-thhe-lgtt. -enss lens 2020. Less WronW.
[27] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield- Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URLhttps: : arxii. www.awwwwww.aaaaeee.eeeeeoweeeee www./bs./////eeooooo/o..o...eeee /200s//222... 11895
12

---

[28] Fabio Petroni, Tim Rocktäschel. Sebastian Riedel. Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models knowledge bases? In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pages 2463 2473, Association for Computational Linguistics, 2019. doi: 10.18653/V 1/D/// 1250. URL https: //do:. org 10. 18663/11/1111 1250
[29] Quang Hieu Pham, Hoang Ngo, Anh Tuan Luu, and Dat Quoe Nguyen. Who's who: Large language models meet knowledge conflicts in practice, 2024. URL htt..tttpaa//aiiii org/ abe/a410.
[30] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. httpp::////// owtperiiii..rggaooo abe.//06./66..... 05250
[31] Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model?, 2020. URL htttpp::/wawwi.or////a/a//
08910
[32] Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard. and Ian Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism, 2023. URL https: //aaas:////xxii/ org/ abs 2310. 16270
[33] Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. LLM-check: Investigating detection of hallucinations in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https //openreview net met//Forummid--Yyev3....
[34] Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Reprexentations, 2019. URL httphtt//ppeereeeiiiie net het//forummddd-SSssgneeee
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. https:/ts:///ant org/ abs/ 1706
03762
[36] Fei Wang, Xingehen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan Ö. Ankk. Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models, 2024. https
//arxiv.org .org abaa 22110/41000000 07176
[37] Wenhao Wa, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu, Retrieval head mechanistically explains long-context factuality, 2024, httpss/::///ax org/ abs//404. 15574
[38] Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp:://pperreiiess
[39] Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Reveal- ing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference Learning Representations, 2024. URL http:/sss//ooeeereee...ccoo www.:eee/orrrrt wuw.ooruuuuidaaAuUUUU/AA wttp://wwwwreeeeeeeeoe/oerrrrrre
[40] Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang. Hongru Wang, Yue Zhang, and Wei Xu. Knowledge conflicts for 1ms: A survey, 2024, URL htttht///a//xiiv org./og...................
[41] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL Uttp::ttt:://ar/i org/ aba/ 1809 .9600
[42] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade- school math and the hidden reasoning process. In The Thirteenth International Conference on Learning Representations, 2025. URL /ttp::/ps////peeree//
net wwwooooomndnnnnnooooo
[43] Tianzhu Ye, Li Dong, Yuging Xia. Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer, 2025. URL LRL htt.pp:://aadv..rr/abbe.ss///// 06268
[44] Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasckar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint-satisfaction lens on factual errors of language models, 2024. URL
org/ aaaaabb//33/ 15098
13

---

[45] Jun Zhao, Can Zu, Xu Hao, Lu, Wei He. Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. LONGA- Achieving question answering for 128k-token-long documents through multi-agent collaboration. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16310-1615 16324, Miami, Florida, USA, November 2024, Association for Computational Linguisties. doi: 10.18653/w/22224.mmm main.912. URL URLttttps:://// pttttp::/aacaanthllogooooo
GENT:
org/2024 emnlp- main 912/
[46] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Jie Jiang, and Bin Cui. Retrieval- generation for ai-generated content: A survey, 2024. URL httth: //axiv. org/ abs 2402 19473
[47] Wenxuan Zhou, Sheng Zhang. Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings the As- sociation for Computational Linguistics: EMNLP 2023, pages 14544-14556, Singapore, December 2023. Association for Computational I Linguistics. doi: 10. 1/218 findings-ddnnnnnn 961. URL. htcps://aallantholloy 2023
14

---

Technical Appendices and Supplementary Material
Limitations
Our study has several limitations that warrant careful consideration. First, while our entity-aware probing framework demonstrates effectiveness in tracing knowledge flows within decoder-only transformers like LLaMA and Qwen series, its generalizability to other architectures (e.g., encoder- decoder models) or significantly larger models remains untested. Second, our experiments use synthetic counterfactual documents constructed via LLM-based entity substitution. While this ensures controlled evaluation, real-world retrieval contexts may contain more subtle conflicts involving temporal reasoning, numerical discrepancies, or conflicting logical implications not merely entity- level replacements. Future work should validate our findings on naturally occurring knowledge conflicts from dynamic knowledge sources.
A Related work
Detrieval-augmenttr generation and knowledge conflicts Large language models often face challenges such as outdated knowledge and hallucinations 333 Retrieval-Augmented Generation (RAG) offers a solution to these issues by retrieving and integrating external data as supplement External knowledge introduces challenges, especially conflicts with model-parametrii knowledge 40 31. Such conflicts introduce source-level uncertainty and may significantly degrade
46
19
model performance 40 29 While some studies suggest a model preference for parametric memory 24]. others highlight greater reliance on external knowledge [4]. Naturally, various methods have However, how models handle such conflicts
been proposed to mitigate these conflicts 36 22
remains an area for further research.
Internal Probing for Model Interpretabilityy A common approach interpretability is probing Yuksekgonul et al. [44). Wu et al. 3 investigated the behavior of attention mechanisms when the model is generating factual answers. There are also approaches known as lens methods. These include LL-based methods [26] that analyze information flow within the attention mechanism [32) and approaches that incorporate Logit Lens during backpropagation (2.. We effectively improve upon these probing methods and adapt them to QA tasks, showing that they yield both promising performance and effective interpretability.
model internals 44 37 5 43
B Intuition and Limitations of Entity-aware Probing
Where is the capital of France? A: The capital of France is _ In this case, the candidate entity tokens include "London" (contextual knowledge) and "Paris" (parametric knowledge). Here, the answer prefix "The capital of France is" essentially serves to clicit the model's conflict reconciliation behavior. In standard generation, not every token prediction involves knowledge-intens resolution. Many tokens (e.g., articles, conjunctions, formatting) are generated through relatively shallow pattern matching. The LoRA updates and the entity triples appended after the document and question in our work can essentially be viewed as a trainable soft prompt. It serves a similar function to "The capital of France is" in eliciting the model's conflict reconciliation behavior. However, we must acknowledge that this approach increases the risk of altering the model's internal behavioral patterns. We believe that constructing more flexible probing methods with minimal intervention in model behavioral patterns represents an important open problem worthy of continued exploration.
First, we consider a simple case: Document: The capital of France is London. Q:
reasoning or conflict
C Attribution dataset adaptation and evaluation metrics
Dataset adaptation: We build our evaluation using three established QA benchmarks, adapting them to support passage- and sentence-level attribution tasks. HotpotQA serves as both the training set of the probe and a sentence-level attribution testing set (see Section 3.3 for training details), while SQuAD and TriviaQA are exclusively used for passage-level attribution evaluation (without dataset-specific probe fine-tuning).
15

---

Designed for multi-hop reasoning, HotpotQA provides sentence-level attribution target through cuman--nnnoaate evidence sentences. Each query is paired with 10 passages. of which exactly 2 contain gold evidence sentences. 'he remaining 8 passages act as natural distractors, enabling fine-grained evaluation of sentence attribution without requiring synthetic negative sampling.
Hotpotqa 41:
TriviaQA [18] For passage-level evaluation, we construct input documents by combining the gold passage from each test sample with 4 randomly selected passages from other test instances.
Following similar passage-level evaluation protocol TriviaQA but with shorter passages, we aggregate each gold passage with randomly sampled irrelevant passages to form evaluation documents.
SQuAD 30
We evaluate the results using three standard metrics: Precision, Recall 1 and F1. Given that the ground truth passage/sentence indices are known, we compute the metrics as follows:
Evaluation metries:
Precision measures the proportion of correctly identified relevant passages among all predicted relevant, computed |Predicted n GroundTruth/ /
Recall measures the proportion of correctly identified relevant passages among all truly relevant ones, computed as Prrdicctd n GroundTruth) /GroundTruthh.
F1 score the harmonic mean of precision and recall, calculated as (2 Precision (ecall))(Precision)) + Recall).
D Implementation Details
The experiments were conducted using the Openrlhf framework with DeepSpeed for distributed training. The model was trained with BF16 mixed precision and Flash Attention optimization to accelerate computation and reduce memory overhead. All model parameters frozen except for the LoRA adapter injected into the imput embedding layer. Training utilized global batch size of 256 with a micro-batch size for memory-efficient gradient accumulation. The Adam optimizer was configured with learning rate of le-5 and ZeRO Stage 2 optimization. The model was trained for 2 epochs. The input sequence length was capped at 3,000 tokens. Training the probe requires approximately days 8 A100 GPUs, and testing single dataset takes about I hour.
E
Visualization of PRAUC Scores for Qwen2.5 and Qwen3
(a). D,
(b). Dr
(c). D. Dr
(d). D. Dr
0.26
6.25
Layer ID
0.34
15-
0.23
0.02
20
10
15
20
25
10 15
20 25
-00
15
22
25
15
20
25
Head
Head
Head
Head ID
Figure 7: Information intensity (measured by PRAUC scores) across all attention heads in Qwen 2.5. In the titles of subfigures (a-d), the left side of the *-²' indicates the type of external documents, where Def represents counterfactual documents and Des D, represents the union of original and counterfactual documents. The right side of the arrow indicates the probing target, where Egf represents counterfactual entities (aligned with contextual knowledge in Def) and & represents factual entities (consistent with both parametric knowledge and contextual knowledge in Dy).
F
Full List of Prompts Used in the Experiments
16

---

Der Ect
(b). Er
(c). Dr Dr
(d). Dz Dr
0.88
10
0.28
0.55
Layer
15-
0.53
9.02
0.33
Head
Head
Head
Head ID
Figure 8: Information intensity (measured by PRAUC scores) across all attention heads in Qwen 3.0. In the titles of subfigures (a-d), the left side of the '-4' indicates the type of external documents. where Def represents counterfactual documents and Def U D represents the union of original and counterfactual documents. The right side of the arrow indicates the probing target, where & represents counterfactual entities (aligned with contextual knowledge in Def) and & represents factual entities (consistent with both parametric knowledge and contextual knowledge in Dy).
(a), Attention Scores across Layers
(b). MLP Scores across Layers
(c). Layer Scores
0.000
0,530
Der Edf
D Ect
Der
Bveraae prauc Store
0.007
5.005
Det Er
0.925
Der Er
Store
0.25
Der
0.000
8.035
0.15
fveraae
3,000
0.030
0.30
0,002
0.805
0.28
5.000
0.000
0.30
Layer ID
ID
Layer
Figure 9: Information intensity probing results (measured by PRAUC scores) for Qwen 2.5 model with counterfactual document Def as context. Def - Eef and Def + &y represent the average information intensity of counterfactual entities and factual entities respectively.
Table 3: Prompt template for data generation.
Example 1:
Passage A: The 2015 Diamond Head Classic college basketball tournament ... Buddy Hield was named the tournament's MVP. Passage B: Chavano Rainier "Buddy" Hield is Bahamian professional baskethall player for the Sacramento Kings of the NBA...
(a), Attention Scores across Layers
(b). MLP Scores across Layers
(c). Layer Scores
5.000
Dr
Der
Average prouu score
0.002
DA
0.175
0.150
De -
dger Store
0.35
0.20
Dr -
5.005
Average praue
8.115
5.008
8.100
0.20
3.001
0.15
0.30
0.025
5.001
0.000
0.20
35
10
20
30
10
25
35
Layer ID
Layer ID
Layer
Figure 10: Information intensity probing results (measured by PRAUC scores) for Qwen 3.0 model with counterfactual document Def as context. Def + E and D.f + E represent the average information intensity of counterfactual entities and factual entities respectively.
17

---

Question: Which team does the player named 2015 Diamond Head Classic's MVP play for? Sacramento Kings
Answer:
Analysis:
Step 1: Using the clue "2015 Diamond Head Classic," identify in Passage A that the MVP "Buddy Hield."
Step Using the clue "Buddy Hield," find in Passage B that he currently plays for the "Sacramento Kings."
Therefore, the answer is the Sacramento Kings.
The entities involved in the aforementioned analysis, in logical order, are: ["2015 Diamond Head Classic", "Buddy Hield", "Sacramento Kings"]
The entities that not reffered to in the questions are: ["Buddy Hield", "Sacramento Kings").
Counterfactual Data Synthesis:
Let's conduct counterfactual analysis by replacing the entities that were mentioned in the analysis but did not appear in the Question.
As we mentioned above, these entities are: ("Buddy Hield", "Sacramento Kings"]
We will replace them with fictional entities: ["Alex Taylor", "Vancouver Bears"]
Revised Passage A: The 2015 Diamond Head Classic was a college basketball tourna- ment Alex Taylor was named the tournament's MVP.
Revised Passage B: Alex Taylor is a Canadian professional basketball player for the
Vancouver Bears the NBA...
Revised Question: Which team does the player named 2015 Diamond Head Classic's MVP play for?
Revised Answer: Vancouver Bears
Revised Analysis:
Step 1: Using the clue "2015 Diamond Head Classic," identify in Passage A that the
MVP is "Alex Taylor."
Step 2: Using the clue "Alex Taylor," find in Passage B that he currently plays for the "Vancouver Bears."
Therefore, the answer is the Vancouver Bears.
The entities involved in the revised analysis, in logical order, are: ["2015 Diamond Head Classic", "Alex Taylor", "Vancouver Bears"]
The entities that reffered to in the questions are: ["Alex Taylor", "Vancouver Bears"]
Example 2:
Paragraph A: Several current and former members of the Pittsburgh Pirates John Milner,
Dave Parker, and Rod Scurry...
Paragraph B: David Gene Parker, nicknamed "The Cobra", is an American former player Major League Baseball...
Question: Which former member of the Pittsburgh Pirates was nicknamed "The Cobra"?
Answer: David Gene Parker
Analysis
Step 1: Using the clue "Pittsburgh Pirates" from the Question, identify in Paragraph A the
list of current and former members that includes "John Milner," "Dave Parker," and "Rod Scurry."
Step 2: Using the clue "nickname 'The Cobra"" from the Question, identify in Paragraph that "David Gene Parker" is nicknamed "The Cobra."
Step 3: Connect the information that "Dave Parker" from Paragraph A is the same individual as "David Gene Parker" from Paragraph B who is nicknamed "The Cobra."
18

---

Therefore, the answer David Gene Parker.
Since 'David Gene Parker' and 'Dave Parker' refer to the same entity, we will use 'David Gene Parker' to refer to it in subsequent analysis.
The entities involved the aforementioned analysis, in logical order, are: ["Pittsburgh Pirates", "David Gene Parker", "The Cobra"]
The entities that not reffered to in the questions are: ["David Gene Parker"]
Counterfactual Data Synthesis:
Let's conduct counterfactual analysis by replacing the entities that were mentioned in the analysis but did not appear in the Question.
As we mentioned above, these entities are: ["David Gene Parker"]
We will replace them with fictional entities: ["Alexander 'T. "Taylor"]
Revised Paragraph A: Several current and former members of the Pittsburgh Pirates John Milner, Alex Taylor, and Rod Scurry..
Revised Paragraph B: Alexander T. Taylor, nicknamed "The Cobra," is an American former player in Major League Baseball..
Revised Question: Which former member of the Pittsburgh Pirates was nicknamed "The
Cobra"?
Revised Answer: Alexander T. Taylor
Revised Analysis:
Step 1: Using the clue "Pittsburgh Pirates" from the Question, identify in Revised Paragraph the list of current and former members that includes "John Milner," "Alex Taylor," and "Rod Scurry."
Step 2: Using the clue "nickname 'The Cobra"" from the Question, identify in Revised Paragraph B that "Alexander T. Taylor" nicknamed "The Cobra."
Step 3: Connect the information that "Alex Taylor" from Paragraph A is the individual as "Alexander" T. Taylor" from Paragraph B who is nicknamed "The Cobra." Therefore, the answer is Alexander T. Taylor.
Entities involved in the revised analysis, in logical order, are: ["Pittsburgh Pirates", "Alexander T. Taylor", "The Cobra")
Entities that are not referred to in the questions: ["Alexander T. Taylor"] You are a logical analysis expert. Please provide your analysis and counterfactual data synthesis of the following material in the manner described above. In your analysis, the entities should be presented in the form of list.
Passage A:
Content)
Passage B: (Passage_Content) Content) Question: (Question Content) Answer: Ainswer_Content Analysis:
Table 4: Prompt template for extracting entities from a sentence.
Example 1:
Sentence:
The thank you letter is an opportunity for you remind the interviewer why you are the best fit for the position and to address any remaining issues.
Answer: [thank you letter, interviever, best fit, position, remaining issues]
The answer in every example a list that contains the entities in the sentence. Entities refer to specific nouns or noun phrases with clear meanings that play key role in the sentence. Therefore, these words are extracted as entities.
Analysis:
19

---

You are language expert. Now, I will give you sentence, and you need to output entity list for the sentence the format similar to the example. If there are no entities, return an empty list.
<sentence> (Sentence) </sentence>
Table Prompt template for citation selection in reading comprehension tasks.
You are an expert reading comprehension. You will be given passage and a question, where the sentences or paragraphs in the passage are segmented by identifiers such as <cO>, Please read the document and question carefully, analyze which sentences or paragraphs need be cited answer this question, and then return a list of citations in the format: <cite> [citation index only numbers] </cite>.
Example:
[Document Start]
<CO>In the last two decades, several hundred planets have detected beyond our Solar-system.
<Ci>Most of these extra-solar planets orbit sun-like stars. <C2>A small number have been detected around stars that are in their late evolutionary state.
<C3>The phase directly after the RGB stage, the Horizontal Branch (HB), however, is still unexplored;
<C4>Besides its evolutionary stage, star's chemical composition appears to be major indicator of its probability. [Document End]
[Question]
How does the metallicity of PSR B1620 b, at around 1% of the Sun's, compare to other known exoplanet host stars?
[Remind]
Please read the document and question carefully, analyze which sentences or paragraphs need to be cited answer this question, and then return list of citations the format: <cite> [citation index only numbers] </cite>.
[Answer Citation List] (cite>\0,33
Now get ready to handle the following test case.
[Document Start]
<context>
[Document End]
[Question]
<question>
[Remind]
Please read the document and question carefully, analyze which sentences or paragraphs need to be cited to answer this question, and then return list of citations in the format: <cite> [citation index only numbers] </cite>.
[Answer Citation List]
20

---

(a), Attention Scores across Layers
(b). MLP Scores Layers
(c), Layer Scores
Det U Dr
Der Eor
Der Dr
Det Dj
0.003
De Dr &
Det Dr &
Average pratec score
5,0053
Average praui score
0.13
5,0040
5,0030
0,0023
0.003
0.05
10
15
20
Layer
Layer ID
Layer
Figure 11: Information intensity probing results (measured by PRAUC scores) for Qwen 2.5 model using the union of original and counterfactual documents Daf U Dj as context. Def U D - Eef and Des UD, + E represent the average information intensity of counterfactual entities and factual entities respectively.
(a). Attention Scores across Layers
(b). MLP Scores across Layers
(c). Layer Scores
6.16
Der Eet
De Th Eet
0.23
17. Dy Ker
0.007
Der D
Der U Diy Er
12 Dp
Average prauc Score
Average praui srore
W
0.30
0.20
0.008
2.00
0.06
Average
0.12
0.004
0.02
0.00
0.03
20
Layer
Layer
Layer
Figure 12:
Information intensity probing results (measured by PRAUC scores) for Qwen 3.0 model using the union of original and counterfactual documents Def U DDj as context. Dof UD, + E and Def UD, + E, represent the average information intensity of counterfactual entities and factual entities respectively.
Table 6: Prompt template for entity-aware citation selection in reading comprehension tasks.
You are an expert in reading comprehension and entity- citation analysis. You will be given a passage with sentences segmented by identifiers such as <c0>, entity information for each sentence, and question.
Please read the document, entity information, and question carefully. Use step-by-step reasoning to analyze which sentences paragraphs need to be cited to answer this question, considering both the semantic content and the entities present each sentence. Then return list of citations in the format: <cite> [citation index only numbers]</cee </ite>.
Example:
[Document Start]
<C0>In the last two decades, several hundred planets have been detected beyond our Solar-system. <Ci>Most of these extra-solar planets orbit sun-like stars. <C2>A small number have been detected around stars that are in their late evolutionary state, such as Red Giant Branch (RGB) stars and pulsars. <C3>The phase directly after the RGB stage, the Horizontal Branch (HB), however, is still unexplored; therefore, there is no empirical evidence for whether close-in planets, i.e., those with semi emmi- axes less than 0.1 AU, survive the giant phase of their host stars.
21

---

<C4>Besides its evolutionary stage, a star's chemical composition appears to major indicator of its probability for hosting planet. <C5>Previous studies, 8.8.,, showed that main-sequence (MS) stars that host giant planets are metal-rich. <C6>This finding supported by the large exoplanet search surveys around MS stars reporting connection between planet frequency and metallicity, and a survey of 160 metal-poor main-sequence stars finding no evidence for Jovian planets.
<C7>Until now, only very few planets have been detected around stars with metallicities low [Fe/H]= $-1$, i.e. 10% of the sun's metallicity. <C8>The detection of PSR B1620 b, a Jovian planet orbiting pulsar in the core of the netal-poor globular cluster M4 ([FeHH)=$-1.29), suggests, however, that planets may form around metal-poor stars, although the formation mechanism of this particular planet might be linked to the dense cluster environment.
<C9>Wl used the Fibre-fed Extended Range Optical Spectrograph (FEROS), a high-resolution spectrograph ($R=48, 000$) attached to the 2.2 meter Max-Planck Gesellschaft/Buroppe telescopel, to observe the star <C10>This star is classified as a red (RHB) star (Fig. 1) and its metal content [Fe/H] _man i.e. about 1% that of the Sun. <C11>So far, is not known as binary system. <C12>Detailed stellar parameters can found Supporting Online Material, Section [texti].
Southern Observatory (MPG/ESO)
-209,
<C13>Previous radial velocity (RV) measurements showed a systematic velocity of about 300 (km {-1} with respect to the Sun, indicating that the star belongs to the stellar halo. <C14>Indeed, the star has been connected to the Helmi stream, group of stars that share similar orbital parameters that stand apart from those of the bulk of other stars in the solar neighborhood. <C15>The Helmi stream stars move on prograde eccentric orbits (R_peri"7 kpc, R_apo" 116 kpc) that reach distances up to \vert 2lvert_max"13 kpc above and below the galactic plane. <C16>From that, it has been concluded that these stars were once bound to satellite galaxy of the Milky Way that tidally disrupted 6-9 Ga ago.
<C17>, with mean metallicity estimate of [Fe/H]=-2.1, is far metal-poor than any previously known exoplanet hosting star (Fig. 3) <C18>For the existing theories hot giant planet formation, metallicity is an important parameter: in particular, it is fundamental for the core-accretion planet formation model. <C19>It might be that initially, in the planet formation phase, had higher metallicity, and that during its subsequent evolution, it lost its heavier elements. <C20>For example, during the giant phase, heavy elements could have had been incorporated into dust grains star's atmosphere. <C21>However, given the star's membership to the Helmi stream, which most metal-rich sub-dwarfs known so far have (Fe/H))"-1.5$, we do not expect its initial Fe abundance to exceed this value. (Document End]
then separated from the
[Entity Information]
CO: ["planets", "Solar-system)
C1: ["extra-solar planets", "sun-like stars"]
C2: ["Red Giant Branch", "RGB", "pulsars"]
C6: ["exoplanet search surveys", "MS stars", "metallicity"]
C7: ("planets", "metallicities", "sun's metallicity"]
C8: ["PSR B1620 b", "Jovian planet", "pulsar", "M4"]
C10: ["red HB", "RHB", "Sun"]
C17: ("metallicity", "exoplanet hosting star"]
22

---

[Question]
How does the metallicity of PSR B1620 b, at around 1% of the Sun's, compare to other known exoplanet host stars?
[Step-by-Step Analysis]
Let me analyze this question step by step:
1. Identify key entities and concepts in the question:
"PSR B1620 (specific exoplanet)
"metallicity" (chemical composition measure)
"1% of the Sun's" (metallicity level)
"exoplanet host stars" (comparison group)
2. Find sentences mentioning PSR B1620 b:
C8 mentions "PSR B1620 b" Jovian planet with metallicity
[Fe/H]=-1.2
Need check if 1% matches this description
3. Find sentences about metallicity comparisons:
C6 discusses the connection between metallicity and planet frequency
C7 mentions "10% of the sun's metallicity"
[Fe/H]=-1
C10 mentions "about 1% that of the Sun" for [Fe/H]=-2. 09
star is "far more metal-poor than any previously known exoplanet hosting star"
C17 states
Determine which sentences are needed:
C6: Establishes that metal-rich stars host giant planets
C7: Provides context about very few planets around lox-metallicity
stars
C10: Provides the 1% metallicity reference point
C17: Makes the direct comparison to other exoplanet host stars
[Answer Citation List] Ccite>[6, 17,
Now get ready to handle the following test case.
[Document Start]
<context>
(Document End]
[Entity Information]
<entities>
[Question]
<question>
(Step-by-Step Analysis]
Let me analyze this question step by step:
1. Identify key entities and concepts in the question:
2. Find relevant sentences based on entities and content:
3. Determine relationships and connections:
4.
Select the minimum necessary citations:
[Anser Citation List]
G Dataset example
23

---

Table 7: Example from the HotpotQA dataset
Field
Content
Query
Which American conglomerate company is subsidiary of Trump Entertain- ment Resorts? Icahn Enterprises L.P.
Answer
Target Sentences
It has been a subsidiary of Icahn Enterprises since 2016.
Icahn Enterprises L.P. an American conglomerate company headquartered at the General Motors Building in New York City, New York.
Evidence Index
[7. 8]
Dataset
HotpotQA
Language
English
Index II)
5a075tb55322977642255.bb
Context (Partial)
<CO> TVS Group an Indian diversified industrial conglomerate with its principal headquarters located in Madurai and presence across the Globe. <CI> Almost all holdings of the group are private. <C2> The largest and most visible subsidiary is TVS Motor Company, the third-largest two-wheeler man- ufacturers in India. <C3> TVS Group, with group revenue of more than US$6 billion, is an automotive conglomerate company, specialized in manufactur- ing of two- wheeeeeeler, three- wheeler, auto-electricals components, high tensile fasteners, die casting products, dealership business, brakes, wheels, tyres, axles, seating systems, fuel injection components, electronic and electrical components and many more. <C4> Trump Entertainment Resorts, Inc. gaming and hospitality company that owned and operated the now shuttered Trump Taj Mahal hotel and casino, well as the now shuttered Trump Plaza Hotel and Casino and the Trump Marina located in Atlantic City, New Jersey, United States. <C5> Formerly known Trump Hotels & Casino Resorts, it was founded in 1995 by Donald Trump, now 45th President of the United States, who has not had any formal role in the company since 2011, 1, if not earlier. <C6> The company filed for bankruptcy in 2004, 2009 and 2014. <C7> has been subsidiary of Icahn Enterprises since 2016. <C8> Icahn Enterprises L.P. is an American conglomerate company headquartered at the General Motors Building in New York City, New York. <C9> The company has investments in various industries including auto parts, energy, metals, rail cars, casinos, food packaging, real estate and home fashion. <C10> The company is currently controlled by investor Carl Icahn.
Table 8: Example from the SQuAD dataset
Field
Content
Query
Where is there a growing presence of Theravada? the west
Answer
Target Sentences
[TThervvada primarily practiced today in Sri Lanka, Burma, Laos, Thai- land, Cambodia well as small portions of China, Vietnam, Malaysia and Bangladesh. It has a growing presence in the west.]
Evidence Index
Dataset
SQuAD
Language
English
Index ID
16d1911775 40000
Continued next page
24

---

Table 8 continued from previous page
Field
Content
Context (Partial)
<CO> Kongo-Brazzaville has had a multi- political system since the early 1990s, although the system is heavily dominated by President Denis Sassou Nguesso; he has lacked serious competition in the presidential elections held under his rule. Sassou Nguesso is backed by his own Congolese Labour Party (French: Parti Congolais du Travail) well
range of smaller parties. <CI> Theraväda is primarily practiced today in Sri Lanka, Burma, Laos, Thailand, Cambodia well small portions of China, Vietnam, Malaysia and Bangladesh. It has a growing presence in the west. <C2> Beyoncé announced hiatus from her music career January 2010, heeding her mother's s advice, "to live life, to be inspired by things again". During the break she and her father parted ways as business partners. Beyoncé's musical break lasted nine months and saw her visit multiple European cities, the Great Wall of China, the Egyptian pyramids, Australia, English music festivals and various museums and ballet performances. <C3> The SI unit of illuminance and luminous emittance, being the luminous power per area, is measured in Lux. It is used in photometry as measure of the intensity, as perceived by the human eye, of light that hits or passes through a surface. It is analogous to the radiometric unit watts per square metre, but with the power at each wavelength weighted according to the luminosity function, a standardized model of human visual brightness perception. In English, "lux" is used in both singular and plural. <C4> Chopin's life and his relations with George Sand have been fictionalized in numerous films. The 1945 biographical film Song to Remember earned Cornel Wilde an Academy Award nomination as Best Actor for his portrayal of the composer. Other film treatments have included: La valse de I' l'adicu (France, 1928) by Henry Roussel, with Pierre Blanchar Chopin; Impromptu (1991), starring Hugh Grant as Chopin; La note bleue (1991); ); and Chopin: Desire for Love (2002). <C5> West has additionally appeared and participated in many fundraisers, benefit concerts, and has done community work for Hurricane Katrina relief, the Kanye West Foundation, the Millions More Movement, 100 Black Men of America, a Live Earth concert benefit, World Water Day rally and march, Nike runs, and a MTV special helping young Iraq War veterans who struggle through debt and PTSD a second chance after returning home.
Table 9: Example from the TriviaQA dataset
Field
Content
Query
Who, c1819, wrote the poem Ode to the West Wind?
Answer
Percy Bysshe Shelley
Target Sentences
[1. Ode to the West Wind is an ode, written by Percy Bysshe Shelley in 1819 near Florence, Italy.]
Evidence Index
Dataset
TriviaQA English
Language
Index ID
[qg22256-Ode_too the _est_Wind.tx
Continued on next page
25

---

Table 10: Prompt Template (Causal Intervention via Zeroing Attention Heads)
Template
## Document: [Decument_Content) Content)
You are helpful QA assistant. Please read the document above and answer the following question. Please reason step by step, and put your final answer within \boxed().
## Question:
## Options: A: (A_Content), B: (B_Content), C: (C_Content), D: (D_Centent)
Table 9 continued from previous page
Field
Content
Context (Partial)
<CO> Ode to the West Wind is an ode, written by Percy Bysshe Shelley in 1819 near Florence, Italy. It was published in 1820 in London part of the collection Prometheus Unbound. The poem expresses hope that its words will inspire change and revolution. <C1> The Wang River is river in northern Thailand. It flows from the Phi Pan Nam Range in Chiang Rai Province to the Ping River in Tak Province. <C2> An empire is an aggregate nations ruled by an emperor or sovereign. It can be territorial or maritime, often associated with imperialism or colonialism. <C3> "The rum ration (also called tot) was daily amount of rum given to sailors on Royal Navy ships. It was abolished in 1970 after concerns that regular intakes of alcohol would lead unsteady hands when working machinery. Tradition: The rum ration, or "tot", consisted of rum at 95.5 proof (54.6% ABV), given out to every sailor at midday. Senior Ratings (Petty Officers and above) received their rum neat, whilst for Junior Ratings was diluted with two parts of water to make grog. The rum ration was served from one particular barrel, also known as the "Rum Tub" which was ornately decorated and was made of oak, reinforced with brass bands, with brass letters saying, "The Queen, God Bless Her". Not all sailors necessarily drew their rum - each had the option to marked in the ship's books as "G" (for Grog) or "T" (for Temperate) if there were members of the Temperance Movement. Sailors who opted to be "T" were given three pence (3d) a day instead of the rum ration, although very few sailors took this option. Instead they gave away their ration in exchange for favours. The time when the rum ration was distributed was called "Up Spirits",, which was between 11 am and 12 noon. common cry from the sailors was "Stand fast the Holy Ghost"". Each mess had "Rum Bosun" who would collect the rum from the officer responsible for measuring the right number of tots for each mess. The officers did not get rum ration. Tot glasses were kept separate from any other glasses. They were washed on the outside, but never inside, in the belief that residue of past tots would stick to the side of the glass and make the even stronger. Sailors under 20 were not permitted a rum ration and were marked on the ship's books as "UA" (Under Age). History: The rum ration was originally beer with daily ration of one gallon. This official allowance continued till after the Napoleonic Wars. When beer was not available, as would often spoil easily, it could be substituted by pint of wine or half a pint of spirits depending on what was locally available. In later years, the political influence of the West Indian planters led to rum being given the preference over arrack and other spirits. The half pint of spirits was originally issued neat; it is said that sailors would "prove" its strength by checking that gunpowder doused with rum would still burn (thus verifying that rum was at least 57% ABV).
26

---

Table 11: Prompt Template (Attention Redistribution via Prompt Manipulation)
Template
Context:
(Context)
Question: {Question_ Content)
Choices:
A.
B.
C. ...
You don't need to analyze, just output your choice.
H Supplementary Experiments for Competing Hypotheses
We employ three methods to enhance the model's attention to specific information within the prompt. and observe how the model's responses change accordingly. To enhance the model's attention to counterfactual information, we adopt the following three prompt construction strategies:
Duplicate the counterfactual sentences and append them the original context to reinforce their salience through repetition (see Table T2).
Duplicate and append:
Insert prompt randomly: Extract all counterfactual sentences, concatenate them, prepend an emphatic instruction, and insert the resulting string random position in the original context (see Table 13).
Tag and append: Extract and concatenate counterfactual sentences, enclose them with <must pay attention to> tags for emphasis, and append this segment to the original context (see Table I4).
Table 12: Duplicate and append method
Component
Content
Description
Duplicate the counterfactual sentences and append them to the end of the original context reinforce their salience through repetition.
Example
The Øresund Region a transnational metropolitan area in northern Europe, centered around the Øresund strait and the cities of Copenhagen, Denmark, and Malmö, Sweden. The region is connected by the Øresund Bridge, which runs nearly 8 km from the Swedish coast the artificial island Peberholm in the middle of the strait. [...]
Context:
The bridge connecting Copenhagen and Malmö is 12 km long.
[... Original context continues with additional infrastructure details and historical facts ...]
The bridge connecting Copenhagen and Malmö km long.
Table 13: Insert prompt randomly method
Component
Content
Description
Extract all counterfactual sentences, concatenate them, prepend an emphatic instruction, and randomly insert the block into the context.
Continued on next page
27

---

Table 13 - continued from previous page
Component
Content
Example
The Øresund Region is transnational metropolitan area in northern Europe, centered around the Øresund strait and the cities of Copenhagen, Denmark, and Malmö, Sweden. The region is connected by the Øresund Bridge, which runs nearly 8 km from the Swedish coast to the artificial island Peberholm in the middle of the strait. [...]
Context:
The following information is important and should be taken into account. The bridge connecting Copenhagen and Malmö is 12 km long.
Note:
[... Original context continues with additional infrastructure details and histor- ical facts ...]
Table 14: Tag and append method
Component
Content
Description
Extract and concatenate counterfactual sentences, wrap them with <must pay attention to> tags, and append this segment to the context.
Example
The Øresund Region is a transnational metropolitan area in northern Europe, centered around the Øresund strait and the cities of Copenhagen, Denmark, and Malmö, Sweden. The region is connected by the Øresund Bridge, which runs nearly 8 km from the Swedish coast the artificial island Peberholm in the middle of the strait. [...]
Context:
<must pay attention to> The bridge connecting Copenhagen and Malmö 12 km long. </must pay attention to>
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's contributions, including the design of adaptive probes, empirical validation of their effectiveness, and analysis of mechanisms that resolve conflicts between internal and external knowledge. These claims are consistently supported by both theoretical discussions and experimental results throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our approach in the Appendix 6 Including only decoder-only models and synthetic counterfactual data limits the scope of our study.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
29

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Our puper does not include formal theoretical results or proofs. Instead, we focus on empirical hypotheses regarding knowledge conflict resolution, which are validated through carefully designed ablation studies and experiments (see Section
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: We provide detailed descriptions of our method in the main paper(see in Section 3.. And data sources, preprocessing steps, and our experimental setup can be seen in Section 4 Additionally, all prompt constructions and relevant implementation details are included in the appendix to support reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
30

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are eager to release the code and data publicly; however, due to ongoing company approval processes, we are currently unable to share them immediately. We plan to make them available as soon these procedures are completed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
public//uiide//Codeuuuiit wow.sooobbsssioooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips https///ccpplliiiggudeee////oooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We will include detailed training and testing settings in the Appendix D to ensure full transparency and reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
31

---

Justification: Training the probe requires substantial computational resources, and our cost budget does not support running the training process multiple times. However, we conducted tests on multiple models as well as zero-shot evaluations across different datasets. The consistent improvements observed provide evidence of the probe's 's effectiveness and accuracy.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8, Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: have carefully reviewed the NeurlPS Code Ethics and confirm that our research fully complies with its principles. Our work does not involve human subjects, sensi- tive data, or applications with potential for harm, and we have ensured faimess, transparency, and reproducibility throughout.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
32

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses on understanding internal mechanisms of models and has not been applied in real-world production or societal contexts. Therefore, we believe it does not have immediate broader socictal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairesss considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work presented in this paper does not involve data or models that pose significant risks of misuse, and therefore no specific safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
33

---

Answer: [Yes]
Justification: We have carefully followed the license terms and conditions associated with all third- parrd assets used in this work, including datasets and code. All original sources, creators, and license information are properly cited in the paper. When applicable, we have also included version numbers and URLs for clarity and reproducibility. Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, Saperswithcode com/coo//oaoooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing experiments or research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
34

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
This paper does not involve crowdsourcing experiments or research with human subjects, and therefore no IRB approval or equivalent review was required.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [Yes]
Justification: We use large language models to generate training data. Detailed descriptions and usage are provided in Section 3.3. All prompts can be found in the appendix.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettps://neur/e occConfferences/22225 /LLM) for what should or should be described.
35


Reviews:
[{"review_id": "q7dUhZJrQZ", "reviewer": "Reviewer_jqeN", "summary": "This paper presents an entity-aware probing framework for investigating how large language models (LLMs) reconcile conflicting parametric and contextual knowledge, particularly in retrieval-augmented generation (RAG) settings. The authors design a lightweight probing mechanism using LoRA to trace entity flow within transformer layers and evaluate its effectiveness in both attribution and knowledge conflict scenarios. They find that attention and MLP layers process parametric and contextual knowledge via distinct mechanisms, and that aligned multi-source knowledge accumulates in a superpositional manner.", "questions": "1. **What is the conceptual connection between the relevance discrimination task and knowledge reconciliation?** Relevance seems like a surface property, whereas reconciliation implies *resolution* of conflicting information. Why is the former a good proxy for the latter?\n\n2. **How does parametric knowledge contribute to relevance judgments?** Given that all relevance predictions seem to depend on context (via `D`), is parametric knowledge playing any functional role in this task?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strength\n\n**Timely and Relevant Topic:** The work addresses an important and relatively underexplored problem—how LLMs reconcile conflicting information from parameters vs. retrieval context.\n\n**Broad and Empirical Analysis:** The paper explores multiple probing dimensions (attention head, MLP, residual stream), across multiple model types and benchmarks, contributing a wealth of experimental insights.\n\n---\n\n### Weakness\n\n**Too Narrow Focus on Relevance Discrimination**: A large portion of the paper is dedicated to building and evaluating the entity relevance probe. However, relevance discrimination alone does not explain *how* reconciliation happens. There is a disconnect between the claim of studying reconciliation mechanisms and the design of experiments, which focus more on *detection* than reconciliation.\n\n**Lack of Grounded Real-World Applications**: The knowledge conflict scenarios are synthetic and overly simplified (e.g., single-entity substitutions). There’s little discussion of how this mechanism translates to more complex real-world retrieval settings with temporal or causal dependencies.\n\n**Weak Link Between Relevance and Parametric Knowledge**: It remains unclear how relevance probing reflects the role of **parametric knowledge**. The entity relevance task seems entirely context-driven (i.e., entities in `D` and their alignment with `q`). The paper does not clarify how parametric knowledge contributes to relevance judgments or whether it is required at all.", "was_revised": true, "final_justification": "The authors provide new results to validate the generality of their conclusions", "rebuttal": "**Dear Reviewer jqeN**,\n\nWe greatly appreciate your thoughtful feedback on our work! We are encouraged by your recognition of `the importance and value of our research topic` as well as `the wealth of experimental insights`. We address your questions and concerns as follows:\n\n> **Q1.** What is the conceptual connection between the relevance discrimination task and knowledge reconciliation? Relevance seems like a surface property, whereas reconciliation implies resolution of conflicting information. Why is the former a good proxy for the latter?\n\n**The dynamic changes in entity relevance (i.e., information intensity) across layers directly reflect the model's mechanism for reconciling conflicting information, rather than being a superficial property unrelated to the reconciliation process.** We illustrate how entity relevance can be used to analyze the model's conflict reconciliation mechanism through a demonstrative example.\n\nConsider the query $q$ = \"*Who is the current U.S. President?*\" Suppose the model's parametric knowledge is $\\mathcal{E}_p$ = \"Biden\" while the external document $D$ contains contextual knowledge $\\mathcal{E}_c$ = \"Trump\". The model now faces a decision: which of these conflicting pieces of knowledge, $\\mathcal{E}_p$ or $\\mathcal{E}_c$, should be used to answer query $q$?\n\n- If only one type of knowledge can be detected in the attention layer's output, i.e., Attention(\"Biden\", \"Trump\") → \"Trump\", this indicates that the attention layer reads prompt=$\\mathcal{D}+q$ but only routes one type of knowledge for answer generation. In this case, conflict reconciliation occurs at the attention layer.\n- If both types of knowledge are routed by the attention layer, but the information intensity of one type rapidly decays in the residual stream at the position where the answer token is generated, i.e., (\"Biden\", \"Trump\") → MLP$_i$ → (\"Trump\"), this suggests that the MLP layer plays the role of conflict reconciler in the residual stream.\n\n(The two hypothetical scenarios above are for illustrative purposes only; the actual conflict reconciliation mechanism is more complex than these two cases. See Section 5.4 of the paper for further discussion.)\n\n> **W1**. Too Narrow Focus on Relevance Discrimination: A large portion of the paper is dedicated to building and evaluating the entity relevance probe. However, relevance discrimination alone does not explain how reconciliation happens. There is a disconnect between the claim of studying reconciliation mechanisms and the design of experiments, which focus more on detection than reconciliation.\n\nIn our response to Q1, we clarify how relevance metrics explain reconciliation mechanisms, which we believe addresses your concerns regarding Weakness 1. The experimental design of this paper consistently centers on studying reconciliation mechanisms and provides valuable insights into these mechanisms. This has been acknowledged by Reviewer 6M6N (Strengths 3, 4, 5) and Reviewer oNGe (Strengths 1, 2, 3). Furthermore, probe construction is a necessary means to measure entity relevance (i.e., the information strength of knowledge encoded in model representations), and the probe constructed in this paper differs significantly from classical probes in technical aspects. Therefore, we believe it is necessary to describe the construction method and evaluate its validity. This point has also been recognized by Reviewer KnAk (Strength 3).\n\n>**W2**. Lack of Grounded Real-World Applications: The knowledge conflict scenarios are synthetic and overly simplified (e.g., single-entity substitutions). There’s little discussion of how this mechanism translates to more complex real-world retrieval settings with temporal or causal dependencies.\n\nThank you for your valuable feedback. We completely agree that the applicability of research work to real-world tasks is an important evaluation criterion. Regarding the concerns you raised, we would like to further clarify the real-world foundations and design considerations of our work:\n\n**(1) LLM-based Knowledge Conflict Generation vs. Simple String Replacement**:\n\nOur knowledge conflict scenario generation is based on LLM chain-of-thought (CoT) reasoning, which is the widely adopted standard method for constructing knowledge conflict scenarios in current research [1,2,3]. This approach is fundamentally different from the simple rule-based string replacement methods used several years ago [4,5,6,7]. When LLMs construct knowledge conflicts, they consider semantic coherence between contexts, different expressions of the same entity within contexts, and logical associations between multi-hop evidence—capabilities that cannot be achieved through simple string replacement.\n\n**(2) Diverse Conflict Types Beyond Simple Entity Substitution**:\n\nOur data construction encompasses not only simple entity replacement conflicts but also complex scenarios including temporal and causal conflicts. Specifically, we construct knowledge conflict scenarios based on HotpotQA, which inherently possesses multi-hop reasoning characteristics. Modifications to entities in the reasoning chain or their related attributes directly impact reasoning outcomes. Here is an example of temporal conflict:\n\nQuery: *Which American rock band was formed first, Soil or Drowning Pool?*\n\nEvidence A: *Soil, often typeset as SOiL, is an American rock band that was formed in Chicago, Illinois in 1997.*\n\nEvidence B: *Drowning Pool is an American rock band formed in Dallas, Texas in 1996.*\n\nAnswer: *Drowning Pool.*\n\nAfter analyzing the question and related evidence, the LLM modifies the formation times of both bands as follows:\n\nRevised Evidence A: *Soil, often typeset as SOiL, is an American rock band that was formed in Chicago, Illinois in 1995.*\n\nRevised Evidence B: *Drowning Pool is an American rock band formed in Dallas, Texas in 1998.*\n\nSince the temporal sequence has been reversed, this affects the answer judgment:\n\nRevised Answer: *Soil*\n\n**(3) Trade-off Between Synthetic and Real Data**:\n\nRelying entirely on collecting real-world conflict samples would be prohibitively expensive. Although synthetic data differs from real data, this represents a feasible and effective solution for batch construction of knowledge conflicts at present.\n\n\n[1] Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts.\n\n[2] CONFLICTBANK: A Benchmark for Evaluating Knowledge Conflicts in Large Language Models.\n\n[3] Intuitive or Dependent? Investigating LLMs' Behavior Style to Conflicting Prompts.\n\n[4] Context matters: A pragmatic study of PLMs' negation understanding.\n\n[5] BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief.\n\n[6] Entity-based knowledge conflicts in question answering.\n\n[7] Context-faithful prompting for large language models.\n\n\n>**Q2&W3**. How does parametric knowledge contribute to relevance judgments? Given that all relevance predictions seem to depend on context (via D), is parametric knowledge playing any functional role in this task?\n\nSince Q2 is a derivative question of W3, we address both concerns together in this response.\n\nThe comparison between Figure 4(c) and Figure 5(c) in our paper directly demonstrates the impact of parametric knowledge on relevance judgment. In Figure 4(c), the lines ($\\mathcal{D}_{cf}\\rightarrow\\mathcal{E}\\_f$) and ($\\mathcal{D}\\_{cf}\\rightarrow\\mathcal{E}\\_{cf}$) represent the probing of factual knowledge $\\mathcal{E}\\_{f}$ and counterfactual knowledge $\\mathcal{E}\\_{cf}$ from the counterfactual context $\\mathcal{D}\\_{cf}$, respectively. Although factual knowledge $\\mathcal{E}\\_{f}$ is never mentioned in the counterfactual context $\\mathcal{D}\\_{cf}$, we can still observe rapid growth in factual knowledge information intensity in the residual stream near the output layers. The only possible source of this information intensity is the injection of parametric knowledge from the model.\n\nFrom the perspective of information intensity magnitude, the factual knowledge in Figure 4(c) is weaker than the counterfactual knowledge. This aligns with the LLM's actual response behavior, as shown in Table 1, where 54.6% of model responses follow counterfactual knowledge when the prompt contains only the counterfactual document $\\mathcal{D}\\_{cf}$. However, when both counterfactual document $\\mathcal{D}\\_{cf}$ and factual document $\\mathcal{D}\\_{f}$ are provided in the prompt, the situation reverses. As shown in Figure 5(c), the $\\mathcal{E}\\_{f}$ from parametric knowledge injection and the contextual knowledge $\\mathcal{E}\\_{f}$ described in the factual document form a superposition, with information intensity directly exceeding that of $\\mathcal{E}\\_{cf}$. This also corresponds to the model's actual behavior in Table 1, where 56.5% of model responses follow factual knowledge, far exceeding the 27.7% that follow counterfactual knowledge. Without the information intensity from parametric knowledge injection, the difference between the two would not be so significant.\n\n\n**Table 1: Model response behavior statistics under different document input settings. Data from experiments in Appendix F of the paper.**\n| Context Type | Model Response as Factual Knowledge | Model Response as Counterfactual Knowledge | Other |\n|-|-|-|-|\n| Only counterfactual document $\\mathcal{D}\\_{cf}$ | 179 (22.7%) | 430 (54.6%) | 179 (22.7%) |\n| Both counterfactual document $\\mathcal{D}\\_{cf}$ and factual document $\\mathcal{D}\\_{f}$ provided | 445 (56.5%) | 218 (27.7%) | 125 (15.9%) |\n\nIn fact, the training data for the probes also includes predicting the relevance of factual entities based on counterfactual documents—entities that are not mentioned in the documents. Therefore, the training of relevance prediction does not rely entirely on the alignment between entities in $D$ and the query $q$."}, {"review_id": "7pPfNNBCNT", "reviewer": "Reviewer_KnAk", "summary": "In RAG,  LLMs are provided with some information in their input context. This information may conflict with the LLM’s internal knowledge. This paper designs a probing method to understand how the contextual and parametric knowledge propagates through the layers and how the conflicts are resolved. What is the role of attention and  MLP layers? The probe consists of a fine-tuned classification head to identify if an entity is relevant to answer the given question. Input to the classifier is the attention, the MLP, or the entire layer residual embedding. If the entity is classified as relevant, then the corresponding source from which the input is taken (attention head, MLP, or the entire layer) is considered relevant for the entity. With counterfactual information in the context, they run the probe for factual and counterfactual entities that are assumed to represent the parametric and contextual knowledge, respectively. The sources (attention, MLP or entire layer) that identify factual entities as relevant are considered important for parametric knowledge, and sources that identify counter-factual entities as relevant are considered important for contextual knowledge.", "questions": "1. “Here, contextual knowledge types (factual vs. counterfactual) compete for attention within contextual heads:” Line 265: Once you identify the contextual and memory heads in (i) from fig 3(a) and 3(b), do they remain fixed for (iii)?\nFor section 5.2 (iii), only the top 10 contextual and memory heads are important. Maybe a better representation would be a bar chart with 10 bars for $E_f$ and 10 bars for $ E_{cf}$; X axis representing top-10 contextual head ids (in descending order); Y axis representing information intensity. That will give a direct comparison between the contextual information intensity gap between $E_f$ and $E_{cf}$. You can repeat the same for memory heads.\n\n2. “However, the presence of factual knowledge in the context has minimal impact on memory heads” line 267: In the case of $E_f$, memory heads and contextual heads are corroborating with each other. Is there any analysis that demonstrates this?\n\n3. “the average information intensity of top-10 memory heads slightly increases” line 269: Are you identifying top-10 memory heads again, or are they the same as identified in section 5.2 (i). \n\n4. The range of prauc scores in Fig. 4a and 5a is not the same. Does that impact your analysis?\n\n5. In Fig. 4a and 5a, how are you aggregating the information intensity across heads?\n\n6. “Contextual knowledge (E_cf ), in contrast, is immediately detectable in early attention updates,” p. 8, line 284:  Isn’t the range on the y-axis for Fig. 4(a) too low to make this comment?\n\n7. Line 303 to 305:  Could you please elaborate on this?\n\n8. “Asymmetric propagation mechanisms reveal contextual knowledge emerges abruptly via attention while parametric knowledge accumulates gradually through MLP layers;” Line 338: Which observation supports the abrupt emergence of contextual knowledge via attention? Gradual accumulation via MLP layers seems to be common to both parametric and contextual knowledge.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1.    The paper aims to understand a pertinent issue that often arises when LLMs are used with dynamic external information. It is important to understand how and when the LLMs rely on contextual information and when they fall back on their parametric knowledge. It is also important from the perspective of designing guardrails for LLMs, as the input context may prompt the model not to follow ethics or may contain profanity, and instructions to override LLM’s internal alignment. Understanding how contextual and parametric knowledge reconcile can help us design better guardrails as well.\n\n2.   The design of counterfactual information is neat and clever. Creation of fictional entities ensures that the contextual information is not in the parametric knowledge. \n\n3.   Design of an experiment to validate the probe using the attribution task is also helpful. \n\n\n**Weaknesses**\n\nThe most crucial part of an analysis paper is the experimental design, and I think there are some issues with it. \n\n\n\n**Probe design:**\n\n1.  During the training of the probe, you are not using the counterfactual data. And you assume that the factual entities represent the parametric knowledge (line 219-220).  So are they representing the parametric knowledge or the contextual knowledge? \n\n2. When training the classification head, do you compute loss using the MLP residuals from all the layers and attention residuals from all the heads of all the layers? If not, then can you rely on its output when it is used to quantify information intensity in the intermediate residual streams? \n\n3. In section 4, eqn 4, which residual stream are you using to compute $P_\\theta$?\n\n4. You are training a LoRA for the embedding layer. Why not simply learn embeddings of the two new special tokens? Why impact all other embeddings via LoRA?\n\n5. In Probing-based attribution (section 4), you need to first identify the entities in the sentence.  I.e, you have additional information about the entities in the document. If those are extracted using a stronger model like GPT-4o, then that gives probing an unfair advantage.\nMaybe for a fair comparison, you could provide the extracted entities as part of the input to the baseline prompting method, and also design a prompt with CoT style one-shot demonstration (the way you do in Table 2 for data generation).\n\n**Using the probe for understanding parametric and contextual knowledge (section 5)** \n\n6. The probe is only identifying if an input entity is relevant to answer the given question or not. I am not sure if its usage to identify processing units (attention heads and MLP layers) for counterfactual (contextual) or parametric knowledge is reasonable or not.\n\n7.  Line 219: “we assume its factual descriptions align with LLMs’ parametric knowledge.”: It is a very crucial assumption, and the entire analysis rests on it. Unfortunately, it may not always be true. If it were always true, then LLMs would get close to 100% performance on HotpotQA even when prompted without any documents, as the required knowledge is in their parameters.\nFor this analysis, maybe you should select those questions where the LLM gives the correct answer out of the box without providing any external document in the context. This will ensure that documents are truly aligned with LLM's internal parametric knowledge.\n\n\n**Typos:**\n\n1. “Our analysis reveals three key insights: Our analysis reveals three key insights:” p. 2\n\n2. “individual attention head outputs $∆h_{l}^{attn}(t)$.” p. 3, line 97 - individual attention head output should have $(k)$ as superscript.\n\n3. “Probing-based attribution: We addresses the..” p. 5, line 186\n\n4. “we probing all mentioned entities” p. 5, line 187. \n\n5. “$E − (E_f ∪ E_f )$” : p. 6, line 230: it should be $E_{f} ∪  E_{cf}$\n\n6. “However, how models handle such conflicts remains an area for further research. research.\"” : p. 9, line 325: there is a closing quote without any opening quote. \n\n7. “The outputs of N attention heads are summed to form the attention sublayer output ...” p. 3, line 91:  $i$  should go from 1 to N and not $k$ to N. The superscript on RHS should be $(i)$ and not $(k)$.\n\n8. “We follow the approach of Ye et al. [41] to resolve the limitations of traditional probing outlined in Section 2.2, adapting the probe fθ for entity relevance probing” p. 4, line 128 to 130 -  Looks like an incorrect citation-- did you mean to cite Physics of Language Models (PoLM) -3.1? You have cited PoLM-2.1.", "was_revised": true, "final_justification": "My concerns were with the experimental setup, and they have been addressed by the authors during the rebuttal. \nHence, I am raising my score to accept.", "rebuttal": "**Dear Reviewer KnAK**,\n\nThank you very much for taking the time to provide such valuable and detailed feedback on our manuscript. We are pleased that you found merit in `the important value of our research on conflict reconciliation mechanisms`, `our neat and clever counterfactual data synthesis`, and `helpful probing validation experiments`. We address your questions and concerns as follows:\n\n> **W1**. Questions regarding the probe training data and the relationship between entities and their corresponding knowledge types.\n\n**The probe training does utilize counterfactual data**. We generate the probe's training data based on the prompt template provided in Table 2 of the paper. The prompt includes both \"Analysis\" and \"Counterfactual Data Synthesis\" components. The latter generates counterfactual versions of original documents and identifies corresponding counterfactual entities. For the three training data types in the probe, **entities map to their knowledge types as follows**:\n\n**Table 1: Probe Training Dataset Categories**\n| Document Type | Positive Entities | Knowledge Type Represented |\n|-|-|-|\n| Factual document $\\mathcal{D}_f$ | Factual entities $\\mathcal{E}_f$ | Superposition of parametric and contextual knowledge |\n| Counterfactual document $\\mathcal{D}_{cf}$ | Factual entities $\\mathcal{E}_f$ | Parametric knowledge |\n| Counterfactual document $\\mathcal{D}_{cf}$ | Counterfactual entities $\\mathcal{E}_{cf}$ | Contextual knowledge |\n\nWe commit to improving the clarity of section3.3 to avoid misunderstandings.\n\n\n> **W2&W6**. Concerns about the probe training inputs and whether probes can measure the information strength in residual streams.\n\nProbes are trained on output layer representations, then used to measure information strength in residual streams. This approach is known as ``early exit`` [1-3], which has proven to be effective even without special training processes [4], as the residual connections [5] in transformer layers make the hidden representations gradually evolve without abrupt changes.\n\n[1] Fast inference via early exiting from deep neural networks.\n\n[2] Depth-adaptive transformer.\n\n[3] Confident adaptive language modeling.\n\n[4] Bert's output layer recognizes all hidden layers? some intriguing phenomena and a simple way to boost bert.\n\n[5] Deep residual learning for image recognition.\n\n\n>**W3**. In section 4, eqn 4, which residual stream are you using to compute ?\n\nWe compute using the [END] token's output layer representation.\n\n>**W4**. Why use LoRA to implement probes?\n\nAs described in line 128 of the paper, we fully follow the approach in [1] for probing entity information, including introducing two new special tokens and a LoRA update. According to the description in Section 4.1 of [1], the \"small\" LoRA update is designed to help LLMs adapt to changes in input format. We hypothesize that completely freezing LLM parameters and relying solely on two new token embeddings would be insufficient to achieve this adaptation.\n\n[1] Physics of language models: Part 2.1, grade school math and the hidden reasoning process.\n\n>**W5**. Fairness concerns in attribution experiments.\n\nFollowing the your suggestion, we enhanced the baseline prompting methods with the following conditions to ensure fair comparison:\n- **+entity**: Providing GPT-4o extracted entity information as input.\n- **+CoT**: Incorporating carefully designed CoT-style one-shot demonstrations.\n\n**Table 2: Attribution Results with Fair Comparison Conditions vs. Probing Methods**\n| Model | Average |\n|-|-|\n| Qwen2.5-7B+entity+CoT | 75.7 |\n| Qwen2.5-7B-Probing | 73.0 |\n| Qwen3-8B+entity+CoT | 81.6 |\n| Qwen3-8B-Probing | 70.8 |\n| LLaMA3.1-8B+entity+CoT | 63.1 |\n| LLaMA3.1-8B-Probing | 68.4 |\n\n**Highlight our motivation**: It is important to clarify that our probing method is not intended to compete with prompting methods in terms of performance, but rather to provide a complementary analytical tool for understanding the internal attribution mechanisms of language models.\n\n**Validation of Intrinsic Model Capabilities**: The results demonstrate that under sufficient conditions (+entity+CoT), instruction-following models can indeed \"articulate\" accurate attributions. This confirms that models intrinsically possess the ability to perceive entity relevance; they simply require appropriate methods to elicit this latent knowledge.\n\n\n>**W7**. Concerns regarding alignment between LLM parametric knowledge and HotpotQA contextual knowledge, and suggestions for testing on filtered subsets.\n\nFollowing the reviewer's suggestion, we filtered the evaluation data to include only questions that LLMs answer correctly without any document prompts. Our main findings remain consistent on these subsets, including distinct routing pathways, within-type competition, and superpositional reconciliation. The primary difference is that factual entities show higher average information intensity compared to the full dataset. Due to rebuttal length constraints, we will include the complete results in the revised paper.\n\n\n>**Q1&Q3**. Do the context heads and memory heads identified in Figure 3(a) and (b) remain consistent in part (iii) of section 5.2? A bar chart would be a better presentation format.\n\nThe context heads and memory heads identified in part (i) from Figure 3(a) and (b) remain consistent in part (iii). There is only one minor exception: the 11th context head in layer 14 falls out of the top 10 in terms of information intensity in subplot (d) \"$\\mathcal{D}_{cf}\\cup \\mathcal{D}_f \\rightarrow \\mathcal{E}_f$\". Due to rebuttal policy constraints, we cannot provide graphical results, so we are using tabular data as a substitute. The context heads and memory heads are ranked according to their information intensity from high to low as shown in Figure 3(a) and (b), respectively. We will supplement the results with bar charts in the revised version of the paper.\n\n**Table 3: Distribution and Information Intensity Values of Context Heads**\n|Layer ID|Head ID|Fig. 3(a) $\\mathcal{D}_{cf} \\rightarrow \\mathcal{E}\\_{cf} $|Fig. 3(c) $\\mathcal{D}_{cf}\\cup \\mathcal{D}_f \\rightarrow \\mathcal{E}\\_{cf}$|Fig. 3(d) $\\mathcal{D}_{cf}\\cup \\mathcal{D}_f\\rightarrow\\mathcal{E}_f$|\n|-|-|-|-|-|\n|16|4|0.119|0.086|0.079|\n|15|7|0.119|0.042|0.061|\n|19|18|0.115|0.039|0.045|\n|12|22|0.092|0.052|0.079|\n|22|4|0.089|0.041|0.037|\n|20|15|0.086|0.048|0.047|\n|14|11|0.069|0.028|0.019|\n|24|28|0.066|0.030|0.040|\n|15|0|0.051|0.030|0.030|\n|20|7|0.048|0.029|0.034|\n\n**Table 4: Distribution and Information Intensity Values of Memory Heads**\n|Layer ID|Head ID|Fig. 3(b) $\\mathcal{D}\\_{cf}\\rightarrow\\mathcal{E}\\_{f}$|Fig. 3(d) $\\mathcal{D}\\_{cf}\\cup \\mathcal{D}\\_f\\rightarrow\\mathcal{E}\\_f$|\n|-|-|-|-|\n|27|14|0.027|0.028|\n|31|26|0.022|0.010|\n|12|22|0.018|0.079|\n|26|23|0.018|0.011|\n|16|16|0.018|0.015|\n|18|9|0.017|0.019|\n|20|23|0.016|0.018|\n|29|25|0.016|0.019|\n|13|25|0.015|0.026|\n|30|23|0.015|0.019|\n\n>**Q2**. Impact of Factual Knowledge in Context on Memory Heads\n\nBased on Table 4's analysis of 10 memory heads, we address this concern systematically. Among these heads, one overlaps with context heads (Layer 12, Head 22, as mentioned in Line 251).\nWhen context shifts from $\\mathcal{D}_{cf}$ to $\\mathcal{D}\\_{cf}\\cup\\mathcal{D}\\_{f}$, factual knowledge exists both as contextual and parametric knowledge. The overlapping head (Layer 12, Head 22) shows a 4x+ increase in information intensity (0.018 → 0.079), as it captures both parametric knowledge and is highly sensitive to newly injected factual knowledge in context.\nIn contrast, the remaining 9 pure memory heads show much smaller magnitude changes with inconsistent directions (some increase, some decrease). Therefore, we conclude that factual knowledge in context has minimal impact on pure memory heads, at least on average.\n\n>**Q4**. The range of prauc scores in Fig. 4a and 5a is not the same.\n\nThe ranges are indeed different, which is an inevitable result of contextual knowledge competition. In Figure 4a, the counterfactual entity monopolizes all attention from the contextual heads. In Figure 5a, factual documents are introduced into the context, where factual entities draw away some of the attention from contextual heads. Therefore, the range of PRAUC scores becomes smaller. This can also be observed in Table 3 from response to Q1&Q3. After introducing $\\mathcal{D}_{f}$, the attention from 10 context heads was diverted to factual entities, reducing the average information strength of counterfactual entities from 0.0854 to 0.0425.\n\n>**Q5** How are you aggregating the information intensity across heads?\n\nMeasure the aggregated information intensity by probing the output representation of the attention layer (described in line 92 of the paper).\n\n> **Q6&Q8.**: Concerns regarding the formation mechanisms and information intensity of contextual vs. parametric knowledge\n\nIn Fig. 4(a), the information intensity of contextual knowledge in early attention updates is already 4-6 times higher than parametric knowledge, representing a substantial gap. This indicates that contextual knowledge emerges in early attention routing, while the complete semantics of parametric knowledge do not yet appear. We hypothesize that parametric knowledge formation requires layer-by-layer abstraction of knowledge fragments until complete semantic units are formed near the output layers.\n\nIt's worth noting that residual connections in modern neural networks enable representations to gradually evolve without abrupt changes, making it difficult to observe very large information intensity values.\n\n> **Q7.** Line 303 to 305: Could you please elaborate on this?\n\nThis indicates that attention competition occurs only within the same type of knowledge (refer to the analysis of Q4), rather than across different categories (refer to the analysis of Q2)."}, {"review_id": "vEqBQSitjw", "reviewer": "Reviewer_6M6N", "summary": "This paper provides a rigorous analysis and mechanistic understanding of how LLMs reconcile conflicting parametric knowledge (internalized in the model parameters during pretraining) and contextual knowledge (present as in context tokens) in retrieval-augmented generation/RAG settings.\n\nSpecifically the authors propose a novel entity based probing technique which leverages specially appended tokens and a finetuned LoRA adapter to track entity-level relevance across transformer layers, without fine-tuning the base model.\nUsing this framework they probe the information of a specified entity at 3 computation points within each transformer layer: MLP update, attention update and layer output.\n\nThey find several interesting patterns, consistent across different models, namely that \n1. parametric and contextual knowledge utilize distinct pathways revealing attention head specialization; \n2. contextual knowledge emerges more abruptly whereas parametric knowledge accumulates more gradually through MLP layers; \n3. super-positional reconciliation shows conflicting knowledge persists in residual streams with similar information from multiple sources reinforcing each other over contradicting information e.g. parametric + factual document versus counterfactual document information.", "questions": "Could authors more clearly explain around why the entity triple being appended does not affect the output? For example even with a randomly initialized LLM the mean F1 is around 20 (one would expect zero for complete independence)", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The approach described as far as I know is quite novel.\n2. The information conflict between parametric and contextual knowledge and controlling/understanding it is a topic of high importance for academic and industrial research.\n3. The experimental approach is very rigorous and well through out. \n4. Experiments are robust with comprehensive and well-motivated ablations and conclusions.\n5. Reveals extremely interesting and insightful patterns around where and how parametric and contextual information is aggregated in the LLM and how knowledge conflicts are handled.\n\nWeaknesses:\n1. I am still less convinced on the triple being appended to the model input not affecting the outputs and conclusions at all. The authors attempt to isolate the impact of the probe by comparing performance on a randomly initialized model (where performance collapses) however it seems counterintuitive that this would have no impact and there are not better ways e.g. probing around the token the entity organically appears in the context (at least for the cases where it does)", "was_revised": true, "final_justification": "My original score was already at a 5. I had a few small questions/clarifications/concerns which authors' have addressed, no change to my original rating.", "rebuttal": "**Dear Reviewer 6M6N,**\n\nWe are grateful for your thorough review and positive assessment. Your recognition of `the novelty of our approach`, `the significant value of our research topic`, and `the rigor of our experiment designs` along with `the depth of our conclusions` motivates us and confirms the value of our research contributions. We address your questions and concerns as follows:\n\n> **W1.** I am still less convinced on the triple being appended to the model input not affecting the outputs and conclusions at all. The authors attempt to isolate the impact of the probe by comparing performance on a randomly initialized model (where performance collapses) however it seems counterintuitive that this would have no impact and there are not better ways e.g. probing around the token the entity organically appears in the context (at least for the cases where it does)\n>\n> **Q1.** Could authors more clearly explain around why the entity triple being appended does not affect the output? For example even with a randomly initialized LLM the mean F1 is around 20 (one would expect zero for complete independence)\n\nWe break down W1&Q1 into several sub-questions and address them one by one.\n\n**(1) Why does a randomly initialized LLM achieve an average F1 score around 20? Is this reasonable?**\n\nAn F1 score around 20 represents the result of random guessing. Let us illustrate this with a concrete example: suppose your attribution evaluation dataset consists of samples where each contains a query and 100 candidate sentences, with 10 target sentences containing information relevant to the query. Your task is to identify these 10 target sentences from the 100 candidates.\n\nNow consider a \"poor\" model that completely ignores the semantic content of candidate sentences and considers all of them relevant to the query—essentially predicting all 100 candidates as target sentences. Let's calculate the F1 score in this scenario:\n\nPrecision = $\\frac{\\text{Number of correctly identified targets}}{\\text{Number of predicted targets}} = \\frac{10}{100} = 0.1$\n\nRecall = $\\frac{\\text{Number of correctly identified targets}}{\\text{Total number of actual targets}} = \\frac{10}{10} = 1.0$\n\nF1 = $2 × \\frac{\\text{precision × recall}}{\\text{(precision + recall)}} = 0.182$\n\nClearly, an F1 score of 0.182 does not indicate that this \"poor\" model possesses any attribution capability.\n\n\n**(2) Since probes require specific input formats, are the probing results consistent with the model's actual behavioral patterns?**\n\nThe probe test results demonstrate high correlation with the model's actual output behavior. Additional analysis based on the LLaMA 3.1 model is provided in Appendix F of our paper, with results summarized in Table 1 below. The methods \"Duplicate and append,\" \"Insert prompt randomly,\" and \"Tag and append\" can be viewed as three approaches to enhance the model's attention to counterfactual knowledge in the context (detailed descriptions are available in Appendix F).\n\n**Table 1: Model Response Patterns Under Different Context Conditions**\n| Context Type | Factual Knowledge Response | Counterfactual Knowledge Response | Other |\n|-|-|-|-|\n| $\\mathcal{D}\\_{cf}$ | 22.7% | 54.6% | 22.7% |\n| $\\mathcal{D}\\_{cf}$ (Duplicate and append) | 22.8% | 55.6% | 21.6% |\n| $\\mathcal{D}\\_{cf}$ (Insert prompt randomly) | 22.8% | 59.9% | 17.3% |\n| $\\mathcal{D}\\_{cf}$ (Tag and append) | 23.2% | 51.3% | 25.5% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ | 56.4% | 27.7% | 15.9% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ (Duplicate and append) | 40.2% (↓16.2%) | 45.3% (↑17.6%) | 14.5% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ (Insert prompt randomly) | 51.1% (↓5.3%) | 40.9% (↑13.2%) | 8.0% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ (Tag and append) | 44.9% (↓11.5%) | 41.2% (↑13.6%) | 13.8% |\n\n\nAs shown in Table 1, when the context contains only counterfactual documents, 54.6% of responses follow counterfactual knowledge, significantly higher than responses following factual knowledge (22.7%). This phenomenon aligns with Figure 4(c) in our paper, where the information intensity of counterfactual knowledge in the output layer exceeds that of factual knowledge, and the magnitude of information intensity determines the degree of influence that response knowledge has on the output.\n\nWhen we enhance the model's attention to counterfactual knowledge in the context through three different methods, the model's output does not shift from counterfactual to factual knowledge. This is consistent with our probing conclusion that \"knowledge competition does not occur across categories\"—that is, enhancing attention to contextual knowledge does not affect the routing of memory heads to parametric knowledge.\n\nWhen the input contains both factual and counterfactual documents ($\\mathcal{D}\\_f\\cup \\mathcal{D}\\_{cf}$ setting), compared to the $\\mathcal{D}\\_f$ setting, the model's output clearly begins to favor factual knowledge (56.4% vs. 27.7%). Why does the model favor factual knowledge when both types appear simultaneously in the context? Our probing conclusion of \"superpositional reconciliation\" explains this well. Factual knowledge from the context and factual knowledge from parameters form a superposition, making its information intensity exceed that of counterfactual knowledge sourced only from the context. Since information intensity magnitude directly affects output, the model favors factual knowledge.\n\nSubsequently, when we enhance the model's attention to counterfactual knowledge, the proportion of counterfactual responses increases significantly. The results of this operation under $\\mathcal{D}\\_f\\cup \\mathcal{D}\\_{cf}$ setting are strikingly different from those under $\\mathcal{D}\\_f$ setting, but this is not surprising. Under $\\mathcal{D}\\_f\\cup \\mathcal{D}\\_{cf}$ setting, factual knowledge comes from both model parameters and context. When we enhance the model's attention to counterfactual context, the two compete for the attention of context heads, weakening the context heads' attention to factual knowledge. This aligns with our probing conclusion that \"knowledge competition occurs within categories.\"\n\n**(3) Are there better methods? For example, probing around context tokens where entities naturally occur?**\n\nWe appreciate the reviewer's suggestion to explore more natural probing methods. Indeed, developing better probing techniques to accurately capture models' internal knowledge processing mechanisms remains an important and open research question worthy of continued exploration and improvement. However, the reviewer's suggested approach of \"probing at positions where entities naturally occur in context\" faces a fundamental limitation, which is precisely the core problem our entity-aware probing framework aims to address: it cannot effectively probe parametric knowledge.\n\nFor example, when an external document claims \"Trump is the US President,\" the model's parametric knowledge \"Biden\" simply does not appear in the input context. We would therefore be unable to probe the model's internal processing and propagation of the parametric knowledge \"Biden.\" Our method enables dynamic tracking of any relevant entity (regardless of whether it appears in the context) by explicitly marking target entities, which is a necessary condition for understanding LLMs' knowledge reconciliation mechanisms."}, {"review_id": "n4py6sKVJr", "reviewer": "Reviewer_oNGe", "summary": "This paper introduces a novel entity-aware probing framework to investigate how LLMs reconcile potentially conflicting parametric and contextual knowledge in RAG settings. The authors reinterpret transformer computation through residual streams and design a LoRA-based probing method that can detect ad-hoc entity relevance without altering base model parameters. Through extensive experiments across QA benchmarks and counterfactual setups, the authors reveal that contextual and parametric knowledge are routed by distinct attention heads, exhibit asymmetric propagation dynamics, and interact through a mechanism of superpositional reconciliation.", "questions": "* Have the authors considered how toxic contextual knowledge that might trigger LLMs' safety mechanisms affects internal knowledge reconciliation? Would similar patterns of attention routing still hold?\n* To what extent do the \\[START\\] and \\[END\\] tokens introduced for probing influence the internal computation of LLMs? Specifically, how much attention do LLMs allocate to these OOD tokens?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The proposed Attention Competition Hypothesis and MLP Arbitration Hypothesis provide novel insights for interpreting conflicting knowledge processing. These hypotheses are empirically validated through a carefully designed probing framework that traces entity-specific signals across transformer layers. \n* The distinction between attention head specialization and MLP-based accumulation mechanisms provides a refined understanding of LLM internal processes. The discovery that conflicting knowledge can coexist in residual streams is particularly revealing.\n* The authors conduct extensive and controlled experiments to validate their hypotheses. Notably, they introduce a key ablation baseline to demonstrate that probe predictions depend on the internal knowledge processing of the base LLM, rather than the capacity of the LoRA adapter.\n\nWeaknesses:\n\n* While the entity-aware probing framework avoids modifying base model weights, it introduces LoRA adapters and injects special tokens (\\[START\\], \\[END\\]) into the input. These interventions may perturb the native inference pathways. Consequently, the behavior traced by the probe may deviate subtly from that of the vanilla LLM, raising concerns about the fidelity of the interpretability conclusions.\n* A key claim is that conflicting knowledge persists in residual streams and is superpositionally reconciled rather than being suppressed. However, this assertion is not sufficiently substantiated with targeted experiments. For example, the authors do not investigate whether zeroing out high-information-intensity attention heads would alter the final model prediction.", "was_revised": true, "final_justification": "Since the authors addressed most of our concerns, I decided to raise the rating to 5.", "rebuttal": "**Dear Reviewer oNGe,**\n\nWe sincerely thank you for your constructive and insightful review. Your positive acknowledgment of `our novel insights for interpreting conflicting knowledge processing`, `carefully designed probing framework`, and `refined understanding of LLM internal processes` is particularly encouraging and validates our research approach. We address your questions and concerns as follows:\n\n> **W1.** While the entity-aware probing framework avoids modifying base model weights, it introduces LoRA adapters and injects special tokens ([START], [END]) into the input. These interventions may perturb the native inference pathways. Consequently, the behavior traced by the probe may deviate subtly from that of the vanilla LLM, raising concerns about the fidelity of the interpretability conclusions.\n\nThe probe test results demonstrate high correlation with the model's actual output behavior. Additional analysis based on the LLaMA 3.1 model is provided in Appendix F of our paper, with results summarized in Table 1 below. The methods \"Duplicate and append,\" \"Insert prompt randomly,\" and \"Tag and append\" can be viewed as three approaches to enhance the model's attention to counterfactual knowledge in the context (detailed descriptions are available in Appendix F).\n\n**Table 1: Model Response Patterns Under Different Context Conditions**\n| Context Type | Factual Knowledge Response | Counterfactual Knowledge Response | Other |\n|-|-|-|-|\n| $\\mathcal{D}\\_{cf}$ | 22.7% | 54.6% | 22.7% |\n| $\\mathcal{D}\\_{cf}$ (Duplicate and append) | 22.8% | 55.6% | 21.6% |\n| $\\mathcal{D}\\_{cf}$ (Insert prompt randomly) | 22.8% | 59.9% | 17.3% |\n| $\\mathcal{D}\\_{cf}$ (Tag and append) | 23.2% | 51.3% | 25.5% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ | 56.4% | 27.7% | 15.9% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ (Duplicate and append) | 40.2% (↓16.2%) | 45.3% (↑17.6%) | 14.5% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ (Insert prompt randomly) | 51.1% (↓5.3%) | 40.9% (↑13.2%) | 8.0% |\n| $\\mathcal{D}\\_{f}\\cup \\mathcal{D}\\_{cf}$ (Tag and append) | 44.9% (↓11.5%) | 41.2% (↑13.6%) | 13.8% |\n\n\nAs shown in Table 1, when the context contains only counterfactual documents, 54.6% of responses follow counterfactual knowledge, significantly higher than responses following factual knowledge (22.7%). This phenomenon aligns with Figure 4(c) in our paper, where the information intensity of counterfactual knowledge in the output layer exceeds that of factual knowledge, and the magnitude of information intensity determines the degree of influence that response knowledge has on the output.\n\nWhen we enhance the model's attention to counterfactual knowledge in the context through three different methods, the model's output does not shift from counterfactual to factual knowledge. This is consistent with our probing conclusion that \"knowledge competition does not occur across categories\"—that is, enhancing attention to contextual knowledge does not affect the routing of memory heads to parametric knowledge.\n\nWhen the input contains both factual and counterfactual documents ($\\mathcal{D}\\_f\\cup \\mathcal{D}\\_{cf}$ setting), compared to the $\\mathcal{D}\\_f$ setting, the model's output clearly begins to favor factual knowledge (56.4% vs. 27.7%). Why does the model favor factual knowledge when both types appear simultaneously in the context? Our probing conclusion of \"superpositional reconciliation\" explains this well. Factual knowledge from the context and factual knowledge from parameters form a superposition, making its information intensity exceed that of counterfactual knowledge sourced only from the context. Since information intensity magnitude directly affects output, the model favors factual knowledge.\n\nSubsequently, when we enhance the model's attention to counterfactual knowledge, the proportion of counterfactual responses increases significantly. The results of this operation under $\\mathcal{D}\\_f\\cup \\mathcal{D}\\_{cf}$ setting are strikingly different from those under $\\mathcal{D}\\_f$ setting, but this is not surprising. Under $\\mathcal{D}\\_f\\cup \\mathcal{D}\\_{cf}$ setting, factual knowledge comes from both model parameters and context. When we enhance the model's attention to counterfactual context, the two compete for the attention of context heads, weakening the context heads' attention to factual knowledge. This aligns with our probing conclusion that \"knowledge competition occurs within categories.\"\n\nWe understand the reviewers' concerns. Developing better probing techniques to accurately capture the internal knowledge processing mechanisms of models remains an important and open research question that merits continued exploration and improvement.\n\n> **W2.** A key claim is that conflicting knowledge persists in residual streams and is superpositionally reconciled rather than being suppressed. However, this assertion is not sufficiently substantiated with targeted experiments. For example, the authors do not investigate whether zeroing out high-information-intensity attention heads would alter the final model prediction.\n\nFollowing the reviewer's suggestion, we provide experimental results for zeroing out context heads and memory heads in Table 2. Clearing the top-10 context heads and memory heads with the highest information intensity causally affects the model's output. This experimental finding, together with the correlation analysis provided in our response to W1, sufficiently demonstrates the interpretability of information intensity for model output behavior.\n\n**Table 2:The causal impact of context heads and memory heads on model output behavior**\n| Experimental Setting | Response Consistent with Factual Knowledge | Response Consistent with Counterfactual Knowledge | Other |\n|---------------------|-------------------------------------------|--------------------------------------------------|-------|\n| $\\mathcal{D}\\_{cf}$ | 22.7% | 54.6% | 22.7% |\n| $\\mathcal{D}\\_{cf}$ w/o Context Head | 34.9% | 43.6% | 21.5% |\n| $\\mathcal{D}\\_{cf}$ w/o Memory Head | 9.2% | 65.5% | 25.3% |\n\nThe response addresses the reviewer's concern about insufficient experimental substantiation by providing targeted ablation experiments that demonstrate the causal impact of high-information-intensity attention heads on model predictions, thereby supporting the claim about superpositional reconciliation of conflicting knowledge in residual streams.\n\n> **Q1.** Have the authors considered how toxic contextual knowledge that might trigger LLMs' safety mechanisms affects internal knowledge reconciliation? Would similar patterns of attention routing still hold?\n\nThank you for raising this important point. We hypothesize that toxic contextual knowledge may exhibit distinct internal processing patterns. RLHF alignment training could potentially suppress contextual heads' attention to such harmful knowledge. This represents a critical direction for future research with significant implications for RAG system robustness and alignment. While our current findings are limited to factual knowledge conflicts, they provide a foundational methodology that can be extended to investigate safety-knowledge interactions—an important area we hope to explore in subsequent work.\n\n\n> **Q2.** To what extent do the [START] and [END] tokens introduced for probing influence the internal computation of LLMs? Specifically, how much attention do LLMs allocate to these OOD tokens?\n\nBased on the responses to W1 and W2, we can see that the probe results align well with the model's actual output behavior. We believe the influence of [START] and [END] tokens, as well as LoRA updates, on LLM working modes can be understood as follows. Under normal operating conditions, LLMs respond to user instructions by continuously predicting the next token. At each time step t, their computation is biased toward the token currently being predicted. Obviously, not every target token at each step is knowledge-intensive. We believe that [START] and [END] tokens may serve as indicators for the model to simulate the computations executed when predicting tokens corresponding to the target entity."}]

Author rebuttal:

Meta-review: This paper presents a comprehensive study on how LLMs reconcile internal, parametric knowledge with external, contextual knowledge provided in RAG settings. The problem of knowledge reconciliation and attribution is of critical importance for the reliability and trustworthiness of modern AI systems. To investigate this, the authors introduce a novel and clever entity-aware probing framework that uses special tokens and a lightweight LoRA adapter to trace the flow of information related to specific entities through the transformer layers.

The reviewers raised valid concerns regarding the experimental design, the potential influence of the probing methodology, and the generalizability of the conclusions. However, the authors provided an exemplary rebuttal, including additional causal experiments where they ablated specific attention heads to observe the direct impact on model outputs. This new evidence substantially strengthened the paper's claims and successfully addressed the reviewers' concerns, leading to a consensus of acceptance.

This is a well-executed and insightful paper that makes a significant contribution to our understanding of knowledge processing in LLMs. The work is timely, the methodology is sound, and the conclusions are well-supported. While the work primarily uses synthetically generated conflicts, this approach provides a controlled environment necessary for this type of detailed analysis. The findings lay a strong foundation for future investigations into more complex, real-world knowledge conflicts. I encourage the authors to continue their work in this promising direction.

I recommend accepting this paper.
Decision: Accept (spotlight)