Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models (2024)

Abstract: Recently, retrieval augmentation and tool augmentation have demonstrated a remarkable capability to expand the internal memory boundaries of language models (LMs) by providing external context. However, internal memory and external context inevitably clash, leading to knowledge conflicts within LMs. In this paper, we aim to interpret the mechanism of knowledge conflicts through the lens of information flow, and then mitigate conflicts by precise interventions at the pivotal point. We find there are some attention heads with opposite effects in the later layers, where memory heads can recall knowledge from internal memory, and context heads can retrieve knowledge from external context. Moreover, we reveal that the pivotal point at which knowledge conflicts emerge in LMs is the integration of inconsistent information flows by memory heads and context heads. Inspired by the insights, we propose a novel method called Pruning Head via PatH PatcHing (PH3), which can efficiently mitigate knowledge conflicts by pruning conflicting attention heads without updating model parameters. PH3 can flexibly control eight LMs to use internal memory (↑ 44.0%) or external context (↑ 38.5%). Moreover, PH3 can also improve the performance of LMs on open-domain QA tasks. We also conduct extensive experiments to demonstrate the cross-model, cross-relation, and cross-format generalization of our method. Our code is publicly available at GitHub 1.

heading: 1 Introduction
text: Language models (LMs) (Brown et al., 2020; Touvron et al., 2023; OpenAI, 2023) have memorized a substantial amount of factual knowledge during pre-training, and stored the knowledge within their parameters as internal memory (i.e., parametric knowledge) (Meng et al., 2022). During the inference phase, LMs rely on their internal memory to
1https://github.com/jinzhuoran/MConflict/ *Corresponding author.
understand and generate text. However, the internal memory may be limited or outdated, making LMs prone to producing factually incorrect content.
To alleviate the problem, one promising solution is to employ additional retrievers or tools to augment LMs by providing external context (i.e., non-parametric knowledge). Nevertheless, internal memory and external context can often contradict each other, which is known as knowledge conflicts (Longpre et al., 2021; Chen et al., 2022; Xie et al., 2023; Yu et al., 2023). Recent works have mainly investigated the behavior and preference of LMs, attempting to determine whether these models are more inclined towards internal memory or external context when faced with knowledge conflicts. However, there is a limited understanding of the underlying mechanism of knowledge conflicts. Insights into the mechanism will facilitate precise interventions at the pivotal point to mitigate knowledge conflicts, which can not only empower LMs to more reliably adhere to internal memory (e.g., ignoring misleading external context) but also enhance faithfulness in generating text based on external context (e.g., correcting outdated memory).
In this paper, we reveal that the pivotal point at which knowledge conflicts emerge in LMs is the integration of inconsistent information flows by various attention heads in later layers. To investigate this, we consider a simple factual recall task (i.e., subject attribute prediction) inspired by the work of Yu et al. (2023). As illustrated in Figure 1, given the question (i.e., “What is the capital of France?”) and the conflicting external context (i.e., “The capital of France is Rome.”), the model can either use internal memory (i.e., “Paris”) or external context (i.e., “Rome”) to predict the subject’s attribute. Following this, we present a set of “topdown” analyses to locate the pivotal point where conflicts emerge and to identify the model components that are significant in knowledge conflicts, which primarily involves the following three steps:
1193
Step 1: We start by answering the first question “What function do model components serve in knowledge conflicts?”. We knock out the activations to examine the functionality of multi-head attention (MHA) blocks and feed-forward network (FFN) blocks. We find that FFNs enrich the semantic information of input elements in early layers, while MHAs play an important role in passing information to the last token in later layers; Step 2: Based on this, the second question naturally arises, namely “When and where do MHAs pass information to the last token?”. We investigate the MHAs by knocking out the attention weights from the last token to other input elements. Results reveal that the question information is first propagated to the last token, and then the last token extracts attribute information from the subject and the attribute in the context; Step 3: Inspired by this, we aim to answer the final question “How do MHAs extract attribute information under knowledge conflicts?”. We find that some attention heads in late MHAs play opposite roles, where memory heads can recall attributes from internal memory, and context heads can retrieve attributes from external context. According to our findings, the mechanism by which LMs use both internal memory and external context can be summarized as three stages in Figure 1: (1) Enriching semantic information; (2) Propagating question information; and (3) Extracting attribute information, where knowledge conflicts arise at the third stage, due to the inconsistent information flows between memory heads and context heads.
Inspired by our insights into knowledge conflicts, we propose a minimally-invasive control method called Pruning Head via PatH PatcHing (PH3),
which can efficiently mitigate knowledge conflicts by intervening on attention heads without updating model parameters. First, we use the path patching (Goldowsky-Dill et al., 2023; Wang et al., 2023a) technique to localize important memory heads and context heads. Our method can avoid the noise interference of other heads, enabling a more accurate calculation of the importance score for the target head. Then, we perform structured pruning on those negative attention heads to mitigate conflicts. In this way, our method can flexibly control LMs to use internal memory or external context. Experimental results on the World Capital dataset show that our method can not only reliably and consistently increase the average internal memory usage rate of eight LMs by 44.0% (from 49.7% to 93.7%) but also increase the external context usage rate by 38.5% (from 50.3% to 88.8%). PH3 also enables LMs to generate answers more faithfully according to retrieved passages in open-domain QA tasks. We conduct extensive experiments to demonstrate the cross-model (e.g., from GPT series to LLaMA2 series), cross-relation (e.g., from World Capital to Official Language), and cross-format (e.g., from triple format to document format) generalization. Our contributions are summarized as follows:
• We perform an exploration into the mechanism of interpreting knowledge conflicts, and reveal that memory heads and context heads at later layers can cause knowledge conflicts when inconsistent information flows merge. • We propose a novel method called Pruning Head via PatH PatcHing (PH3), which can efficiently mitigate knowledge conflicts by pruning those conflicting attention heads.
• We demonstrate that our PH3 can flexibly control LMs to use internal memory (↑ 44.0%) or external context (↑ 38.5%). We also prove the cross-model, cross-relation, and cross-format generalization ability of our method.

heading: 2 Background
text: In this work, we mainly focus on the autoregressive transformer-based language models. Given a sequence of input tokens x = [x1, · · · , xN ], the LM G first embeds each token xi into a vector x0i ∈ Rd using an embedding matrix E ∈ R|V|×d, over a vocabulary V . The input embeddings are processed by L transformer layers. Each layer consists of an MHA and an FFN. Formally, the hidden state xℓi of token xi at layer ℓ is calculated as:
xℓi = x ℓ−1 i + a ℓ i +m ℓ i , (1)
where aℓi and m ℓ i are the outputs from the MHA block and the FFN block in the ℓ-th layer. Then, the vocabulary head ϕ(·) and the softmax function σ(·) predict the output probability:
pLi = σ ( ϕ ( xLi )) . (2)
MHA. A MHA block consists of M attention heads, which are capable of aggregating global information from the hidden states (Halawi et al., 2023; Wang et al., 2023b). An individual attention head h in layer ℓ consists of three learnable matrices, Wℓ,hQ ,W ℓ,h K ,W ℓ,h V ∈ Rd× d M . Formally, for
the input Xℓ−1 = [ xℓ−11 , · · · ,xℓ−1N ] in layer ℓ:
Aℓ = [ Hℓ,1; · · · ;Hℓ,M ] Wℓo, (3)
Hℓ,h = sℓ,hXℓ−1Wℓ,jV , (4)
sℓ,h = σ   ( Xℓ−1Wℓ,hQ )( Xℓ−1Wℓ,hK )T √ d/M   (5)
where Aℓ = [ aℓ1, · · · ,aℓN ] is the MHA block’s output. Wℓ,hO ∈ Rd×d is a learnable output matrix.
FFN. A FFN block can work as a key-value memory to store factual knowledge (Geva et al., 2021), enriching the hidden states of token i:
mℓi = f (( xℓ−1i + a ℓ i ) Wℓ1 ) Wℓ2. (6)

heading: 3 Experimental Setup

heading: 3.1 Tasks
text: In this paper, we conduct controlled experiments to construct knowledge conflicts, wherein the internal memory is factual while the external context is counterfactual. To avoid the LM being influenced by other irrelevant factors (i.e., reasoning ability), we adopt a simple factual recall task (Geva et al., 2023), which requires predicting the corresponding attribute am based on the given subject s and relation r. Building on previous work (Yu et al., 2023), we use the World Capital dataset to interpret this problem in §4, where the LM needs to predict the capital city of the country based on the question q:
Q: What is the capital of {s}? A:
We retain those questions that the LM can correctly predict the factual attributes am based on internal memory, then provide the counterfactual attributes ac in the external context c to construct conflicts:
The capital of {s} is {ac}. {q}
To mitigate knowledge conflicts, we further construct three datasets for verifying the generalization of our method in §5, including the Official Language, Country, and Continent datasets. We also generate a more complex World Capital D dataset based on the World Capital dataset, using gpt-3.5-turbo to rewrite the external context from triplet form into document form. More details about these datasets are shown in Appendix B.

heading: 3.2 Models
text: We analyze two GPT-series LMs: GPT-2 XL (Radford et al., 2019) and GPT-J (Wang and Komatsuzaki, 2021) in §4. Additionally, we also validate the effectiveness of our method on six LMs: OPT1.3B, OPT-2.7B (Zhang et al., 2022), Pythia-6.9B, Pythia-12B (Biderman et al., 2023), LLaMA2-7B and LLaMA2-13B (Touvron et al., 2023) in §5.

heading: 4 Interpreting Knowledge Conflicts
text: We utilize a “top-down” analysis approach to locate the pivotal point where conflicts emerge and to identify the model components that are significant in knowledge conflicts. We start by examining the functionality of model components by knocking out activations, and reveal that MHAs in the middle and late layers play a crucial role in passing information to the last token (§4.1). Then, we further investigate MHAs by knocking out the attention weights. We find the question information is first
passed to the last token, then the last token extracts information from the subject and the attribute in the context (§4.2). Last, we discover that some attention heads in later MHAs play opposite roles in conflicts, where memory heads can recall knowledge from internal memory, and context heads can retrieve knowledge from external context (§4.3).

heading: 4.1 Examining Component Functionality
text: We start by exploring the functionality of model components (including FFNs and MHAs across various layers) in knowledge conflicts.
Experiment 1: Knocking Out Component. We examine which component in the transformer layer is critical for the attribute prediction by knocking out activations. Then, we divide the input into six elements for analysis: context subject sc, context relation rc, context attribute ac, question subject sq, question relation rq, and the last token xN . To measure the impact on the final prediction results, we zero-out the updates to the specified input element from the MHA and FFN blocks within each layer. For example, to intervene in the update of the ℓ-th MHA (FFN) to the input element sc, we set aℓ
′ i = 0 (m ℓ′ i = 0) for i in the token range of sc and ℓ′ = max (1, ℓ−W/2) , · · · ,min (L, ℓ+W/2), where W denotes the window size. We define the effect of a model component as the change in the original prediction probability after knocking it out.
Results. Figure 2 illustrates the effect of model components (FFNs and MHAs) in GPT-2 XL with the window size W = 5. Our observation reveals that destroying the FFN blocks in the early layers has a significant effect on the prediction probability while destroying the FFN blocks at the late layers shows minimal or no impact (Figures 2a and 2d). Moreover, the MHA blocks at the middle and late layers are crucial for the last token (Figures 2b and 2e). A possible explanation of the model’s behavior on the factual recall task is that the early FFNs first enrich the semantic information of input elements, and then the enriched semantic information about attributes is extracted to the last token via late MHAs, where knowledge conflicts may arise at the later stage. To verify this hypothesis, we will examine the attribute extraction function of MHAs.
Experiment 2: Extracting Attributes via MHAs. We adopt the extraction rate (Geva et al., 2023) to examine the attribute extraction function of MHAs. We apply the early exit (Schuster et al., 2021; Geva et al., 2022) to project the MHA update aℓN for the last token xN over the vocabulary. Then we check whether the top token tℓ of each update aligns with the attribute t∗ predicted at the final layer L:
t∗ = argmax ( pLN ) , (7)
tℓ = argmax ( σ ( ϕ ( aℓN ))) . (8)
We consider that the MHA correctly performs attribute extraction when t∗ = tℓ. For comparison, we also examine the extraction rate of FFNs.
Results. As illustrated in Figures 2c and 2f, it is evident that the attribute extraction rate of MHAs significantly exceeds that of FFNs. Moreover, attribute extraction mainly takes place at the 24-48 layers. Results for GPT-J show similar trends in Appendix C. The above findings motivate us to conduct an in-depth study on the information flows of MHAs from input elements to the last token.

heading: 4.2 Tracing Information Flow
text: The analysis presented above confirms that the last token extracts attribute information for prediction through MHA blocks. Following this, we explore the order and importance of the information flow from the various elements to the last token.
Experiment 3: Blocking Information Flow. We localize the information propagation from the input elements (including sc, rc, ac, sq and rq) to the last token by knocking out attention edges between them. For example, to block the information flow from the input element sc to the last token xN in the layer ℓ, we set the attention weight sℓ,h [N, i] = 0 for i in the token range of sc, h = 1, · · · ,M , and ℓ′ = max (1, ℓ−W/2) , · · · ,min (L, ℓ+W/2). In this way, we can restrict the last token from attending to the target element. If blocking the information propagation between them has a significant impact on the original prediction probability, this indicates that it is a crucial information flow.
Results. Figure 3 illustrates the information flow in GPT-2 XL with the window size W = 9. We can observe that in the early to middle layers, blocking
the attention to the question relation leads to a decrease in the prediction probability. Similarly, in the subsequent layers, blocking the attention to the question subject also results in a decrease in the prediction probability. This suggests that the critical relation and subject information in the question are sequentially transmitted to the last token.
Then, in the middle to late layers, blocking the attention to the context subject and context attribute has the opposite effect on the final prediction probability. Taking Figure 3a as an example (when the model predicts the attribute based on internal memory), blocking the attention to the context attribute can improve the prediction probability, however, blocking the attention to the context subject can reduce the prediction probability. This suggests that the last token can extract the internal knowledge from the context subject, and extract the external knowledge from the context attribute. In addition, the last token also extracts a certain degree of internal knowledge from the question subject. Results for GPT-J show consistent trends in Appendix C.
Overall, this shows that there are two specific stages in the process of information flow passing to the last token: (1) the question information is first passed to the last token; (2) the last token extracts or copies the attribute from the context subject or the context attribute. In the later stage, knowledge conflicts arise during the process of merging inconsistent information flows from MHAs.
Experiment 4: Extending to Conflicts between Contexts. We extend our analysis to a more complex scenario in which the model is presented with both supporting context and conflicting context relative to internal memory. Supporting context and conflicting context contain am and ac respectively:
C1: The capital of {s} is {am}. C2: The capital of {s} is {ac}. {q}
We find that GPT-2 XL prefers to choose attributes consistent with internal memory 97.6% of the time. Hence, we only analyze the cases where the model makes predictions based on its internal memory.
Results. As illustrated in Figure 3c, we can observe that the question information is first passed to the last token in the first stage, which is consistent with the trend of a single conflicting context. In the second stage, a notable distinction is that the model no longer extracts the memory attribute from the subject; instead, it opts for a more straightforward approach of copying the memory attribute from the context. The above findings indicate that there exists a mechanism within MHAs capable of distinguishing and selecting between internal knowledge and external knowledge. This motivates us to conduct further analysis of MHAs.

heading: 4.3 Looking Deeper into Attention Heads
text: Attention heads serve as the fundamental component of an MHA block. For example, GPT-2 XL contains a total of 1,200 attention heads. This motivates us to conduct an investigation into the role of attention heads in handling knowledge conflicts.
Experiment 5: Discovering Important Heads. To discover the attention heads that are crucial for predicting memory attributes or context attributes, we compute the gradient-based importance score (Michel et al., 2019; Bansal et al., 2023) for each head. Given a dataset D with a set of inputs x and
outputs y, the importance score of an attention head h captures the expected sensitivity of the model to h and is computed as follows:
I l,h(D) = E(x,y) ∣∣∣∣Hl,h T ∂L(y | x) ∂Hl,h ∣∣∣∣ , (9)
where L(·) is the loss function of conditional autoregressive generation. The proxy score of head h for predicting internal memory is calculated as:
Sl,hm (Dm,D′m) = I l,h(Dm)− I l,h(D′m), (10)
where (x, am) ∈ Dm denotes the original outputs are memory attributes, (x, ac) ∈ D′m denotes replacing the original outputs with context attributes. In this way, we can also calculate the proxy score of head h for predicting external context as:
Sl,hc (Dc,D′c) = I l,h(Dc)− I l,h(D′c). (11)
We compute the proxy score of each head across different layers to discover important heads.
Results. As shown in Figure 4a (Figure 4b), the deeper color of the red square indicates a more significant contribution from this attention head to the model’s predictions based on internal memory (external context). We can observe that there are a specific number of attention heads within middleto-late layers that play opposite roles in predicting attributes. Accordingly, we refer to those heads that contribute to the prediction of memory attributes as memory heads, and those that facilitate predicting context attributes as context heads. Therefore, we
claim that they may serve in a mutually exclusive capacity during knowledge conflicts. The heatmaps of GPT-J are provided in the Appendix C.
Experiment 6: Extracting Specific Attributes via Heads. We further analyze the two types of heads discovered above to verify their role in knowledge conflicts. We rank the attention heads in descending order based on their importance scores, Sl,hm for memory and S l,h c for context, subsequently identifying the top-5% of heads as memory heads and context heads, respectively. For comparison, we also randomly choose an additional 5% of the attention heads as other heads. Then, we examine their memory extraction rate when tℓ = am, and context extraction rate when tℓ = ac.
Results. As shown in Figure 4c, memory heads and context heads are responsible for extracting different attribute information to the last token with a significant difference between memory and context extraction rates. Therefore, we discern that the pivotal point at which knowledge conflicts emerge in LMs is the integration of inconsistent information flows by memory heads and context heads.

heading: 5 Mitigating Knowledge Conflicts
text: Building on the above insights, we propose a novel method called Pruning Head via PatH PatcHing (PH3) to efficiently mitigate knowledge conflicts by intervening on attention heads without the need to update model parameters (§5.1). Then, we conduct extensive experiments to show that our method can flexibly control LMs to use internal memory or
external context (§5.2). Moreover, we analyze the generalization capability of our method (§5.3).

heading: 5.1 Method
text: Our method consists of two stages, first identifying the important heads through path patching, then intervening on these heads via structured pruning.
Localizing Memory Heads and Context Heads via Path Patching. When we use the gradientbased method in §4.3 to estimate the importance score of the target head h, it is subject to interference from other heads. The calculated gradients may not fully reflect the contribution of the target head, but rather a mixture of the influences from other heads. Therefore, we adopt the path patching technique (Goldowsky-Dill et al., 2023; Wang et al., 2023a) to analyze the causal relationship between the head h and the output attribute (including am and ac) in conflicts. To calculate the important score Sℓ,hc of the target head h, our path patching method consists of three steps shown in Figure 5:
1. Run on the original input x ∈ Dc to record the original activations of all heads; 2. Run on the corrupted input x to record the corrupted activations of all heads, where x is:
The capital of {s} is ⟨unk⟩. {q}
where ⟨unk⟩ is the special token; 3. Run on the original input x, while keeping
all the heads frozen to their activations on x, except for the target head h whose activation is set on x. Then measure the important score as the change of output logits.
The important score Sℓ,hc of head h is computed as:
Sl,hc (Dc) = E(x)[(Px(ac)− Px(am)) − ( P x (ac)− P x(am) ) ].
(12)
We adopt similar steps to calculate the importance score Sℓ,hm of the target head h for memory attribute prediction in Appendix D. We also provide the importance score heatmaps of memory and context heads for various models in Appendix E, and our method can clearly distinguish between them.
Pruning Attention Heads to Mitigate Knowledge Conflicts. By ranking all the attention heads in ascending order based on the importance score Sl,hc (Sl,hm ), we can prune the top-k% attention heads that negatively impact the model’s capability to predict context (memory) attributes, thereby enhancing the model’s ability to utilize external context (internal memory). To prune a head h in layer ℓ in practice, we set Hℓ,h to be the zero matrix.

heading: 5.2 Experiment
text: Setups. We evaluate our method on five datasets, including World Capital, World Capital D, Official Language, Country, and Continent. To verify the generalization of PH3, we only calculate the importance scores of the attention heads on the World Capital dataset, and then directly evaluate PH3 on other datasets. We also select 1,000 test samples from an open-domain QA dataset NQ (Kwiatkowski et al., 2019), providing the LM with the top-5 retrieved passages, and ensuring that at least one relevant passage is among them. We validate the effectiveness of PH3 on eight LMs.
Metrics. We use the internal memory usage rate RM = fmfm+fc+fo and the external context usage rate RC = fcfm+fc+fo to assess how effectively the method controls the reliance of LMs on either internal memory or external context, where fm is the frequency of relying on internal memory, fc is
Method GPT-2 XL GPT-J OPT-2.7B
Base 45.6 54.8 51.4 Prompt 47.6 57.4 54.1 CAD 44.5 55.0 50.2 Gradient 45.3 55.0 50.8
the frequency of relying on external context, and fo is the frequency of other answers. For the opendomain QA task, we use Recall to evaluate whether the model can provide correct answers based on the retrieved passages following Adlakha et al. (2023).
Baselines. We compare with the following baselines: (1) Prompt: We instruct the LM to generate answers based on internal memory or external context through specific prompts; (2) CAD: Shi et al. (2023a) leverage contrastive decoding (Li et al., 2023b) to encourage the LM to attend to its context during generation; (3) Gradient: We replace our path patching method with the gradient-based method to discover the attention heads. We select the optimal pruning rate k on the development set for both Gradient and PH3. More details about hyperparameter settings are in Appendix B.2.
Results. Table 1 shows the results of GPT-2 XL, GPT-J and LLaMA2-7B, and more results of other models are in Table 3. Throughout our experiments, we note the following key observations:
(1) PH3 significantly outperforms other baselines. Experimental results show that PH3 can not only increase the average internal memory usage rate of eight LMs by 44.0%, but also increase the average external context usage rate by 38.5%. When PH3 is combined with Prompt, it can more effectively control the LMs to use external context.
(2) As shown in Table 2, PH3 can also achieve an average 6.2% Recall improvement on open-domain QA tasks. By pruning a small number of negative context heads, PH3 can make LMs generate answers more faithfully based on retrieved passages.
(3) Although Prompt and CAD can effectively increase the external context usage rate, there are limitations. CAD cannot directly enhance internal
memory, and Prompt may even have the opposite effect. In contrast, our method offers a viable solution to enhance the internal memory usage rate.

heading: 5.3 Analysis
text: We conduct a thorough analysis of the generalization ability of PH3. For cross-model generalization, PH3 is effective across a wide range of models. This shows that our method is not limited to small models, but can also be adopted on relatively large models, including the popular LLaMA2 series. For cross-relation generalization, by intervening on the attention heads discovered on World Capital, our method can also well resolve knowledge conflicts on other relation types. This indicates that PH3 does not identify attention heads specific to a certain type of relation. Instead, it identifies universal memory and context heads. For cross-format generalization, PH3 can transfer well from triple-form context to document-form context. This indicates that our method does not merely remember the relative positions of elements in context, but is capable of understanding the external context. Compared to the Gradient, our method has demonstrated superior generalizability. We also analyze the impact of the number of pruning heads in Appendix G.

heading: 6 Conclusion
text: In this paper, we perform an exploration into the mechanism of interpreting knowledge conflicts and reveal that memory and context heads in later layers can cause knowledge conflicts when merging inconsistent information flows. Based on our insights, we propose a novel method called Pruning Head via PatH PatcHing (PH3), which can mitigate knowledge conflicts by pruning those conflicting attention heads. We prove that PH3 can flexibly control LMs to use internal memory or external context. We also demonstrate the cross-model, cross-relation, and cross-format generalization.
Limitations
For further study, we conclude some limitations of our work as follows:
• Similar to previous works on mechanism interpretability that adopt tasks such as antonym generation (Todd et al., 2023), fact recall (Meng et al., 2022; Geva et al., 2023), arithmetic operation (Hanna et al., 2023; Stolfo et al., 2023), and text classification (Bansal et al., 2023; Wang et al., 2023b), our work
also selects a relatively simpler task to interpret the mechanism behind knowledge conflicts. Simple tasks enable us to better control variables and minimize external distractions. In the future, we plan to extend our analysis to more complex and realistic scenarios (Li et al., 2024), such as where irrelevant information is present within the external context, or where the model needs to reason with both internal and external knowledge. • Although our research has delved into the attention heads in LMs, there may be more basic elements involved in knowledge conflicts. Furthermore, the memory and context heads we have discovered may not only be responsible for extracting knowledge from internal memory or external context. These heads may also have other functions, such as helping the model capture global dependencies of input texts. By pruning these heads, the original capabilities of the model may be affected. Therefore, we will further explore mitigating knowledge conflicts through more subtle intervention methods.
In summary, the mechanism behind knowledge conflicts remains a largely unexplored area, and we hope our work can offer some useful insights for further research.

heading: Acknowledgements
text: This work is supported by the National Key Research and Development Program of China (No. 2022ZD0160503), the National Natural Science Foundation of China (No. 62176257).
Ethics Statement
To enhance the reproducibility of our research, we will make all source code and datasets publicly available upon the acceptance of this paper. Our work focuses on uncovering the mechanisms behind knowledge conflicts in LM, thereby better controlling the model in retrieval augmentation and tool augmentation. Through effective intervention, our method can make the LM more controllable and trustworthy. On the one hand, it can prevent prompt injections from attacking the model, and on the other hand, it can correct the biased knowledge that the model learned during pre-training. Nonetheless, the impact of head pruning on the model’s original capabilities remains unexplored. These factors should be taken into careful consider-
ation for future research.

heading: A Related Work
text: A.1 Investigating Knowledge Conflict
To address the issue of hallucination, one promising solution is to employ retrieval-augmented generation (RAG) (Lewis et al., 2020; Izacard and Grave, 2021; Ram et al., 2023; Shi et al., 2023b; Jin et al., 2023). Owing to challenges like misinformation, divergent perspectives and the evolving nature of knowledge, knowledge conflicts widely exist in RAG. Previous research (Longpre et al., 2021; Chen et al., 2022; Yu et al., 2023; Xie et al., 2023; Wang et al., 2023c; Neeman et al., 2023; Jin et al., 2024) on knowledge conflicts primarily seek to answer the question: do language models prefer internal memory or external context? Yu et al. (2023) find that language models are more inclined to internal memory as the frequency of a fact in the pre-training corpus increases. Xie et al. (2023) demonstrate that large language models (LLMs) are highly receptive to external conflicting evidence. They also reveal that when both supportive and contradictory evidence to their internal memory are present, LLMs show a strong confirmation bias and tend to cling to their parametric memory. The above observed phenomena contribute to a better understanding of knowledge conflicts. However, the underlying mechanism of knowledge conflicts remains unclear. We observe that knowledge conflicts arise when the late attention heads integrate different information flows from internal memory and external context.
A.2 Resolving Knowledge Conflict
Existing work (Shi et al., 2023a; Zhou et al., 2023; Li et al., 2023a; Yu et al., 2023; Qian et al., 2023) has conducted preliminary exploration into the mitigation of knowledge conflicts. Shi et al. (2023a) propose a simple method to encourage the LM to attend to the external context via contrastive decoding (Li et al., 2023b). Yu et al. (2023) use head attribution to identify individual attention heads that either promote the memorized answer or the in-context answer, then scale the value vector of these heads to increase the rate of the in-context answers. Our work is inspired by their exploration of attention heads, and we propose further analysis to improve understanding of the way knowledge conflicts are formed. Furthermore, while most existing methods (Shi et al., 2023a; Yu et al., 2023) primarily focus on improving the model’s faithfulness to the context, enabling the model to adhere
to its internal memory remains a challenging task.
A.3 Mechanistic Interpretability
Recently, there has been a growing interest in the mechanistic interpretability (Cammarata et al., 2020; Elhage et al., 2021) of parametric knowledge in LMs, with efforts focusing on reverse engineering the computational processes of model parameters. Dai et al. (2022) use a knowledge attribution method (Hao et al., 2021) to identify the knowledge neurons in FFNs. Meng et al. (2022) reveal that FFNs at a range of middle layers can recall facts by using the causal mediation analysis method (Vig et al., 2020). Geva et al. (2023) find that knowledge extraction is typically done via attention heads. Besides, there are some works investigating LMs in mathematical reasoning (Hanna et al., 2023; Stolfo et al., 2023) and in-context learning (Hendel et al., 2023; Olsson et al., 2022; Bansal et al., 2023). Besides, there are some studies (Yang et al., 2023; Sakarvadia et al., 2023a,b; Zhang and Nanda, 2024; Yuan et al., 2024) focused on interpreting attention heads in LMs. Our work is highly inspired by previous wisdom in mechanistic interpretability, focusing on interpreting and mitigating knowledge conflicts in LMs.
B Implement Details
B.1 Datasets
We construct Official Language, Country, and Continent datasets by sampling knowledge triples from Wikidata. The Official Language dataset requires the LM to predict the official language of the given city or country:
The official language of {s} is {ac}. Q: What is the official language of {s}? A:
The Country dataset requires the LM to predict the country to which the given city belongs:
The city {s} is located in {ac}. Q: Which country is the city {s} in ? A:
The Continent dataset requires the LM to predict the continent on which the given country is located:
{s} is in the continent of {ac}. Q: Which continent is {s} located in ? A:
We also generate a more complex World Capital D dataset based on the World Capital dataset, using gpt-3.5-turbo to rewrite the external context from triplet form into document form.
B.2 Hyperparameter Settings
Our implementation is based on HuggingFace’s Transformers2, PyTorch3 and baukit4. For the Prompt method, we use the following prompt to enhance the internal memory:
Please answer the question based on your internal memory, ignoring the given context.
and we use the following prompt to enhance the external context:
Please answer the question based on the given context, ignoring your internal memory.
For Gradient and PH3, we select the optimal pruning rate k ∈ {1, 3, 5, 7, 9, 15} on the development set with 200 samples. To mitigate knowledge conflicts, setting the pruning rate k of PH3 to 5 usually achieves excellent results. For enhancing the opendomain QA capabilities, we usually set the pruning rate k of PH3 to 3. Details about the models used in this paper are in Table 4. All experiments are conducted with NVIDIA GeForce RTX A6000 GPUs.
2https://github.com/huggingface/transformers/ 3https://github.com/pytorch/pytorch/ 4https://github.com/davidbau/baukit/

heading: C Additional Results for GPT-J
text: We provide here additional results for GPT-J. Figures 6 and 7 show the effect of FFNs and MHAs on internal memory, and Figures 8 and 9 show the effect of FFNs and MHAs on external context. Figures 10 and 11 illustrate the information flow in GPT-J with the window size W = 9. Figure 12 shows the information flow in GPT-J when providing both supporting context and conflicting context relative to internal memory. Figures 13 and 14 show the gradient-based important scores of memory heads and context heads in GPT-J.

heading: D Method Details
text: To calculate the important score Sℓ,hm of the target head h, our path patching method consists of the following three steps:
1. Run on the original input x ∈ Dm to record the original activations of all heads; 2. Run on the corrupted input x to record the corrupted activations of all heads, where x is:
The capital of ⟨unk⟩ is {ac}. Q: What is the capital of ⟨unk⟩ ? A:
where ⟨unk⟩ is the special token;
3. Run on the original input x, while keeping all the heads frozen to their activations on x, except for the target head h whose activation is set on x. Then measure the important score as the change of output logits. The important score Sℓ,hm of head h is computed as:
Sl,hm (Dm) = E(x)[(Px(am)− Px(ac)) − ( P x (am)− P x(ac) ) ].
(13)

heading: E Heatmaps of Attention Heads
text: We calculate the important scores of memory heads and context heads via our path patching method, then provide the heatmaps for GPT-2 XL (Figures 15 and 16), GPT-J (Figures 17 and 18), OPT-1.3B (Figures 19 and 20), OPT-2.7B (Figures 21 and 22), Pythia-6.9B (Figures 23 and 24), Pythia-12B
(Figures 25 and 26), LLaMA2-7B (Figures 27 and 28) and LLaMA2-13B (Figures 29 and 30). The red squares indicate heads that have a significant positive impact, while the blue squares represent heads that have a negative effect.

heading: F Additional Experimental Results
text: We report experimental results in Table 3 and 2.

heading: G Number of Pruning Heads
text: As shown in Figures 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41 and 42, we analyze the impact of the number of pruning heads (sparsity ratio) on the Gradient and PH3 methods.

heading: Future Work
text: These factors should be taken into careful consider-
ation for future research