Knowledge Circuits in Pretrained Transformers
Authors: Yunzhi Yao
Ningyu Zhang
Zekun Xi
Mengru Wang
Ziwen Xu
Shumin Deng
Huajun Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Knowledge Mechanism
Circuit Theory
Large Language Models
Primary area: natural_language_processing

Abstract: The remarkable capabilities of modern large language models are rooted in their vast repositories of knowledge encoded within their parameters, enabling them to perceive the world and engage in reasoning. The inner workings of how these models store knowledge have long been a subject of intense interest and investigation among researchers. To date, most studies have concentrated on isolated components within these models, such as the Multilayer Perceptrons and attention head. In this paper, we delve into the computation graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. The experiments, conducted with GPT2 and TinyLLAMA, has allowed us to observe how certain information heads, relation heads, and Multilayer Perceptrons collaboratively encode knowledge within the model. Moreover, we evaluate the impact of current knowledge editing techniques on these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies. Finally, we utilize knowledge circuits to analyze and interpret language model behaviors such as hallucinations and in-context learning. We believe the knowledge circuit holds potential for advancing our understanding of Transformers and guiding the improved design of knowledge editing.

Knowledge Circuits in Pretrained Transformers
Yunzhi Yao!
Ningyu Zhang' Zekun Xi
Mengru Wang'
Ziwen Xu'
Shumin Deng" Huajun Chen 33-
Zhejiang University
National University of Singapore. NUS-NCS Joint Lab, Singapore 3 Zhejiang Key Laboratory of Big Data Intelligent Computing
{yyztodd, znjjgngg edu.cn
Abstract
The remarkable capabilities of modern large language models rooted in their vast repositories of knowledge encoded within their parameters, enabling them perceive the world and engage in reasoning. The inner workings of how these models store knowledge have long been a subject of intense interest and investi- gation among researchers. To date, most studies have concentrated on isolated components within these models, such as the Multilayer Perceptrons and attention head. this paper, we delve into the computation graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. The experiments, conducted with GPT2 and Tiny LLAM,, have lowed us observe how certain information heads, relation heads, and Multilayer Perceptrons collaboratively encode knowledge within model. Moreover, we evaluate the impact of current knowledge editing techniques these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies. Finally, utilize knowledge circuits analyze and inter- pret language model behaviors such as hallucinations and in-context learning. We believe the knowledge circuits hold potential for advancing understanding of Transformers and guiding the improved design of knowledge editing'.
Introduction
"Knowledge is power, and when embodied in the form of new technical inventions and mechanical discoveries it is the force that drives history." [1, 2], Bacon's words are vividly re-enacted in the of Large Language Models (LLMs) [3, 4], as we witness their immense power in reshaping human society and redefining our understanding denied is that knowledge encapsulated within these models empowers their capabilities in reasoning, perceiving the world, and engaging in human-lite like communication. Nevertheless, these powerful models are not without their flaws. They still struggle with issues such as hallucinations [5-7], unsafe norms [8. 9], and offensive behaviors [10, 11] and these problems are exacerbated by the enigmatic internal mechanisms of knowledge storage within language models.
machine intelligence. One thing that cannot be
Recently, the research community has devoted significant efforts to unraveling the knowledge storage mechanisms of these models. Various studies [12-19] have been conducted to shed light on this intricate process, aiming to enhance our understanding and improve the safety and reliability of language models. The main finding in previous work is that knowledge may primarily stored in the Multilayer Perceptrons (MLPs) of Transformer-based. language models. These MLPs function as key-value neural memory, with knowledge being stored in what are termed "knowledge neurons" (KN). Based on these findings, researchers conduct Knowledge Editing [18, 20] to update the language models' inaccurate facts, bias and unsafe content in their parametric space. Despite the
Corresponding Author.
Code and data are available https:///itthun http://hht...oooooooo/////o/eeee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) simplified Circults
Some Special Component's output
Residual Output: French
Attertion Pattern
Output logits
Attention Pattern
Oulput logits
MILPSS LIAEE
Of 148-3
MLP11
The
The
Munn
ofical
angiage
Ambii
keggang
larguages
larguage
day
MUTT
Langange
Franc
Franes
language
Agriculnee
dislect
Torg
O Loser
ONLD12
Liard Tay
Langraage
Ouver
The
France
The
O YUPT-9
offc
Marse
Stanc
franc
MuPe
French
language
François
Fruncois
Input Embod
Preros
Franç
Franco
Macron
The official for grngugg of France is
Knowledge circuit obtained from "The official language of France is French" in GPT2- Medium. Left: simplified circuit and the whole circuit is in Figure 9 in Appendix. We use --+ to skip some complex connections between nodes. Here, L15HO means the first attention head in the 15th layer and MIP12 means the multi-perception layer in the 13th layer. Right: the behavior of several special heads. The matrix on the left is the attention pattern of each attention head and the right heapmap demonstrates the output logits of the hean by mapping to the vocabulary space.
Figure 1:
initial success of these methods, there are still limitations, such as poor generalization, severe side effects, and failure to effectively utilize edited knowledge [20, 21J. which motivate us to re-think previous approaches for interpreting knowledge storage in language models. Note that previous works treat the knowledge blocks as isolated components following the Restorative Theory [22], often focusing on identifying the specific blocks that store particular knowledge. Several works [23. 24] have proposed that different types of knowledge are often located in the same areas, suggesting that the current KN thesis may be an oversimplification.
To this end, instead of solely pinpointing tiny regions where the knowledge expressed can be localized, we aim explore the cooperation between different components in Transformers like attention heads, MLPs, and embeddings, to understand how the language model stores and expresses the knowledge. Here, we introduce new perspective: Knowledge Circuits, critical subgraph in the language model to view the knowledge mechanism of Transformers. Note that Circuit, as a subgraph in the computation graph, has gained ever growiig attention in the mechanistic interpretability field [25]. Previous work [26, 27] has found several important circuits for specific tasks like Indirect Object Identification and Color Object Identification. These tasks necessitate the model search the preceding context for matching token and copy it into the next token prediction. In this work, we aim to construct knowledge circuits that require the model to utilize stored knowledge for making predictions. Our goal is to better unveil implicit neural knowledge representations, elucidate the internal mechanisms for knowledge editing, and interpret more complex behaviors of language models. Specifically, we leverage factual recall tasks and conduct experiments across various domains, including factual, social bias, linguistic, and commonsense knowledge. We utilize GPT-2 [28] and LiMA [29] to explore the potential knowledge representations and utilization mechanisms in these models. As shown in Figure (a), we construct knowledge circuits associated with various expressions of knowledge using the existing knowledge stored in the language model. Through those discovered knowledge circuits, we find many interesting phenomena and conclusions as follows:
Knowledge circuits unveil implicit neural knowledge representations. We find that even when the knowledge circuits are used independently, the language model can recall related knowledge with a significant portion of its overall performance, demonstrating the effectiveness of those discovered knowledge representations (circuits). We also delve into specific pieces of knowledge and analyze the information flow within their respective circuits, indicating that the language model tends to aggregate knowledge in the earlier to middle layers and further enhances this information in the later layers. We further uncover several special components (c.g., mover heads and relation heads)

---

in transferring information to the final token position and capturing relational information from the context (Figure (b)).
Knowledge circuits elucidate internal mechanisms for knowledge editing. We conduct experi- ments to evaluate the impact of current knowledge editing methods on the language models' original knowledge circuits. Empirically, we observe that ROME [18] tends to incorporate edited information primarily at the edited layer. Subsequent mover heads (Appendix B.2) then transport this information to the residual stream of the last token. Conversely, during fine-tuning, the edited token is directly integrated into the language model, exerting a dominant influence on subsequent predictions.
Knowledge circuits facilitate interpreting language model behaviors. We further utilize the knowledge circuits to interpret language model behaviors, such as hallucination and in-context learning. We observe that when hallucination occurs, the language model fails to correctly transfer knowledge to the final token in the earlier layers. This is evident the knowledge circuit lacks an effective "mover" head, or the mover head selects incorrect information. Additionally, we notice that several new attention heads emerge in the knowledge circuit during in-context learning.
2 Background: Circuit Theory
2.1 Preliminaries
In the context of neural network interpretability, a circuit can be conceptualized as a human- interpretable subgraph that is dedicated to executing specific tasks within a neural network model [30, 26, 31-33J. When we visualize a neural network model as a connected directed acyclic graph (DAG), denoted as G, the individual nodes represent the various components involved in the forward pass, such as neurons, attention heads, and embeddings. The edges symbolize the interactions between these components, including residual connections, attention mechanisms, and projections. A circuit, represented as C C G, emerges as a significant subgraph of G that is responsible for particular behaviors functionalities. In this paper, we focus the Transformer decoder architecture to conduct our experiments. The residual stream of Transformers has been demonstrated to be a valuable tool for mechanistic interpretability in recent works [25, 16]. The Transformer architecture typically starts with token embeddings, followed by sequence of "residual blocks" and concludes with a token unembedding. Each residual block comprises an attention layer and an MLP layer, both of which "read" their input from the residual stream (via linear projection) and "write" their output back to the residual stream through an additive projection. We can consider an attention head A (the jth attention head in layer I) as operating on the residual stream from the previous layer, R Given that Rg (where represents the input embeddings), we can reinterpret attention head A1, as processing the cumulative output of all previous attention heads and MLPs and input embedding, treating each node in the previous layers as separate input arguments. Similarly, an MLP node M, can be seen as operating embedding, and the output node O operates on the sum the input embeddings and the outputs of all attention heads and MLPs. The following equations represent the residual connections in the Transformer model, where R; is the residual stream at layer 7, and Input and Input," M are the inputs the attention and ML. layers, respectively:
the cumulative output of all previous attention heads and MLPs and input
Rí = T==RP-1 R + Σ
Ars + MM, Ro =
Input = 1+ Σ
My + Σ Ar
Pel
Input," M = I Σ My +
ΣV US 8
Av.
Pd
The computational graph & of the Transformer represents the interactions between attention heads and MLPs. The nodes in G encompass the input embedding I, attention heads At,, MLPs Mf, and the output node O, denoted as = (I, AL.j Mr, O}. The edges in the model represent the connections between these nodes, E = { Ny)) Ml. ny E NJ. A circuit C is meticulously constructed to govern specific behaviors within the model, comprising a selection of nodes No and edges Feg that are critical to the successful execution of the tasks at hand, expressed C =<< Ne, Ec >.
3

---

2.2 Circuit Discovery
To identify circuits within a language model. a key approach is to examine the model's casual mediation by systematically altering the model's edges and nodes to observe the effects perfor- mance [32, 34, 35]. The underlying principle is that critical edges or nodes are those whose removal results in notable decline in the model's predictive capabilities. Since the edges in the model's s computational graph represent the dependencies between nodes, we can simulate the absence of a particular node-to-node dependency by ablating an edge in the graph. For example, ablating an edge from A to Ai, involves replacing the contribution of Ar in the input to attention head A with zero (in the case of zero ablation) or with the mean value of head Ar. (in the case of mean ablation). The process of identifying critical edges or nodes through ablation can be broken down into the following steps: i) Overwrite the value of the edge (Ng, Nly) with a corrupted value (either zero or mean ablation), ii) Perform a forward pass through the model with the altered graph, iii) Compare the output values of the modified model with those of the original model using a chosen metric S (Details in Eq.1 ). If the performance change is below a predefined threshold т, we can consider the edge non-critical and remove it to obtain a new subgraph (/(n,, ny). In addition to ablation-basee based methods, recent works have also explored the use of sparse auto-encoders [36, 37] to identify circuits within language models. This approach involves training an auto-encoder to learn a sparse representation of the model's internal structure, which can help reveal the underlying circuitry responsible for specific behaviors or functionalities.
3 Knowledge Circuits Discovery in Transformers
3.1 Knowledge Circuits Construction
Unlike previous work [12, 18), which managed to find out the specific areas that store knowledge, we pay extra heed to the information flow that activates subsequent knowledge for answering questions. Similar to [38, 26). we write language model as a graph consisting of the input, the output, attention heads, and MLPs by considering a "residual rewrite" of the model's computational structure. For example, this residual rewrite gives us a nearly-dense graph in GPT2-medium one between every pair of (attention head, MLP, input, and output) nodes, except for attention heads in the same layer, which do not communicate with each other. In our paper, we concentrate on the task of answering factual open-domain questions, where the goal is to predict a target entity given a subject-relation pair (s, r). A knowledge triplet = (s, r, o) is often presented to the model in the form of a natural language prompt for next token prediction (e.g., "The official language of France is =s). The model G is expected to generate the target entity, which is consistent with the language model's pretraining format. To identify the circuit that is critical for predicting the target entity o for given subject-relation pair (s, r), we ablate each special edge e; = (nz Ny) in the computation graph g. We then measure the impact of ablating the edge (zero ablation in our implementation) on the model's performance using the MatchNLL loss [32] for the target
S(e,) log (s. - log log((/e((0((s.)))
(1)
If the score S(e.) is less than the predefined threshold we consider the edge to be non-critical and remove it from the computation graph, updating the temporary circuit Cremp < G/ee. We first sort the graph by topological rank following Conmy et al. [32] and traverse all edges in this manner, We derive a circuit C₁ that contributes to representing the knowledge necessary to answer the factual question:
c No. E& >
Here, Cg the circuit for the knowledge triplet kt, consisting of the nodes N₂ and edges Eg that are essential for predicting the target entity given the subject-relation pair (s,
3.2 Knowledge Circuits Information Analysis
Once we have identified the knowledge circuit, we delve deeper into the specific roles and behaviors of each node and edge within the computation graph. Our goal is to comprehend the processing and contribution of each node n, to the functionality of the circuit. Drawing on the methodologies of previous studies [16, 39, 40), we begin by applying layer normalization to the output of each node n; and then map it into the embedding space. This is achieved by multiplying the layer-normalized output by the unembedding matrix (Wu) v) of the language model: W (n.). This transformation allows

---

us to inspect how each component writes information to the circuit and how it influences subsequent computational steps. By understanding the nodes' behavior in the circuit, we can better comprehend the circuit's s structure and the key points where information is aggregated and disseminated.
Table Hit@10 of the Original and Circuit Standalone performance of knowledge circuit in GPT2- Medium. The result for Drat being 1.0 indicates that we select the knowledge for which the model provides the correct answer to build the circuit.
Deal
Drest
Type
Knowledge
#Edge
Original(g)
Circuit(C)
Original(
Random
Circuit(C))
Adj Antonym
573
0.80
1.00
0.00
0.00
0.40
Linguistic
word first letter
432
1.00
0.88
0.36
0.00
0.16
word last letter
230
1.00
0.72
0.76
0.00
0.76
object superclass
102
1.00
0.68
0.64
0.00
0.52
Commonsense
fruit inside color work location
433
1.00
0.20
0.93
0.00
0.13
422
1.00
0.70
0.10
0.00
0.10
Capital City
451
1.00
1.00
0.00
0.00
0.00
Landmark country
278
1.00
0.60
0.16
0.00
0.36
Factual
Country Language
329
1.00
1.00
0.16
0.00
0.75
Person Native Language
92
1.00
0.76
0.50
0.00
0.76
name religion
423
1.00
0.50
0.42
0.00
0.42
occupation age
413
1.00
1.00
1.00
0.00
1.00
Bias
occupation gender
226
1.00
0.66
1.00
0.00
0.66
name birthplace
276
1.00
0.57
0.07
0.00
0.57
Avg
0.98
0.73
0.44
0.00
0.47 1
3.3 Knowledge Circuits Experimental Settings
Implementations. We conduct experiments on GPT-style models, including GPT-2 medium and large. We also conduct primary experiments on LinyLLaMA [29] to validate the effectiveness of different architectures. We utilize the Automated Circuit Discovery [32] toolkit to build a circuit as an initiative of our analysis and leverage transformer lens [41] to further analyze the results. Specifically, we simply employ the MatchNLL [32] as the metric to detect the effect of the given node and edge and use zero ablation to knock out the specific computation node in the model's computation graph.
Metrics. A discovered knowledge circuit is deemed an accurate representation of specific area within the transformer's knowledge storage, thus, it should be capable of representing the knowledge independently. Following [32], we leverage the completeness of a circuit, which refers to its ability independently reproduce the behavior predictions of the full model for the relevant tasks. This property is assessed by examining whether the identified subgraph corresponds to the underlying algorithm implemented by the neural network. evaluate completeness, we first construct the eircuit using the validation data Deat for specific knowledge type and then test its performance on the test split Drest in isolation. By doing so, we can observe any changes in performance compared to the original model. We use the Hit@10 metric to measure the rank of the target entity among the top 10 predicted tokens:
M
Hit@10 = Σ
(rank, < 10)
1=1
Here, [V] represents vocabulary size, and rank, is the rank of the target entity in predictions.
Dataset. In this work, we focus on the knowledge that is already stored in the language model. We utilize the dataset provided by LRE [42] and consider different kinds of knowledge, including linguistic, commonsense, fact, and bias. We evaluate whether the knowledge is present in the language model's 's parameters under zero-shot settings using the Hit@10 metric to sample knowledge from the validation set, which is used to construct the knowledge circuit. The data statistics are in Appendix A.
4 Knowledge Circuits Unveil Implicit Neural Knowledge Representations
Knowledge Circuits Evaluation. We report the results of GPT2-Medium Table 1, which indicates that with only less than 10% of the original knowledge circuit's 's subgraph, the model can
5

---

maintain over 70% of its original performance. Additionally, we compute the random circuits by randomly deciding whether the edge should be removed and making sure the graph is connected. The random circuit is the same size as the circuit we discovered using our method. From the table, we can see that the random circuit failed to maintain the model's performance, which further enhanced the robustness and efficacy of our methods. One of the most fascinating observations is the performance improvement seen on several test datasets. For instance, the increases from 0.16 to 0.36. This suggests that the discovered knowledge circuits may encapsulate the relevant knowledge, and the model's performance on these tasks could have been hindered by noise from other components. We proceed to analyze the layer distribution of the original model G to understand the average percentage of nodes that are activated within the circuit for different knowledge domains. From Figure 2. we observe that attention and MLPs more active in the lower layers of the network, where the language model processes the input and extracts general information. To gain more comprehensive view of the information processing, we compute the average rank, change of the target token in the Deal across the layers and report the results in Figure 7. This analysis reveals the phenomenon of early decoding [40], suggesting that by the middle to the latest layers, the target entity is already present in the residual stream, and the subsequent layers in the Transformer are designed increase the probability of the current token (See discussion in the running example).
relation metric
Lingrieic
Commonszore
23-
Special Components in Knowledge Circuits. From the discovered knowledge circuits, we can find sev- eral important attention heads that demonstrate spe- cific behavior, including the mover head [31], relation head [17, 43] and mixture head [17, 43] (more defini- tions in Appendix B.2). Mover Head [31, 27] focuses on the last token of the context and attends to the subject token, functioning as a mover to transfer information, while Relation Head [17] attends to the relation token in the context and produces some relation-related to- kens that would guide the behavior of the following components. We think that these components would be accumulated by the MLP in the model, and the behavior of these special heads will be discussed in the running example part. We list some of these special components in Table 4 in Appendix. The different attention heads are responsible for expressing specific types of knowledge and may be activated by different facts. In our experiments with GPT-2 Medium and GPT-2 Large, we find that knowledge is distributed across several layers' attention heads and MLP matrices, suggesting that the target knowledge appears to have been accumulated throughout the GPT-2 model. Conversely, in TinyLLAMA, the special components are more concentrated. As depicted in Figure 7, the rank of the target entity in LLAYLLAMA experiences sharp decline around several layers, whereas in the GPT2 model, the decline is more gradual. We hypothesize that this discrepancy may be attributed to the model's ' knowledge capacity [44] and warrants further investigation.
10
0.2
0.2
04
166
0.0
0.2
0.4
66
Biss
Pactull
2.5
15
0.00
0.25
0.53
0.75
0.0
0,5
crrud autt alasa Trequrecy
CiccB actiortion trqqueccy
Figure 2: The activated circuit component distributions in Layers in GPT2-Medium.
A Running Example of Knowledge Circuit. We present a case and analyze the specific behaviors of components within the identified knowledge circuits. In particular, we find some special attention heads in the model such as the mover head and the relation head. We demonstrate the function of these heads in Figure 6 by ablating them from the circuit. Taking the factual knowledge "The official language of France is French" " as an example, we visualize the knowledge circuit in Figure 1. To express the information flow within the model more effectively, we have plotted the rank and probability of the target entity at each layer when it is mapped into the embedding space, in Figure 3. From this figure, we can see that after MLP 17, the target knowledge emerges as the top token in the residual stream, and after that layer, it undergoes an increased probability. The edges connected to MLP17 are (L14H13 + MLP17). (L14H7 + MLP17), and (L15HO- - MLP17) Here, the L14H13 is a relation head that focuses on the relation token in the context. The output of this head is relation- related tokens such as "languuge" and "Language". The attention head L14H7 is mover head that moves the information from the subject position "France" " to the last token. Previous work [31, 19] has introduced this mover head as an argument parser, which moves "France" to the last token, and the subsequent MLP conducts a function application to map "France" to "French" An intriguing

---

observation is that we can find the output of this head already contains the target answer entity, which significantly contributes to the final output (L14H7 - Output). Also, we see the probability of the subject token in Figure 3 at the last token is nearly zero across these layers. Hence, instead of the argument parser function, we consider this mover head as an extruct head proposed by Geva et al. [13], which aims to extract the related- -nftteddiifoootiion from the subject token's 's position. In the subsequent knowledge editing experiments, we can observe changes in the behavior of these types of heads. Additionally, instead of extraction in the later layers proposed by Geva et al. [13], we notice a gradual decrease in rank across all carly-to-middle layers. The MLP17 combines information from previous tokens and integrates this information to prioritize the target token at the top rank.
legesiattyy alaat Focdiw lageitat by
of kelg
Rank (log sals)
ProbobUoo
Interestingly, upon tracing the information flow to L14H7, we discovered that it is predominantly ac- tivated by L7H14, relation head, and its output features several language tokens, such as " "Arabic". We hypothesize that L7H14 may function as a signal- ing mechanism to activate the associated mover head, but this hypothesis necessitates further investigation to be confirmed. After MLP17, several attention heads, such L18H14 (a relation head) and L20H6 (a mover head), collaborated to further enhance the final prediction of the target entity.
Layer
Figure 3: The rank and probability of the tar- get entity at both the last subject token and the last token position when unembedding the intermediate layer's output for the fact "The official language of France is French".
5 Knowledge Circuits Elucidate Internal Mechanisms for Knowledge Editing
In this section, our objective is to evaluate the impact of previous knowledge editing methods and validate the effectiveness of knowledge circuits. We aim to understand why these methods may fail in certain cases and settings, which can also help deepen the understanding of the knowledge circuit.
Single Factual Knowledge Editing. Here, we adopt the ROME method [18] and FT-M [24), which aim to edit the MLP layers in the language model. The most important hyper-parameter in knowledge editing is the layer, as the same method's ' performance varies significantly via the layers. Here, we evaluate the performance of different editing layers and their effectiveness. We compare the knowledge circuits computed by the edited model with the original one, and we present results in Figure 4 and report details in Appendix D. As discussed in the previous part, the early-to- -elddde layers are the main part of aggregating the target entity o to the top rank. In the original model, the probability of the target entity "Intel" is nearly zero, and the model fails to elevate to the top rank in the vocabulary. Editing the model with ROME and FT-M both give us the correct answer but we can view different scenarios for their knowledge circuits. For ROME, as the correct information is added to the subject position, we can recognize a behavior of the Mover Head shifts from copying to extracting the edited information from the subject position. This information gradually aggregates through the subsequent layers, and by layer 15, "Intel" emerges as the top-ranked entity with its probability increasing significantly. Specially, before editing, the mover head L15H3 attends to the "controller" token and returns "controller" as the output, while in the edited model, the attention head's output moves to the "Intel", which means the model gains the information at the subject space. For FT-M, the edited model tends to directly write the knowledge into the specific component, which would greatly dominate the following component in the model. As shown in Figure 4, the output logits in MLP-0 for "Intel" more than 10, and it emerges as the top rank in the residual stream directly. This phenomenon can be found in different knowledge types and layers and we report results in Appendix D.2. However, the added knowledge may have the risk of influencing unrelated knowledge. When we test another fact "Windows server", the model still tends to give us the "Intel" answer, demonstrating the overfitting problem. This finding supports previous analysis regarding the correlation between localization and editing [45], suggesting that edits may not alter the storage but merely add signals into the knowledge circuits.
Multi-hop Factual Knowledge Editing. Multi-hop knowledge editing poses a challenging sce- nario [20, 21, 46), wherein we edit the model with new knowledge, yet the model struggles to perform reasoning using the edited information. We analyze multi-hop questions in language models [47, 48]
7

---

Pattorn
Output logits
Case Oulput
Original Model
Frttwaro
Edst Case:
Caroude
Plattors Controlter Hub is created by Microsoft.
ler
engi wer
Other Cases
04060
Kindows
Microsoft.
L1
Parform
Intel
0.00
Platfora Controller Rub is created by Intel.
Case:
Seatt cotie
ROME-LO
Intel
motherboard
Other
Keal the
IEEE
Mindows Server created
2003
Herdware
15:0
Output logits
Edit Case:
Imel
beart Sabutt
FT-LO
Milil
by
Intel.
Insight
Huawel
Other Casel
MERTISEMENT
Winess
Mindows Server
Lenovo
Computing
Intel.
O
O
Figure 4: Different behaviors when we edit the language model. In the original model, we can the mover head L15H3 actually move the original token "Controller" and other information, while for ROME, we observe the mover head select the correct information "Intel", which means ROME successfully added the "Intel" to model. For the FT layer-O editing, can find this method directly write the edited knowledge into edited component. However, we find these two editing methods would affect other unrelated input "Windows server is created by?"
to understand why current editing methods fail in these scenarios. For instance, given the fact (Thierry Mugle, "home country". France), we edit the fact another country, such (Thierry Mugle, "home country", France + China). We then assess the model's performance on questions based on the edited knowledge, including "The official currency of the home country of Thierry Mugle is" and "The capital city of the home country of Thierry Mugle is". While the unedited model could correctly answer these questions, we observe that the edited model would provide the answer "China" for subsequent hop reasoning. We find that the mover head in the original multi-hop reasoning circuit initially extracts the second-hop answer but, after editing, extracts "China", demonstrating that the edited information dominantly saturates and influences the circuit. Furthermore, we observe an in- triguing phenomenon: even in the original model's multi-hop reasoning settings, would directly provide the answer if we remove the context of the first-hop texts (Details in Appendix C. This further confirms the findings that the model relies on relational and subject-related information, regardless of grammatical adherence.
6 Knowledge Circuits Facilitate Interpreting Language Model Behaviors
In this Section, our aim to validate whether the identified knowledge circuits are actually utilized by the model when it employs knowledge. To address this, as shown in Figure 5, we investigate three phenomena: hallucination, in-context learning, and reverse relations (Details in Appendix C.3).
Factual Hallucination. If the knowledge is stored and expressed by the circuit we discovered, we aim to discover what happened when the model gave us the incorrect answer. We focus on factual hallucinations, which occur when the model provides an incorrect target entity for given subject s and relation r. In our experiments (Figure and Appendix C.2), we observe that the model fails to move the correct knowledge to the final token in the earlier layers. This failure is evident as the

---

oh carving wwwaleddiaeeeeooooooooooo od
Residual Output
of Contertinig
athartion head
Original circuit
Troudday
L14H11
L12H18
L17H2
Embed
corpardia
Malays
warmer
Kuala
lighter
Malaysia
cleaner
Malayaian
Singapore
sharper
roayda
Brune
The
coaler
Indonesia
heavier
Jakarta
shorter
040
Cheong
happler
046
RM
fewer
L18HO
Attention Pattien
Output logits
L12H13
Attention Pattern
Output logits
(a) hallucination Case
in-context learning Case
Figure 5: Left: fact hallucination case "The official currency of Malaysia is called", we observe that, at layer 15, the Mover Head selects incorrect information. Right: In-context learning case, we notice that some new heads focusing on the demonstration appear in the knowledge circuit.
circuit lacks an effective mover head or the mover head selects incorrect information. For instance, in the prompt "The official currency of Malaysia is called", both the correct answer "Ringgir" and the incorrect one "Malaysian" " are accumulated before layer 15. However, at layer 16, the mover head L15H10 extracts the erroneous information. Despite rank drop of the true one in layers 20-22, this insufficient to correct the previous mistake.
In-Context Learning. Despite storing a vast amount of knowledge, language model may still provide incorrect answers. However, with demonstrations or examples (based on RAG [49]), it can quickly generate correct responses. To this end, we focus on the scenario where the model initially provides an incorrect answer but can then produce the correct response upon receiving the appropriate demonstration. We consider the original knowledge circuit and introduce new knowledge circuit based on the demonstration. Our analysis reveals that, compared to the zero-shot knowledge circuit, several new attention heads appear the computation graph when the demonstration is incorporated. We show the behavior of these attention heads in Figure 5. We can see these heads mainly focus on the demonstration's context: "The comparative form of small is smaller" and works as the Induction Head [50] that look back over the sequence for previous instances of the current token and find the token that came after it last time. To better view the function these heads, we conduct experiments by ablating the newly appeared attention head in the ICL circuit in Table 2. We find that compared to the randomly selected attention head by ablating this attention, the probability drops significantly in the prediction, demonstrating the importance of these identified attention heads. These aligned with previous work where Todd et al. [51] have identified a concept known as the Function Vector, which represents the average of some key attention heads and provides the task learning ability.
7 Related Work
Knowledge Mechanism of Transformers. How the language model stores and utilizes knowledge is an ongoing research topic. Previous works find that the MLP in Transformers works as key- value memory and stores enormous knowledge [12, 15, 14, 18]. As the relation between entities, Hernandez et al. [42] observes that facts can be decoded linearly from the enriched residual stream of the subject by mapping the subject entity to the object entity. Instead of viewing the knowledge storage in isolation, Geva et al. [13], Lv et al. [31], Yu and Ananiadou [16] find the knowledge is accumulated during the layers. Regarding knowledge analysis, Bayazit al. [52] also attempts to discover critical knowledge in language models. However, they only consider several layers in the

---

Table Performance change via ablating the newly appeared attention heads in the ICL circuit and random heads.
Knowledge
Origin Model
Ablating extra head
Ablating random head
Linguistic
adj_ comparative
62.24
32.55
58.18
word_see gord_sentiment
89.02
55.50
88.61
Commonsense
substance phaase
78.74
52.85
71.24
Bias
occupation gender
86.97
59.54
86.54
Factual
person_occupation
35.17
23.27
31.60
model and use the pruning method, which may overlook the connections between components. More related works can be found in Appendix E.
Manipulate Language Models. Recently, many works aim to manipulate the language models to make the model aligned with world knowledge or social value norms, such as knowledge editing [20, 24), machine unlearning [53, 54] and detoxification [55, 56]. Most of these works are elicited by previous knowledge mechanism findings such as knowledge neuron [57]. They modify the MLP in the LLM [18, 12] to change the model's behavior based on specific factual knowledge. However, recent works [58, 59] demonstrate the pivotal role of the attention part in knowledge representation. Hase et al. [45] also observe that the performance of editing within layer may not reliably pinpoint the location of the fact. In this paper, we try to manipulate specific knowledge of language models via knowledge circuits, including both MLP and attention components across different layers.
Conclusion
In this paper, we present new perspective on knowledge storage based on circuit theory and conduct a preliminary analysis to demonstrate its effectiveness. We found that knowledge circuits in the model are not only responsible for expressing knowledge but can also guide behavior in different settings. We hope these findings can advance our understanding the knowledge mechanisms of language models and provide insights for better designing and editing language models, enhancing knowledge, and improving reasoning to enhance factuality and alleviate hallucinations.
Limitations and Broader Impacts
In this work, we employ the causal mediation method to automatically construct circuits tailored to specific knowledge domains. However, this circuit discovery-base patching approach is time- intensive. Contemporary research efforts have introduced more efficient methodologies for modeling information flow [60-62]. Additionally, alternative techniques for discovering circuits through mask training [63, 64) and Sparse Auto-Encoders [65, 36] have been proposed, highlighting diverse facets of circuit behavior within large language models (LLMs). We posit that the field of knowledge circuit discovery holds significant potential for advancement. Furthermore, recent studies [66] have developed 'circuit breakers' to manage representations associated with potentially harmful outputs. We hope that our approach can contribute to ensuring the safety and privacy of information, thereby fostering the development of trustworthy AI.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities 2226-0023-0018)) Zhejiang Provincial Natural Science Foundation of China (No. LGG22F03001)) 1), Yongjiang Talent Introduction Programme (2021A-156-6
CIPSC-SMP-Zhip Large Model Cross-Disciplinary Fund, Ningbo Science and Technology Special Projects under Grant No. 2023Z212, Information Technology Center and State Key Lab of CAD&CG, Zhejiang University, NUS-NCS Joint Laboratory (A-00085442000-00) and the Ministry of Education, Singapore, under the Academic Research Fund Tier (FY2023) (Grant 000000996000-00), We gratefully acknowledge the support of Zhejiang University Education Foundation Qizhen Scholar Foundation.
10

---

References
[1] Francis bacon. https://iep.... wtwpllepppmmmmemmmmmeeeemeee. wtw.uuwm..eeee......
[2] Francis Bacon. The advancement of learning [1605]. In Primer of preeleectual pages
172-192. Harvard University Press, 1949.
[3] OpenAl and the Co-authors. Gpt-4 technical report, 2024.
[4] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, 1at/2233.112233 2023, doi: org/10. wrg/...48550/ 48554//555i...... http:////.xvv.233.... 18223.
10.48550/ARXIV.23303 18223. URL https://doi.
[5] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong. Zhangyin Feng. Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023.
[6] Cunxiang Wang. Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang. Xing Xie, Zheng Zhang, and Yue Zhang. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity, 2023.
[7] Xiang Chen, Chenxi Wang. Yida Xue, Ningyu Zhang, Xiaoyan Yang. Qiang Li, Yue Shen, Lei Liang. Jinjie Gu and Huajun Chen. Unified hallucination detection for multimodal large language models. CoRR, abs/2402.03190,, 2024. doi: 10.45500AAXX.22222..00199 URL https://doi.. http://o..rrrgg///// org/10. 48550 arriv. 248..0332222
[8] Helena Bonaldi, Yi-Ling Chung, Gavin Abercrombie, and Marco Guerini. Nlp for counterspeech against hate: survey and how-to guide, 2024.
[9]
Lichao Sun, Yue Huang, Haoran Wang. Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu. Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang. Zhikun Zhang, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang. John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying. Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Yc, Yinzhi Cao, and Yue Zhao Trustllm: Trustworthiness in large language models. CoRR. abs/2401 105661,, 2024. doi: 10.48550/ARXIV.401..1 055661.. URL https://doi. http://do..rrg////////// 48550/arXiv. 2401 2401.05561.
[10] Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. CoRR, Sttps/2300000455 2023. doi: 10.48550/ARXiV, 2309.07044 URL https: //doi. 48550//...5///////// .309.000455
[11] Aiqi Jiang and Arkaitz Zubiaga. Cross-lingual offensive language detection: A systematic review of datasets, transfer approaches and challenges, 2024.
[12] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguis- ties (Volume 1: Long Papers), pages 4493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 18.18653/v 1/255/////02222aallooog
[13] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216-12235, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ 1/2023.emmll-main.

---

[14] Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg, Transformer feed-forwer beed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zoriisaa Kozareva, and Yue Zhang. editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 18653// htte://w112222..lp-mm. main..3.
[15] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer nred-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen- tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.1 10155//112222..mlppmmoo m464..44....
[16] Zeping Yu and Sophia Ananiadou. Locating factual knowledge in large language models: Exploring the residual stream and analyzing subvalues vocabulary space, 2024.
[17] Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms, 2024.
[18] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances Neural Information Processing Systems, 36, 2022.
[19] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic, 2024.
[20] Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10222-10240, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v11 21663/vv12002 hwtp:/222..mmmmamnnn.mmm 6322
[21] Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the Ripple Effects of Knowledge Editing in Language Models. Transactions of the Association for Computational Linguistics, 12:283-296, 04 2024. ISSN 2307-387X. doi: 10.1162/tacl a 00644. URL https:// org/10. 2112/taccc_.. aa_taaaa____________
[22] Belinda Hopkins. Restorative theory in practice: Insights into what works and why. Jessica Kingsley Publishers, 2015.
[23] Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have to do with knowledge? In The Twelfth International Conference Learning Representations, 2024. URL
net/
[24] Ningyu Zhang. Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao. Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint a arXiv:2401.01286 2024.
[25] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transsfrmer-eiieee pub/t02 1et20001fffaama h/frammwwwr/iiddee..tntt. html..
[26] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference Learning Representations, 2023. URL https:
/ ///ppnreview.neee www.tn/oorrrndddnssSSSeeeaae
[27] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in trans- former language models. In The Twelfth International Conference on Learning Representations, 2024. URL /ttp:::///ppeeeeeeeeeeee met//ForumiddffooAYveess.
[28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99 2019.
12

---

[29] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An small language model, 2024.
[30] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.239 //ddistil.000022..0000 https://iistill.pp//00/ciciiiii putpi0000uerreiis///ommiii..
[31] Ang Lv, Kaiyi Zhang, Yuhan Chen, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in Transformer-base language models, 2024.
[32] Arthur Conmy, Augustine Mavor-Parkes Aengus Lynch, Stefan Heimersheim, and Adria Carriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 16318-16352. Curran Asso- ciates, Inc., 2023. URL
www.:ss..ccepeppceei..eeeoss ptwppprrrrrpppppppappeepppeeeeee ffllss/aperr/2233/ pdf.
file/34er
[33] Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety review, 2024.
[34] Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl, pages 373-392. 2022.
[35] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 18:12388-1200 2020.
[36] Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt, 2024.
[37] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600, 2023.
[38] Nicholas Goldowsky-Doe Dill. Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023.
[39] Shahar Katz and Yonatan Belinkov. VISIT: Visualizing and interpreting the semantic informa- tion flow of transformers. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14094-14113, Sin- gapore, December 2023. Association for Computational Linguistics. doi: 10.1 125.8//221222... findings-emnlp.999. URL
or wwg./r///......../ findings-emen eeppss.enlp...... 9939.
[40] nostalgebraist. interpreting GPT: the logit lens. 2020. URL https:
com/
pssts/AcckBBsiilddašvvvv/f hwt.://rrrrnnneeeeeiinn-eeeeeeee gpt teapriiingggt-tte--oli--------- the-logit-
[41] Neel Nanda and Joseph Bloom. Transformerlens. https: wwt.gitt......eeeeeeeeeeeeeeeeee www.b/cooo/neeenadaaiiooo io/ TransformerLens., 2022.
[42] Evan Hernandez, Amab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob An- dreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In Proceedings of the 2024 International Conference on Learning Representations, 2024.
[43] Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. A primer on the inner workings of transformer-based language models, 2024.
[44] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. 2024.
[45] Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality- -based localization vs. knowledge editing in language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URI. https:///pommeeeieeeeee/ffoooore www..e/ooorrmiiddiiadltlttttt
net
13

---

[46] Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15702, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.53v we20233emm/pmmmaii....
[47] Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning?, 2024.
[48] Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. Investigating multi-hop factual shortcuts in knowledge editing of large language models, 2024.
[49] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. CoRR, abs/2312.10999 2023. doi: (0.48550/RRXI.23122.000 URL https:://oll org/10. 1/8550/arXiv 2312. 18997.
[50]
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https//transformeecccccii potp:/022/iinccoeee///eiinn.ana heads/index.html.
[51] Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. LLMs represent contextual tasks compact function vectors. In The Twelfth International Conference on Learning Representations, 2024. URL hwwph:ttt//pppttt /tpteree/pperrevieww. wtwp://www.ee/oormmmmiie.... AwyxtyMwaG.
[52] Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. Discovering knowledge-critee subnetworks pretrained language models, 2023.
[53] Nianwen Si, Hao Zhang. Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for Ilms: Tasks, methods, and challenges, 2023.
[54] Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for LLMSs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041-12055 Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/11/2023 1/2653//1/223..emmmmmmaaa hwn.::/a..mmlmmmaa..555 URL https: attp://acaaannnn aclanthology....mm org/2023. emnlp-main. 738.
[55] Xinshuo Hu, Dongfang Li, Baotian Hu, Zihao Zheng, Zhenyu Liu, and Min Zhang. Sepa- rate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty- Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium Ed- ucational Advances Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 18252- 18260. AAAI Press, 2024. doi: 10. 12.16099AAAAL.338....2009 URL https://dof orgoorg////// (069///aai.. w38.16........ 29784.
[56] Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing, 2024.
[57] Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4493-8502. Association for Computational Linguistics, 2022.
14

---

[58] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. In Thirty-seventh Con- net//
ference Neural Information Processing Systems, 2023. URL. Yorum?id==LLLypnne.y
[59] Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer- based language models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interprering Neural Networks for NLP, pages -56,, Singapore, December 2023. Association for Computational Linguisties. doi: 10. 0018883311122222 blackboxnlp- 1.26.
[60] Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. arXiv preprint arXiv:2403.0088 2024.
[61] Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In NeurIPS Workshop on Attributing Model Behavior at Scale, 2023. URL https://openrevits
[62] Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URL
[63] Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. arXiv preprint 24xx:24466.16778, 2024.
[64] Lei Yu, Jingcheng Niu, Zining Zhu, and Gerald Penn. Functional faithfulness in the wild: Circuit discovery with differentiable computation graph pruning. arXiv preprint arXiv:2407.0377 2024.
[65] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodde Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Hemighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transformer- htmml.
[66] Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with short circuiting. arXiv preprint urXiv:2406.0m 04310, 2024.
[67] Joshua Ainslie, James Lee-Thorp Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1895-4901. Singapore, December 2023. Association for Computational Linguistics. doi: 10. 00.1653/v1/2023. amnlp-main.295 URL. htttss://aclantteee
org/ 2023. omnlp-main..20 298.
[68] Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation, 2024.
[69] Yung-Sung Chuang. Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: / ///penreview.noe http:iiwww..e/.fooom/idttttl///
[70] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling carly exit inference and self-speculative decoding. arXiv preprint arXiv:2404.1677 16710, 2024.
15

---

[71] Beren and Sid Black. The singular value decompositions of transformer weight matri- ces are highly interpretable. the
https///wwono.oooooos////ss////e 2022.
[72] Junlin Zhang. Parametric reflection of the world: Why can gpt generate intelligence through next token prediction. https:// zhihu. wwwhh.com/p/
2023.
[73] Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on is b" fail to learn "b is a". The Twelfth International Conferencee on Learning Representations, 2024, URL https://openrev/ http::///ooemeereeeeeee/rrrrrnrn (o-
[74] Peng Wang. Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, et al. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269, 2023.
[75] Maximilian Li, Xander Davies, and Max Nadeau. Circuit breaking: Removing model behaviors with targeted ablation. Workshop on Challenges in Deployable Generative Al at International Conference Machine Learning, 2023.
[76] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5484-5495, Association for Computational Linguistics, 2021. doi:
1tt.://1112221..ELPPP emnlp-main. 446.
URL https: //doi. org/10. 18653/v1/2221..
[77] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 30-45 45. Association for Computational Linguistics, 2022. doi: 10. 00.11853//1/20222.202 22T.:/20002.MMPL-MAAMM URL https: //doi.org/10. 18653/v1 20063//w/2222..em1pppi...... emnlp-main.
[78] Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Journey to the center of the knowledge neurons: Discoveries of banguage-independer knowledge neurons and degenerate knowledge neurons. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 17817-17825 AAAI Press, 2024. doi: 10. 10.1609/AAAL TTW.:AAAA.VVIII1.... V3816. URL https::///// /tt../oiiioooo org/ 10. 1609/aaai. v38i 6. 29735.
[79] Yuheng Chen, Pengfei Cao, Yubo Chen, Yining Wang. Shengping Liu, Kang Liu, and Jun Zhao. The da vinci code of large pre-trained language models: Deciphering degenerate knowledge neurons. CoRR, 137331 2024. doi: 10.45550AARXI22222..11777 URL https://doi...
org/10.
2402. 13731.
[80] Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview net/pdf? www..e//dd////eeeecccHHHYY d-=MkbcAHTygyS.
[81] Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mecha- nistically explains long-context factuality, 2024.
[82] Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. CoRR, abs/2402. 181544 2024. doi: 10. 10.48550/ARXIV224022 18154. URL https:///of org/ 48550/arXiv. http://rXiv..2400 2402. 18154.
[83] Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. CoRR, abx/2310.15213. 2023. doi: 10.48550/ ARXIV.2310.1221 URL https: //to///oioo// /10/ 48550/ ariei./rrvvv.... 2310. 15213.
16

---

[84] Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How think step-by-step: A mechanistic understanding of chain-of-thouge reasoning. CoRR. abs/2402. 18..1,2.1831, 18312, 2024. doi: 1048550AARXW.22222.83322 URL https://doi.org: 48550/arXiv. http:/ariiv...2..111 18312.
[85] nostalgebraist. interpreting gpt: the logit lens. https://www. www.://wwwnn.com/opee..ssss
http://ldaasorrr//nteerrrretteeo seaeingggttttheeeeeeee thhe-logit-eeme 2020.
[86] Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint-satisfactio lens on factual errors of language models. In The Twelfth International Conference on Learning Representations, 2024. URL https::t///ooooo /oppnr/vie/ereeee...eeeoe net/ (or/Foorrmmi--g-FFTFFRR.
[87] Fahim Dalvi, Hassan Sajjad, and Nadir Durrani. Neurox library for neuron analysis of deep NLP models. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2023, Toronto, Canada, July 10-12, 2023, pages 234, Association for Computational Linguistics, 2023.
[88] Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders. Transformer debugger. https://githhnn
2024.
[89] Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes:
A unifying framework for inspecting hidden representations of language models, 2024.
[90] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Statfeld-DDodde Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. CoRR, abs/2209. 11022299..000 10652, 2022. doi: 10.4855/ARXXIV22000066 URL https://do1 org/10 10. 48550/arXiv www.:a.wvvv......... 10652.
17

---

Appendix
A Implementation Details
Plyper-parameter The primary hyperparameter for constructing circuit is the threshold used to detect performance drops. Setting т too high may result in an incomplete circuit, while setting too low can introduce numerous unnecessary nodes. In our experiment, we test values from the set {0.02, 0.01, 0.005 to determine the appropriate circuit size for different types of knowledge.
Implementation. We utilize ACDC² construct circuits that encode specific knowledge repre- sentations. Additionally, we re-implement the code to compute the relevant dataset and impact metrics for the knowledge used. Since TinLLLAMA incorporates the Grouped Query Attention Mechanism [67], we interleave and repeat the key and value pairs to analyze the specific behavior of each attention head. We use the NVIDIA-A800 (40GB) conduct our experiments. It took about 1-2 days to compute the circuit for the knowledge type in GPT2-medium.
Dataset Details. All the data used in our paper is sourced from Hemnnnde al. [42], with the detailed information provided in Table 6. In the original setting, they use the data for few-shot settings, but in our experiments, we consider zero-shot knowledge storage, so here we sam- ple the data using the Hi re1 0 to detect whether the model understands knowledge for the given prompt based on the and We sample the test set in a 1:1 ratio with the validation set to ensure balanced evaluation.
Category
# Rel.
# Examples
# GPT-2 Corr.
Factual
26
9.696
4,721
Commonsense Linguistic
8
374
240
6
806
483
Bías
7
213
149
Table 3: Information about the dataset. Table is bor- rowed from Hernandez et al. [42]
B
More Experiment Results
B.1 Rank Change Across Layers
To gain a clearer understanding of the knowledge aggregation phenomenon, we compute the rank of the target entity rank, within the vocabulary space [V] at the output of each layer. As depicted in Figure 7, we observe that the model initially elevates the target entity to the top ranks of the vocabulary. Once the entity reaches the top of the vocabulary, subsequent layers continue to enhance its probability mass. These findings corroborate previous work by [68-70], who note the substantial difference in logit entropy between layers, contributing to the model's improved prediction for the target entity. our work, we aim to delve deeper into how the model, as well as specific components within it, give rise to these behaviors.
B.2 Special Components in Knowledge Circuit
When zooming into the discovered circuit, we can find several kinds of special attention heads, or MLPs, in the model that play pivotal role in the final prediction, similar to what previous research has indicated [31]. Apart from mover head and relation head, there is another kind of head named Mix Head [17, 43] which would focus both the relation token and the subject tokens. In experiments, we found these heads usually work similarly to the mover head. In particular, mover heads contribute more to the subject-specific information, as ablating them would increase other relation-related tokens' probability. Moreover, if we ablate the relation head, we can find the model tends generate some meaningless tokens instead of the relation information. From Figure 3 we can find ablating the mover head would increase the probability of "Italian", " "English" and "Spanish", which are not subject-related. While ablating the relation head would lead the increase of some meaningless words "a", and "that", which are not relation-related
Ly et al. [31] posits that these mover heads responsible for extracting the "argument" from the context and passing it for further processing, such as function application. Consequently, Ly et al. [31], Merullo et al. [19] suggests that the MLP within the language model performs function
wtw.://w...comtttturuoooooo "Checkpoin: https: http:://wuggingaeeee.///iyyyiiip
1B- -ttpr//mmm-------i-eieeeitt----- 1431k-3T
18

---

Top 10 Token Output Probability
French
French
English
0.7
Englishh
0.16
the
Spanish
0.30
Spanish
the
0.12
0.5
0.25
spoken
Italian
not
0.10
0.4
Latin
Portuguese
0.15
F
English
0.06
0.2
0.10
Italian
Catalan
0.04
0.1
Franc
now
German
that
original output
(b) ablating mover head
(c) ablating relation head
Figure 6: The output of the model. Ablating the head would increase the probability of "Italian", "English" and "Spanish", which are not subject-related. While ablating the relation head would lead the increase of some meaningless words "a". "that", which
relation-related.
peided 290
MAR LAT
1,07
L007
Figure 7: The average rank of the target entity for the D in the vocabulary when mapping the output of each layer the model to the embedding space. We can find that the in GPT2-Medium and GPrg-Large, the model would get the knowledge at middle-to-later layers. While for TinALLAMA, the layer may be more later.
application that transforms the subject into object, with the subject's probability ranking higher than the object's. s. However, these findings are limited to specific cases, as the studies only examine two related tasks, such capital city identification or color objection. Our analysis suggests that these conclusions may not be universally applicable to all knowledge domains and require further investigation. When we examine the ranks of subject and object entities, we rarely observe an overwhelming superiority at the last token position of the subject token and usually, the probability of the subject token the last position is uniformly low. In our experiment, we view the model as collaboration between different nodes in the identified circuit. They perform different kinds of
19

---

(b) Seme Companen's outpot
Attertion Pattern
Output logits
Attenttoon Pattern
Output logite
MLPI
Francc
English
C15
Natoip
Qunina
OuI
Franio
Frinich
Franco
ammer
L0-amb
cank podsubiltty unity rocihtl
Frerchh
-000
1117-19
5
nothor Muglo
Firbee
franc
simplified ineiddddde Knowled Cirouit
Figure 8: A simplified knowledge circuit found TinyLLAMA for the knowledge "The mother longue of Thierry Mugler is French".
attention heads, like mover head and relation head. We list some heads that are responsible for the storage of different kinds of knowledge and relations Table
Component Reuse Phenomenon. Merullo al. [27] have identified shared circuits for the IOI task and the Colored Objects task. We also observe this phenomenon in the factual recall task. As depicted in Table 4, we can observe that for related relations such as "city in country". "name birth follb_place_ and
their circuits include both 1,21112, which stores and maps country-related information. Additionally, we found that some relation heads are activated by different relations. For instance, in our experiments, the head 'L.7H14" appears in the circuits of both "official_ langgugge and "official cutrrnccc_rrer""...a We speculate that these reused heads, rather than task-specific heads, can be considered topic heads, proposed by [71, 60]. We believe that further investigation into this distinction is warranted future research.
Table 4: Special component behaviour in circuits as task-specific head. Find more results in Appendix
Model
Type
Fact
Critical Component in Circuit
Linguistic
Antonym
L17H2, LI8HI, L13H12, L13H8
Factual
city country
L21H12, L16H2 L19H15, L14H4, L13H3 L16H6, T.21
GPT2-Medium
Commonsense
work location
Bias
name country
Linguistic
Antonym
L25H5, L24H16, L19H13, LI8H8 L30H6, L25HI3 LI8H13, L28H18, L30H5 L21H19, 1,29H2
Factual
company hq work location name country
GPT2-Large
Commonsense
Bias
Linguistic
Verb past tense Landmark country Fruit Inside Color
LI7HO. MLP20
Factual
LISHI1, LI7H19, MLP18
TinyLLAMA
Commonsense
L18H25, MLP18
Bias
name country
LISHII, MLP17
B.3 A Case on TinYLLAMA
In this part, we demonstrate one circuit we found in the TAMALAMMA in Figure Actually, in TiMA the attention heads bearing specific behaviors in the later layers is usually less than
20

---

Figure 9: The knowledge circuit from the "The official language of France is French" in GPT2- Medium.
GPT2. We can also view some mover heads and relation heads in the circuit that would generate the target token the output, such L15H3 and L17H19.
C More Circuit Utilization Analysis
While previous work on knowledge storage suggests that knowledge may be localized in specific areas of the model, it is essential ascertain whether the model actively employs this knowledge when encountering related contexts or if it relies on shortcuts.
C.1 Multi-hop Factual Knowledge Editing
We consider scenarios where the model makes correct prediction for all the multi hop questions and single-hop questions. We found a circuit reuse phenomenon in the one-hop and multi-hop knowledge circuits. Here, we first compute the proportion of the nodes N Nsingle in the single-hop circuit Csingle that appears in N Nmuttiple the set of nodes the multi-hop circuit Cmulttiple
Nmuuttiple n siiigee
Hitnode
[Neingle]
Actually, there are two ways for the model to conduct multi-hop reasoning. As shown in the figure, the model can also answer the question by combining the two hop relations together (in the given case, combine "hometown" and "language" "mother tongue") If the model is capa- ble of combining two-hop relations in a more integrated or semantic way, such as inferring that the "mother tongue" is the language of one's hometown, this suggests more complex reasoning process that goes beyond the simple overlap of nodes and edges. To capture this kind of reason- ing, we assess the model's s ability to integrate information from different hops. Rantegrated as the set of integrated relations (new paths created by combining information from different hops)
We observe overlap of these circuit nodes,
Table 5: Hit for different hop
indicating that the language model utilizes large portion of the nodes in the original single hop's is circuit, especially the mover head. From the overlap analysis, it seems the model utilizes single-hop information to conduct reasoning. We discover an intriguing phenomenon in GPT-
First-hop
Second-hop 70.27
Integrate 71.42
node
83.33
edge
63.20
45.27
49.42
2 and LinALLAMA: the models can correctly answer first-hop knowledge and perform multi-hop reasoning based on it. However, interestingly, when we delete the first-hop knowledge while retaining only the second-hop relation and the first-hop subject, the models can still correctly answer the multi-hop question. Figure illustrates specific case in our findings. It further enhances our previous findings that the model actually conducts the factual recall with the relation head and the gathered information about the subject. Moreover, we found this phenomenon is hugely alleviated by the aligned model, which requires further investigation in the future.
21

---

factual knowledge
The home country of the Thierry Mulger is
France
Multi-hop knowledge
The official currency of the home country of the Thierry Mulger is the
Franc
Corrupted knowledge
The official currency of Thierry Mulger is the
Franc
Without first-hop
Figure 10: a specific case in Multi-hop reasoning. When we removed the context of the first hop question, we found the model also directly gave us the answer. The phenomenon appears both GPT2 and TiMYLLAMA.
C.2 Hallucination
The results are presented in Figure 5. We observe an interesting phenomenon: the correct answer and the wrong answer are both accumulated the previous layer, but at some specific layers, the wrong answer is selected as the answer to extract. We hypothesize that there may be a circuit competition here proposed by [72] and we detect the behavior of the specific component between them.
C.3 Reverse Relation
Reverse Curse [73] is important issue in language models when models successfully give the correct answer o for (s, r), but with a reverse relation ř and o, the models fail to give us the correct subject In this part, relation when they successfully store the knowledge. We first sample facts where the language model successfully predicts the given knowledge and the reversed fact. Then, we compute the overlap between these two circuits, Ca and C, under node levels based on equation 4. We select the "superhero perrro"___oo" relation to see the difference between these two circuits and test the node overlap of the two circuits in the model. We notice that the overlap between the two circuits is about 70%, indicating the language model may store the related information in the same place. We also found the activated mover heads for the two relationships to be identical.
endeavor investigate how the language model manages the reverse
D Edit Experiments
D.1 Method Implementation
ROME As proposed by Meng al. [18], ROME views knowledge editing minimal optimization problem. ROME regards the MLP module a simple key-value store. Specifically, the key represents subject and the value encapsulates knowledge about that subject, the MLP can reestablish the association by retrieving the corresponding value for the key. To add a new key-value pair. ROME applics rank-one modification the MLP's weights, effectively "writing in" the information directly. This method enables more direct and precise modification of the model's 's knowledge. The ROME method for model editing conducted based the EasyEdit(74) framework, utilizing the default parameters provided by EasyEdit. The experiments are performed on an A800 80G GPU, with approximately 8GB of memory consumption.
For Fine- Fine-Tuning (FT-M), follow Zhang et al. [24]. It trains the MLP layer using the cross- entropy loss on the target answer while masking the original text. This approach aligns more closely with the traditional fine-tuning object. The FT-M method is conducted using the EdsyEdit [74] framework, with the default parameters provided by EdsyEdit. The experiments are also performed on an A800 80G GPU, with a memory consumption of approximately 10GB.
FT-M
22

---

Table Number of examples per relation and the count of accurate predictions by different LMs. This table borrowed from Hernandez et al. [42] and here we sampled with different ways.
Correct Hit@10
Category
Relation
#
GPT2-Medium
GPT2-large
TinyLLaMA
person mother
994
83
144
361
person father
991
359
385
474
person sport position
952
400
489
596
landmark continent
947
835
543
694
person native language
919
220
260
landmark in country person occupation company hq
836
600
489
709
821
57
76
149
674
308
312
470
product by company person plays instrument
522
422
432
460
513
510
505
498
star constellation name
362
266
148
297
plays pro sport
318
317
316
315
company CEO
298
20
52
125
factual
superhero person
100
35
50
superhero archnemesis
96
6
23
person university pokemon evolution country currency food from country city in country country capital city country language country largest city person lead singer of band
91
14
37
35
44
11
13
16
30
25
25
30
30
25
29
27
20
23
27
24
24
24
24
24
24
24
24
24
24
21
16
21
president birth year president election year
19
11
12
19
17
18
object superclass
64
word sentiment
60
14
9
34
task done by tool
52
44
45
45
substance phase of matter
50
12
16
48
commonsense
work location
28
24
27
fruit inside color
36
36
35
36
task person type
32
28
27
26
fruit outside color
30
16
20
21
word first letter
241
235
241
word last letter
241
135
73
114
adjective antonym
100
80
81
84
linguistic
adjective superlative
80
24
19
63
verb past tense adjective comparative occupation age univ degree gender birthplace
76
15
76
15
63
45
18
20
18
38
14
35
38
31
29
30
31
bias
name religion
31
31
characteristic gender name gender occupation gender
30
26
30
30
19
19
19
19
19
19
19
19
D.2 Edit Cases on FT-M and ROME
When circuit is established for particular piece of knowledge, we can manipulate the model's s computation by targeting critical points within the circuit. Li et al. [75] ablates small number of important causal pathways by masking the edges in the circuit and making the model less toxic and safer, which proves the effectiveness of the circuit. present the changes in the ranking of the predicted probabilities for the target new token when editing layer 6, 12, and 18 of the GPT-2 medium model using FT-M and ROME methods. When applying FT-M for model editing, it is evident that the rank of the target new token's probability sharply
illustrated in figure and figure 13, we
23

---

Lager W
ESI WItFT
EalLav
EitLaae FT
Figure 11: The knowledge circuit obtained from the edited model for the case "Platform Controller Hub was created by" with the target entity "Intel" shows that when editing the model using different layers, the fine-tuned settings allow the edited MLP to directly provide the edited information.
Clylss Coultly
Sa Np-CCrclllate
Figure 12: FT-M Rank Change Across Different Layers
declines at the corresponding edited layer, resulting in vertical line in the figure. This indicates that FT-M directly embeds the editing information into the model's information flow. Conversely, when using the ROME method for editing, this effect is mitigated. The predicted probability of the target new token reaches its peak only a few layers after the edited layer. This observation is consistent with our previous analysis in Section
E More related Work and Discussion
E. More Related Work
Knowledge MLP Geva al. [76) claim that MI.P serves as key-value memories for knowledge in LMs. Geva al. [77] further propose the knowledge neuron theory, suggesting that the key and

---

Chessoaayy
Speieesfess
E
20
Figure 13: ROME Rank Change Across Different Layers
value vectors in MLPs encode factual knowledge. Based on the above findings, Chen al. [78] observe that multiple distinct sets of KNs can store identical facts. Chen et al. [79] explore the structural and functional among neurons by neurological topology clustering method. Meng et al. [18] and Meng et al. [80] confirm that MLP modules do store factual knowledge and pioneering use of knowledge editing methods modify outdated knowledge stored in language models. Anthropic recently introduces scaling monosemanticity" which extracts highly abstract features that respond to and behaviorally cause abstract behaviors.
Liet al. [58] reveal that some attention heads are capable of truthful answers. Wu et al. [81] investigate 4 model families, 6 model scales, and 3 types of finetuning, and find retrieval heads, which are responsible for retrieving relevant information from long context. Jin et al. [82] suggest that memory heads can retrieve knowledge from internal memory, while context heads can recall knowledge from external context. Todd et [83] use causal mediation analysis on diverse range of in-context-leaming and find some attention heads, dubbed function vectors, which trigger the ability of in-context-leaming
Knowledge in Attention Heads
Knowledge in Hybrid Components Recent works emphasize the importance of connections of
components among language models for knowledge representation and utilization. Geva et al. [13] describe factual recall by the following three steps: (1) subject enrichment in MLP sublayers, akin ROME [18], (2) propagation of relations to the END token, and (3) selective extraction of attributes by attention heads later layers. Lv al. [31] apply projection and intervention explore mechanisms in factual recalls tasks and conclude that task-specific attention head may move the topic entity to the final position of the residual stream, while MI.P conducts relation function.
Circuit Cireuit discovery also plays significant role in analyzing the internal mechanisms of the entire model [25]. Specifically, a circuit, comprising components such as MLP and attention, subgraph of the computation graph. Commy et al. [32] design automated circuit discovery approach that implements the specified behavior. Wang et al. [26] explain the circuits for the Indirect Object Identification (IOI) task. They use causal interventions the flow of information. Instead the above task-specific circuit, Merullo et al. [27] presents evidence a circuit is shared by similar tasks IOI and Color Object (CO) tasks. Dutta et [84] construct circuits using attention heads, and further observe that attention heads are pivotal to chain-of-thougee
discover circuits responsible for
reasoning, i.e., attention heads that move information along ontological relations exclusively appear in the initial half of the layers, while the tokens responsible for writing the answer predominantly appear in the later half of the layers. However, the above-mentional circuit studies either focus solely
hhtto::/ttransforer--coooo http:///wccotttrttooooo/og/////a
25

---

on single component (MLP or attention) or only explore IOI and CO tasks. IOI and CO tasks necessitate the model to search the preceding context for a matching token and then copy it into the next token prediction. Also, despite the success of previous circuit discovery, we can hardly make it into real usage. Hence, in this work, we attempt to analyze a knowledge circuit consisting both MLP and attention components and investigate the effect of current editing methods on the circuit to shed light on the future.
Tools There are also many tools that are designed analyze the LM's behavior, such as Logit Lens [85], Attention len [86], Attribution lens [42] and transformer-lens [41]. NeuroX [87] imple- ments various interpretation methods under unified API and provides insights into how knowledge is structured in representations and discovers the role of neurons in LM. Transformer Debugger [88] an interpretability tool provided by OpenAI, which deploys the GPT-4 and sparse auto-encoder to explain the language neurons and attention head. PatchScope [89] is tool provided by Google that uses new model explain the hidden states in the original model.
E.2 Limitation and Future Discussion
Despite of the attempt to combine the attention head and MLP to view the knowledge storage as a whole, this work operates with a relatively coarse granularity of circuits. For instance, the neurons within an MLP may necessitate finer level of granularity to fully capture their behavior and contributions. Even though we now know these components work together to express the knowledge, why they are activated is still opaque. Our methodology employs the logit lens as means to detect and analyze component information. However, this approach may encounter discrepancies between the middle layers and the output unembedding matrix. Such discrepancies can hinder comprehensive and concrete analysis of the circuit components' behavior in the early layers. This limitation suggests the need for more robust techniques to bridge the gap between intermediate representations and final outputs. Recently, the Attention Lens method [86] has been proposed, which involves training a specific unembedding matrix to map each attention head into the vocabulary space. While this method is promising, it also resource-intensive. Nevertheless, it represents potential starting point for deeper understanding of the knowledge circuits within neural models. Moreover, our research indicates that several mover heads are reused across different types of knowledge or relational contexts. The mechanisms by which these heads are activated and the conditions under which they operate require further exploration and may shed light on why neurons are sometimes "monosemantic" responding single feature, and sometimes "polysemantic" [90] responding to many unrelated features.
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract and Section Introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justifiation::ectionn 8 Limitations and Appendix E.2.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
27

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report the setup throughout the paper as well as in the Subsection 3.3 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [Yes]
Justification: We use publicly available datasets (Appendix A), Code and Data are also provided in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
Sublic//uides/Codeeenii
ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (ht /nips.cc/publl ceooolccggiiie SeeidddeeeCoeebeeieee iotooisssion ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We report the setup throughout the paper as well as in the Subsection 3.3 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: [No]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurIPS Code of Ethics https:://eurtre: https///cb.....tttccss......eee.
Answer:[Yess
We use publicly standard datasets that do not contain information about individual people or offensive context to our knowledge. Ethical considerations are discussed in Section 8.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
In Section 8
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available artifacts and show them in Section 3.3.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper swi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "Yhldh6Wkti", "reviewer": "Reviewer_Njw6", "summary": "This paper presents a circuit-based analysis of factual recall and model editing in language models. The authors make two key claims about knowledge storage and retrieval: 1) Factual recall is compatible with circuit-based frameworks. 2) Traditional model editing methods influence behavior change within the model's \"knowledge circuit.\" The paper compellingly advocates for studying knowledge storage and recall using the circuit framework, presenting preliminary but promising results.", "questions": "Please see the weaknesses section.", "limitations": "The authors have adequately addressed the limitations.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written, and the experiments are presented with clarity. While some of the results are still preliminary, the paper provides a novel perspective on interpreting and understanding the subject.", "weaknesses": "## General Comments\n\nDespite the strengths and contributions highlighted, a significant shortcoming of this paper is the absence of a concrete, successful application of circuits for the tasks mentioned in the introduction and Section 6, such as model editing, hallucination detection, and in-context learning (ICL). This is why I described the review as \"preliminary,\" even though the paper provides concrete experiments and evidence on the existence of a connection between factual knowledge recall and circuits.\n\nDespite its shortcomings, this paper offers an intriguing and promising new approach to addressing the problem by building on Niu et al.'s [21] call to \"...examine the entire decision-making circuit\" when studying how knowledge is stored and recalled in language models (LMs) with supporting evidence and experiments. The authors highlight the limitations of the previous knowledge neuron theory of LMs, noting its inadequacies as a fundamental theory for interpreting the factual recall process in LMs. The community needs a new theoretical framework, and I believe this paper points us in the right direction.\n\n### Knowledge Circuit Information Analysis\n\nThis section of the experiment aims to demonstrate Claim #1: that circuits can adequately explain the process of factual storage and recall. While the experimental results provide some preliminary support for this claim, the evidence is not as compelling as suggested in the introduction. Although the model shows initial success in using circuit-based interpretations for factual recall, some of the results are underwhelming. Notably, the relation \"capital city\" has a test set performance of 0%. Despite the authors' assertion that the standalone performance of the circuit is good, there is no baseline performance for comparison. Perhaps using random circuits as a baseline could provide a clearer benchmark.\n\n### Related Work\n\nThe paper provides an adequate summary of prior work in Section 7 and the introduction; however, there is room for improvement. The introduction initially highlights the issue with previous knowledge editing methods at lines 40-42, noting that \"different types of knowledge are often located in the same areas.\" While this is a valid issue, it is not the most critical one. The relevant work section (lines 336-339) offers a more comprehensive summary of the issues. Therefore, it would be beneficial to expand the introduction by incorporating a condensed summary from lines 336-339 to provide a clearer and more thorough context for the readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Njw6,\n\nThank you very much for your well-considered review! We are very happy that you appreciate the work we put into exploring a novel perspective on interpreting and understanding the subject. Thank you also for the insightful questions, and please let us know if you have any more.\n\n**Response to weakness:**\n\n> **WK1:** **absence of a concrete, successful application of circuits.**\n\nWe agree that our current work still has room for improvement in the proposed application. However, as you mentioned, the main contribution of our work is proving the existence of a connection between circuits and these applications, including model editing, hallucination, and ICL. We are also investigating to address these issues and we believe our work is a good start for future work in this field.\n\n> **WK2: no baseline performance for comparison.**  Thanks for your advice. We have conducted experiments that compare the performance of randomly constructed circuits. The results can be seen as follows:\n\n|  | Random Circuit  | Our discovered Circuit |\n| --- | --- | --- |\n| Adj Antonym | 0 | 1.00 |\n| world first letter | 0 | 0.88 |\n| world last letter | 0 | 0.72 |\n| object superclass | 0 | 0.68 |\n| fruit inside color | 0 | 0.20 |\n| work location | 0 | 0.70 |\n| Capital City | 0 | 1.00 |\n| Landmark country | 0 | 0.60 |\n| Country Language | 0 | 1.00 |\n| Person Native Language | 0 | 0.76 |\n| name religion | 0 | 0.50 |\n| occupation age | 0 | 1.00 |\n| occupation gender | 0 | 0.66 |\n| name birthplace | 0 | 0.57 |\n\nHere, we compute the random circuits by randomly deciding whether the edge should be removed and making sure the graph is connected. **The random circuit is the same size as the circuit we discovered by our method.**\nFrom the table, we can see that the random circuit failed to maintain the model’s performance, which further enhanced the robustness and efficacy of our methods.\n\nMeanwhile, Regarding some odd performances like the \"capital city”, we conduct more analysis here. We found the test data here may not be stored in the language model. In this case, even if we find the circuit for the knowledge, the circuit itself still cannot provide us with the correct answer as the LM itself does not store the knowledge in the model. \n\n> **WK3: related work to be improved.** \n\nThanks for your advice. Due to the limited pages of the submission, we put some of the related work in Appendix E. Your insights are greatly appreciated and will be taken into consideration.  We would polish our introduction to make it more convincing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FBYL1pErhu", "reviewer": "Reviewer_jmbU", "summary": "The authors propose utilizing techniques from mechanistic interpretability to explore the connections between model components involved in factual recall. They systematically ablate connections between model components in a reverse topological order, maintaining a list of the connections that most significantly recover loss on factual recall tasks. Several properties of the resultant \"knowledge circuit\" are measured, including its sufficiency for recovering model performance on factual recall.\n\nThe authors then demonstrate the roles of various components of the knowledge circuit, such as relation heads, mover heads, extract heads, and MLPs. They interpret the roles of these components by projecting their outputs directly into the logit space of the model. This approach is applied to understanding hallucinations and model editing failures.\n\nThe authors demonstrate how knowledge editing methods like ROME switch the role of the subject mover head to instead extract the final answer directly from the subject token rather than moving information about the subject.  The authors also manage to explain failures in multi-hop knowledge editing - the edited knowledge \"dominantly saturates\" the subject mover head preventing it from performing its natural function.  They also show how factual hallucination failures can be attributed to a mistake on the part of the subject mover head.", "questions": "- Why did the authors choose to perform ROME edits at layer 0 of the model, in contrast to the original ROME paper's approach of editing in the middle layers using a causal tracing algorithm, and what impact might this choice have on the results presented in the paper?\n- In Table 1, what explains the inconsistent effects of ablating everything but the circuit on task performance, such as improving accuracy for \"Country Language\" while decreasing it for \"fruit inside color\"?", "limitations": "The authors discuss the limitations of their work, and how it can contribute to ensuring safety and privacy information to promote trustworthy AI.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- As far as I am aware, this paper is the first to provide a mechanistic interpretation of the impact of knowledge editing techniques on a model.  The authors demonstrate how knowledge editing methods like ROME switch the role of the subject mover head to instead extract the final answer directly from the subject token rather than moving information about the subject.  The authors also manage to explain failures in multi-hop knowledge editing - the edited knowledge \"dominantly saturates\" the subject mover head preventing it from performing its natural function.  This work will be quite relevant to other researchers thinking about improving model editing techniques.\n- The authors provide a new perspective on the origins of hallucinations in the model. They attribute hallucination failures to the inability of the subject mover head to move the correct answer to the final token position. This mechanistic understanding offers valuable insights into why hallucinations occur and how they might be mitigated.\n- The authors contextualize their findings amongst other papers that attempt to mechanistically interpret factual recall in models.  The authors also reproduce a claim mentioned in several previous works, involving components in the early layers of a model being more important for knowledge retrieval (Figure 2).", "weaknesses": "- The authors choose to perform ROME [1] edits at layer 0 of the model.  In contrast, the original ROME paper performs edits in the middle of the model.  The precise layer is chosen using a causal tracing algorithm.  This choice is not justified in the paper.  It is unclear what difference this makes in the results provided in this paper.\n- In table 1, it is unclear why ablating everything but the circuit improves performance on some tasks, but decreases others.  For example, restricting the model to just the circuit increases the accuracy of \"Country Language\" but decreases the accuracy of \"fruit inside color\".\n\n[1] Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov. \"Locating and Editing Factual Associations in GPT.\" (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer jmbU,\n\nWe are very happy that you are optimistic about the impact of the work. In this comment, we respond to the question and the suggested improvements.\n\n**Response to weakness & questions:**\n\n> **WK1&Q1: About the editing layer we select in our experiments.**  \n\nThe layer in current editing methods is an important hyper-parameter; currently, there is no conclusion on which layer is the best.  In ROME[1] paper, they edit layer 0 for the FT method and in our experiments, we follow their setting as the case to compare the behaviors of ROME and FT under the same setting.\n\nMoreover, we **also conduct editing experiments under different layers in Appendix, Figure 11 and 12**. We found the same phenomenon across these different layers, which enhances our previous findings. Actually, despite the casual tracing results pointing out that the middle layer stores the knowledge, the identified layer is not always the best layer to edit. In Figure 12, editing ROME using the mid-layer 6 failed to overturn the model's results for the ‘Super-hero’ knowledge. This phenomenon is also found in previous work[2]. Our work provides a new view of what happened when we edited the knowledge.\n\n> **WK2&Q2: inconsistent effects of ablating everything but the circuit.**  \n\nOur experiments test the Hit@10 to measure whether the model knows the knowledge. As we ablate everything but the circuit, we anticipate a little drop-down in the prediction.\nIn our analysis, we found that for \"fruit inside color\",  the original rank of the target token is around 6-8, and after ablating other things, the rank is slightly higher than 10. \nCommonly, the drop-down is acceptable and the most interesting finding is that we can see an increase in several kinds of knowledge, suggesting that the discovered knowledge circuits may encapsulate the relevant knowledge and the model’s performance on these tasks could have been hindered by noise from other components.\n\n---\n\n[1] Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov. \"Locating and Editing Factual Associations in GPT.\" (2023).\n\n[2] Peter Hase, Mohit Bansal, Been Kim, Asma Ghandeharioun. “Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models.” (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o06ohWHU0j", "reviewer": "Reviewer_7LFR", "summary": "This paper focus on the problem of interpreting the knowledge storage mechanism of the large language models. The authors proposed a new perspective, which uses the knowledge circuit to understand how the language model stores and expresses the knowledge. Knowledge circuit is a subgraph of the computation graph of the model. Specifically, the knowledge circuit of a given factual knowledge is defined as the smallest subgraph to recall it \"from the knowledge stored in the model\". They proposed a knowledge circuit construction method, where the nodes are ablated one-by-one to measure the impact of removing it from the computation graph. The resulted knowledge circuit will then be a necessary computation graph for maintaining the original inference result. Experiments are conducted across a variety of domains of factual recall tasks. To intuitively interpret the constructed knowledge circuits, they probes the information flow by projecting the latent representations into the space of token embeddings. Under this interpretation, several conclusions are drawn as follows.\n- language model tends to aggregate knowledge in the earlier to middle layers and further enhances this information in the later layers.\n- the knowledge editing method ROME tends to incorporate edited information primarily at the edited layer.\n- when hallucination occurs, the language model fails to correctly transfer knowledge to the final token in the earlier layers.\n- several new attention heads emerge in the knowledge circuit during in-context learning.", "questions": "- Please refer to the weakness part.\n- According to Table 1, the performance drop of Circuit varies across different datasets. Is there any analysis of this phenomenon?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- It is in general well formatted and easy to follow.\n- Experimental details are well included.", "weaknesses": "- Regarding the novelty of this work, reference [36] appears to be highly relevant, as it involves the construction of a flow graph and the presentation of semantic information flow. However, this reference is not discussed in the related work section.\n- The robustness of the knowledge circuit construction method should be addressed. For instance, the results of the construction method should be unique or equivalent under different traversal orders. If not, the solution may not be well-defined.\n- The interpretation based on the knowledge circuit lacks clarity. Although several conclusions are drawn in the paper, it is unclear if these conclusions align with existing objective observations on related phenomena, such as in-context learning, and whether they could aid in predicting the model's behavior. The effectiveness of an explanation method should be validated by deriving a conclusion or guidance that can be subsequently verified through observation. This aspect is not clearly demonstrated in the paper.\n- It would be beneficial to include formal definitions of mover head, relation head, and mixture head, as these are crucial concepts in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 7LFR,\n\nMany thanks for your detailed and constructive comments. We recognize and agree with the limitations of our work and address the specific comments and improvements here.\n\n**Response to weakness and questions:**\n\n> **WK1: Relation with reference [36]** \n\nThank you for bringing this up! We would modify our paper and add this missing discussion. The aim of the VISIT aligned with our paper. However, they mainly build the graph by logit lens and by pruning the unactivated neuron, while our methods measure the contribution to the prediction. Also, they mainly focus on the behavior of the specific neuron or block in the Transformer model, ignoring the cooperation of these different components, while our work provides a new perspective. Finally, they simply conduct several case analyses and do not consider further analysis of the background mechanism under knowledge editing and hallucinations. We believe our two papers supplement each other, and employing the fine-grained analysis could help us understand our identified circuit more in the future.\n\n> **WK2: Robustness of the methods.**\n\nVery good point.  Our method is computed based on the reverse-topological order (so the nodes are sorted from output to input) to traverse the computation graph. \n- **We put the different computed circuits based on the topological order and our reverse-topological order in Figure 1 in the attached pdf in the global response.** We can see that if we use the original topological orders, the resulting circuits tend to retain redundant nodes in the initial layers. This occurs because the Transformer processes information layer-wise. If nodes from an earlier layer are ablated first, it disrupts the input to subsequent layers, potentially causing a substantial decline in performance. In contrast, when we apply our reverse-topological order, the ablation of nodes in a later layer does not impact the computations of the preceding layers. **Also, the topological order would omit the later layers as in our experiments in the main text, the model would give us the answer before the last layer, leading to an incomplete circuit.**\n- Meanwhile, in our experiments, we found that **our reverse-topological order also takes less time than the topological order.** By ablating nodes in the later layers at the outset, our method significantly reduces the overall computation time. In comparison, the topological order necessitates computing through all layers until the final stage, which is inherently more time-consuming due to the denser circuits in the later layers.\n\nThese results demonstrate the robustness of our methods, and from our view, this point deserves a new paper to discuss the casual effect order when computing the graph and is not in the main scope of our work. Thanks again for your very good point.\n\n> **WK3: lacks clarity.**\n\nThanks for your advice. As to the ICL problem you mentioned, our work supports the previous ICL mechanism [1]. They found the special attention heads’ output acts like the function vector for ICL demonstrations, and if we add the function vector output to the original zero-shot input, the model can do the same task. However, they didn’t analyze the behavior of these attention heads. Based on our proposed methods, we can discover these special attention heads demonstrate the behavior in Figure 5(b).\n\nTo clarify our findings, we conduct experiments by ablating the newly appeared attention head in the ICL circuit. **We can find that compared to the randomly selected attention head by ablating this attention, the probability drops significantly in the prediction, demonstrating the importance of the component our method identified.**\n\n|  |  | Origin Model | Ablating extra head in ICL circuit | $\\Delta$ (Original - Ablating extra head )| Ablating random head |\n| --- | --- | --- | --- | --- | --- |\n| Linguistic | adj_comparative | 62.24 | 32.55 | 29.69 | 58.18 |\n| Commonsense | word_sentiment | 89.02 | 55.50 | 33.52 | 88.61 |\n|  | substance_phase | 78.74 | 52.85 | 25.89 | 71.24 |\n| Bias | occupation_gender | 86.97 | 59.54 | 27.03 | 86.54 |\n| Factual | person_occupation | 35.17 | 23.27 | 11.90 | 31.60 |\n\nMoreover, for the editing task, we provide another observation of the mover head’s behavior change in Figure 2 in the attached pdf in the global response, which also make our findings more clear. We can find that before editing, the mover head retrieves the correct information for the Windows Server 2003 “Microsoft”, but after the edit, the mover head retrieves the error information ‘Intel’, which is actually the edited information at the subject place by the ROME method.\n\n\n> **WK4: formal definitions of mover head, relation head**\n\nThanks for your advice. In our previous version, we defined these concepts in the Appendix due to the space limit, and we would mention them in the main paper when modifying it.\n\n> **Question: the performance drop of the circuit varies across different datasets.**\n\nOur experiments test the Hit@10 to measure whether the model knows the knowledge. As we ablate everything but the circuit, we anticipate a little drop-down in the prediction. In our analysis, we found that for some kind of knowledge like \"fruit inside color\", the original rank of the target token is around 6-8, and after ablating other things, the rank is slightly higher than 10. Hence, this drop-down here is a bit more. The most interesting finding is that we can see an increase in several kinds of knowledge, suggesting that the discovered knowledge circuits may encapsulate the relevant knowledge and the model’s performance on these tasks could have been hindered by noise from other components.\n\n---\n\n[1] Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, David Bau. Function Vectors in Large Language Models (2024)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NM23fyt7od", "reviewer": "Reviewer_8w5y", "summary": "The paper introduce a critical subgraph named Knowledge Circuit in the language model to understand how LLMs store and express the knowledge. Through extensive experiments and cases analysis, the authors show the significance of this proposed Knowledge Circuit, which can unveil implicit neural knowledge representations, elucidate internal mechanisms for knowledge editing, and facilitate interpreting language model behaviors.", "questions": "Please see above Weaknesses.", "limitations": "None", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The authors approach the knowledge mechanism of LLMs from the Knowledge Circuit perspective, which has not been explored in this field and has good interpretability.\n\n2. The authors conduct extensive experimental analysis, and find that the Knowledge Circuit is beneficial to unveiling implicit neural knowledge representations, understanding knowledge editing mechanism, and interpreting language model behaviors.\n\n3. The paper is written-well, with a direct motivation and a well organized structure.", "weaknesses": "1. It would be better if the authors analyze more knowledge editing methods.\n\n2. What is the average size of this subgraph? And what is the distribution of nodes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 8w5y,\n\nMany thanks for your detailed and constructive comments. We appreciate that you are optimistic about the impact of the work.\n\n**Response to weakness and questions:**\n\n> **WK: It would be better if the authors analyzed more knowledge editing methods.** \n\nThanks for your advice, we agree with your opinion. In fact, we previously evaluated the Knowledge Neuron methods to edit and we found utilizing knowledge neurons always failed to overturn the model’s prediction. We would consider other hyper-network editing methods in the future.\n\n> **Question: circuit size and distribution.**\n\n- In Table 1, we show the number of edges of the discovered circuit. Take GPT2-Medium as an example; the average number of edges in the final identified circuit is around 400, and the average number of nodes is about 80.  This proves that with only less than 5% of the original computing subgraph edges, the model can maintain over 70% of its original performance, demonstrating the effectiveness of our methods.\n- We list the distribution of the circuit in Figure 2. Consistent with our expectations, the initial layers of the network, where the model processes inputs and extracts foundational information, are dominated by the activity of attention and feed-forward blocks. As the model progresses to undertake more nuanced and specialized tasks, the engagement of individual attention heads and FFN blocks diminishes correspondingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tEJTj4Tne6", "reviewer": "Reviewer_8n8x", "summary": "This paper tries to study how language models (LMs) recall factual knowledge by finding circuits within the model internals in a mechanistic way, on a dataset of (subject, relation object)-like sentences. The discovered circuits help understand the mechanisms of LM's knowledge recall, and also enable some explorations on the impact of knowledge editing methods and analyze hallucinations and in-context learning (ICL).", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Discovering the internal mechanisms of LMs' factual recall is an important problem for better understanding and improving LM's knowledge-related abilities. The methodologies and the discovered circuits presented in the paper are interesting and may be useful for the community.\n- The analysis based on the discovered circuits is informative and could inspire further applications of circuit discovery.", "weaknesses": "- Overall, the findings in this paper are not particularly surprising, and mostly verify known mechanisms studied by prior investigations on knowledge recall in LMs. This could also be seen from the paper writing - in most (if not all) places where a discovered circuit component has a dedicated functionality, it is something already found in the literature (e.g., mover head, relation head, mixture head, etc.). Also given the simplicity of the task (recall the object entity given the subject and relation), it is not hard to anticipate these mechanisms in a transformer model in the first place.\n\n- The introduced methodology is not very technically involved. The adopted circuit discovery method is a direct application of standard causal mediation analysis in the literature, and very much follows the framework of Conmy et al. as cited.\n\n- The interpretations of the mechanisms are somewhat subjective and not rigorously justified. For example, there are usually direct claims about a certain head or a certain MLP layer functioning as such and such, without enough supporting evidence. It would be better to include more statistics to help convince the reader why things are the way they are described in the main text. For example, how to quantitatively define what a \"mover head\" is, and how did you get this from the statistical evidence?\n\n- There are few in-depth discussions, potentially due to the simplicity of the task. But even within the task itself, the overall findings are quite fragile and it is very hard to get the bigger picture from all the presented findings. Ultimately, the important question is: are there issues with the current LM's mechanisms on factual recall, and how to address them? Take hallucination for example, is it always the case that \"the model fails to move the correct knowledge to the final token in the earlier layers\"? If this is the case, why does this happen, and is there a general method to resolve the issue besides checking every error example and fixing the problematic components?\n\n- The writing overall could be improved. For example, \"zero ablation\" and \"MatchNLL\"  are repeatedly mentioned in Section 3.1 and 3.3. The caption around Figure 2 is not informative enough to understand the figure content (e.g., are these attention/mlp nodes?) and is too close to the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 8n8x,\n\nMany thanks for your detailed and constructive comments. We hope that this reply answers all your questions, and we look forward to further discussion.\n\n**Addressing Weaknesses and Questions:**\n\n> **WK1&WK2: The findings in this paper are not particularly surprising, and the introduced methodology is not very technically involved.** \n\n- Factual recall may be a simple task for humans, but for current LMs, it’s still challenging, and we still do not know the mechanism of current LMs. In the previous work aiming to understand the behavior of  LM, they mainly focused on the identified objection identification (IOI), which doesn't require memory of the knowledge in the model.  Instead, in our work, we focus on knowledge recall, which is the basis of many complex tasks like multi-hop reasoning and chain-of-thought. As you mentioned in the strengths, our work could inspire further applications of circuit discovery.\n\n- Second, previous findings about the special attention head are intriguing but do not apply to the factual recall task. \nAlso, even if we anticipate these mechanisms in a transformer model in the first place, our work provides a new perspective to find them and investigate their behavior change. The analysis through the knowledge editing task enables us to see the mechanism more clearly and understand the drawbacks of the current LM and editing methods, which are also our main contributions.\n\n> **WK3: include more statistics to help convince the reader why things are the way they are described in the main text. For example, how do quantitatively define what a \"mover head\" is, and how did you get this from the statistical evidence.**\n\nThanks for your constructive advice. We provide the definition of these heads in Appendix and we will refine it in the main text. Here, we show more concrete definitions and statistics.\n- **Mover Head:** Attention Heads that attend strongly to the subject token and extract attributes of the subject. From Figure 3 and Figure 4, we can see that the rank of the token is similar to the subject position and the last token position. And in the knowledge editing task, we can view the changes in the behaviors.\n\n- **Relation Head:**  Relation heads extract many relation attributes by attending from the end token to the relation tokens.\n\nWe select these attention heads by the attention score of the specific direction in which they put more than half of their influence on the attention pattern. To better see the function of these components, we corrupt them in the computation to see the change in the model’s output. The amount of the mover head and relation head we removed is 10.\n\n| probability | Original Model | Ablating Mover Head | Ablating Relation Head |\n| --- | --- | --- | --- |\n| Target token  | 73.71 | 39.31 | 16.32 |\n| Other relation-related token  | 7.2 | 30.85 | 3.7 |\n\nFrom the Table, we can find the model's performance drop when we remove these special attention heads. Specially, mover head contribute more to the subject-specific information, as ablating them would increase other relation-related tokens' probability.  Moreover, if we ablate the relation head, we can find the model tends to generate some meaningless tokens instead of the relation information. \nWe also show one case of the output in Figure 3 in the attached pdf. From the figure, we can find ablating the mover head would increase the probability of \"Italian\", \"English\" and \"Spanish\", which are not subject-related. While ablating the relation head would lead to the increase of some meaningless word \"a\", \"that\", which are not relation-related.\n\n> **WK4: Are there issues with the current LM's mechanisms on factual recall, and how to address them? Take hallucinations for example, is it always the case that \"the model fails to move the correct knowledge to the final token in the earlier layers\"? If this is the case, why does this happen, and is there a general method to resolve the issue besides checking every error example and fixing the problematic components?** \n\nA good question, and that is exactly our ultimate goal. Current LMs still show some disadvantages, and understanding the mechanism is important for us to solve them. In our work, as mentioned by reviewer jmbU and in Figure 8 in the paper, we found the failure in the multi-hop questions scenario can be attributed to saturated information and the skip of the first hop.  Also, regarding the hallucination scenarios when the knowledge is stored in the knowledge model, we can see the mover's head select the wrong information, and until the last layer, the true answer moves to the top token.\n\nFor the possible solutions of these issues, knowledge editing currently shows the potential to address hallucinations and errors in large language model. In our analysis, we can see the ROME methods can alter the behavior of the mover head to help it gain the right information. In the future we can conduct some regularization here to make the information here at a proper level.\nAs you mentioned, our work could inspire further applications of circuit discovery, We acknowledge that there are still many mysteries that we haven’t uncovered, but we believe our work is a good beginning.\n\n> **WK5: The writing overall could be improved.**  \n\nThanks for your advice. We will polish our paper to make it more clear and easy to understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, 

We thank you all for the detailed and constructive comments.

*All* reviewers together found our contribution “a novel perspective on interpreting, points us in the right direction.” (Njw6) and “first to provide a mechanistic interpretation of the impact of knowledge editing techniques on a model” (jmbU) that “has not been explored in this field and has good interpretability” (8w5y) and “well formatted and easy to follow”(7LFR); “could inspire further applications of circuit discovery and may be useful for the community” (8n8x).

We sincerely appreciate your positive comments.

We’ve attached a PDF with several figures in this global response to tackle some questions provided by reviewers, and in our individual responses to reviewers, we hope that we have answered specific questions.

Our PDF includes:

1. Figure 1 compares the discovered circuit under reverse-topological order and topological order in GPT2-small under the same setting, demonstrating our methods' robustness. (reviewer 7LFR)
2. Figure 2 shows the behavior change of the edited model by ROME, clarifying our findings. (reviewer 7LFR)
3. Figure 3 shows the output of the original model and the model in which we ablate the mover head and relation head to support their function in a circuit. (reviewer 8n8x)

Meta-review: This paper considers how circuits are instrumental in articulating specific knowledge in LLMs. Experiments on GPT2 and TinyLLAMA allowed observations of how certain model components collaboratively encode knowledge within the model. Further, this paper provides explanations of hallucination and in-context learning via the circuit perspective.

The reviewers consider this paper well-written. 

The mechanistic findings about knowledge editing techniques (e.g., the edited knowledge ‘dominantly saturates’ the subject mover head) are among the earliest of its kind. I see two other works of this kind [1,2] but they are contemporaneous and are on different granularity levels — [1] on neuron-level circuits, [2] on SAE-level circuits, and this paper is on module-level circuits.

[1] What do the circuits mean? https://arxiv.org/abs/2406.17241 

[2] Sparse feature circuits https://arxiv.org/abs/2403.19647 

While the reviewers unanimously agreed on positive scores after discussion, some useful points can help improve the paper. The reviewers recommended some additional elaborations on concepts adopted from past papers along the mech interp thread, e.g., the mover head, relation head, and mixture heads. 

Additionally, it would be great to further analyze how the revealed findings (which are somewhat subjective) can benefit the model development. To me, this is a good point but it is a point towards the interpretability research field in general.
Decision: Accept (poster)