Knowledge Circuits in Pretrained Transformers

Authors: yunzhi  yao
ningyu  zhang
zekun  xi
mengru  wang
ziwen  xu
shumin  deng
huajun  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 15:13:25

Abstract: The remarkable capabilities of modern large language models are rooted in their vast repositories of knowledge encoded within their parameters, enabling them to perceive the world and engage in reasoning. The inner workings of how these models store knowledge have long been a subject of intense interest and investigation among researchers. To date, most studies have concentrated on isolated components within these models, such as the Multilayer Perceptrons and attention head. In this paper, we delve into the computation graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. The experiments, conducted with GPT2 and TinyLLAMA, has allowed us to observe how certain information heads, relation heads, and Multilayer Perceptrons collaboratively encode knowledge within the model. Moreover, we evaluate the impact of current knowledge editing techniques on these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies. Finally, we utilize knowledge circuits to analyze and interpret language model behaviors such as hallucinations and in-context learning. We believe the knowledge circuit holds potential for advancing our understanding of Transformers and guiding the improved design of knowledge editing.

Main paper: 
 introduction: 
 "Knowledge is power, and when embodied in the form of new technical inventions and mechanical discoveries it is the force that drives history." [1, 2] , Bacon's words are vividly re-enacted in the era of Large Language Models (LLMs) [3, 4] , as we witness their immense power in reshaping human society and redefining our understanding of machine intelligence. One thing that cannot be denied is that knowledge encapsulated within these models empowers their capabilities in reasoning, perceiving the world, and engaging in human-like communication. Nevertheless, these powerful models are not without their flaws. They still struggle with issues such as hallucinations [5] [6] [7] , unsafe norms [8, 9] , and offensive behaviors [10, 11] and these problems are exacerbated by the enigmatic internal mechanisms of knowledge storage within language models.Recently, the research community has devoted significant efforts to unraveling the knowledge storage mechanisms of these models. Various studies [12] [13] [14] [15] [16] [17] [18] [19] have been conducted to shed light on this intricate process, aiming to enhance our understanding and improve the safety and reliability of language models. The main finding in previous work is that knowledge may primarily stored in the Multilayer Perceptrons (MLPs) of Transformer-based language models. These MLPs function as a key-value neural memory, with knowledge being stored in what are termed "knowledge neurons" (KN). Based on these findings, researchers conduct Knowledge Editing [18, 20] to update the language models' inaccurate facts, bias and unsafe content in their parametric space. Despite the initial success of these methods, there are still limitations, such as poor generalization, severe side The official language of France is French" in GPT2-Medium. Left: a simplified circuit and the whole circuit is in Figure 9 in Appendix. We use to skip some complex connections between nodes. Here, L15H0 means the first attention head in the 15th layer and MLP12 means the multi-perception layer in the 13th layer. Right: the behavior of several special heads. The matrix on the left is the attention pattern of each attention head and the right heapmap demonstrates the output logits of the hean by mapping to the vocabulary space. effects, and failure to effectively utilize edited knowledge [20, 21] , which motivate us to re-think previous approaches for interpreting knowledge storage in language models. Note that previous works treat the knowledge blocks as isolated components following the Restorative Theory [22] , often focusing on identifying the specific blocks that store particular knowledge. Several works [23, 24] have proposed that different types of knowledge are often located in the same areas, suggesting that the current KN thesis may be an oversimplification.To this end, instead of solely pinpointing tiny regions where the knowledge expressed can be localized, we aim to explore the cooperation between different components in Transformers like attention heads, MLPs, and embeddings, to understand how the language model stores and expresses the knowledge. Here, we introduce a new perspective: Knowledge Circuits, a critical subgraph in the language model to view the knowledge mechanism of Transformers. Note that Circuit, as a subgraph in the computation graph, has gained ever-growing attention in the mechanistic interpretability field [25] . Previous work [26, 27] has found several important circuits for specific tasks like Indirect Object Identification and Color Object Identification. These tasks necessitate the model to search the preceding context for a matching token and copy it into the next token prediction. In this work, we aim to construct knowledge circuits that require the model to utilize stored knowledge for making predictions. Our goal is to better unveil implicit neural knowledge representations, elucidate the internal mechanisms for knowledge editing, and interpret more complex behaviors of language models. Specifically, we leverage factual recall tasks and conduct experiments across various domains, including factual, social bias, linguistic, and commonsense knowledge. We utilize GPT-2 [28] and TinyLLAMA [29] to explore the potential knowledge representations and utilization mechanisms in these models. As shown in Figure 1 (a), we construct knowledge circuits associated with various expressions of knowledge using the existing knowledge stored in the language model. Through those discovered knowledge circuits, we find many interesting phenomena and conclusions as follows:Knowledge circuits unveil implicit neural knowledge representations. We find that even when the knowledge circuits are used independently, the language model can recall related knowledge with a significant portion of its overall performance, demonstrating the effectiveness of those discovered knowledge representations (circuits). We also delve into specific pieces of knowledge and analyze the information flow within their respective circuits, indicating that the language model tends to aggregate knowledge in the earlier to middle layers and further enhances this information in the later layers. We further uncover several special components (e.g., mover heads and relation heads) in transferring information to the final token position and capturing relational information from the context (Figure 1 Knowledge circuits elucidate internal mechanisms for knowledge editing. We conduct experiments to evaluate the impact of current knowledge editing methods on the language models' original knowledge circuits. Empirically, we observe that ROME [18] tends to incorporate edited information primarily at the edited layer. Subsequent mover heads (Appendix B.2) then transport this information to the residual stream of the last token. Conversely, during fine-tuning, the edited token is directly integrated into the language model, exerting a dominant influence on subsequent predictions.Knowledge circuits facilitate interpreting language model behaviors. We further utilize the knowledge circuits to interpret language model behaviors, such as hallucination and in-context learning. We observe that when hallucination occurs, the language model fails to correctly transfer knowledge to the final token in the earlier layers. This is evident as the knowledge circuit lacks an effective "mover" head, or the mover head selects incorrect information. Additionally, we notice that several new attention heads emerge in the knowledge circuit during in-context learning.2 Background: Circuit TheoryQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: Abstract and Section 1 Introduction Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminaries: 
 In the context of neural network interpretability, a circuit can be conceptualized as a humaninterpretable subgraph that is dedicated to executing specific tasks within a neural network model [30, 26, [31] [32] [33] . When we visualize a neural network model as a connected directed acyclic graph (DAG), denoted as G, the individual nodes represent the various components involved in the forward pass, such as neurons, attention heads, and embeddings. The edges symbolize the interactions between these components, including residual connections, attention mechanisms, and projections. A circuit, represented as C ⊆ G, emerges as a significant subgraph of G that is responsible for particular behaviors or functionalities. In this paper, we focus on the Transformer decoder architecture to conduct our experiments. The residual stream of Transformers has been demonstrated to be a valuable tool for mechanistic interpretability in recent works [25, 16] . The Transformer architecture typically starts with token embeddings, followed by a sequence of "residual blocks" and concludes with a token unembedding. Each residual block comprises an attention layer and an MLP layer, both of which "read" their input from the residual stream (via a linear projection) and "write" their output back to the residual stream through an additive projection. We can consider an attention head A l,j (the jth attention head in layer l) as operating on the residual stream from the previous layer, R l-1 . Given that R 0 = I (where I represents the input embeddings), we can reinterpret attention head A l,j as processing the cumulative output of all previous attention heads and MLPs and input embedding, treating each node in the previous layers as separate input arguments. Similarly, an MLP node M l can be seen as operating on the cumulative output of all previous attention heads and MLPs and input embedding, and the output node O operates on the sum of the input embeddings and the outputs of all attention heads and MLPs. The following equations represent the residual connections in the Transformer model, where R l is the residual stream at layer l, and Input A l and Input M l are the inputs to the attention and MLP layers, respectively:R l = R l-1 + j A l,j + M l , R 0 = I Input A l = I + l ′ <l   M l ′ + j ′ A l ′ ,j ′   Input M l = I + l ′ <l M i ′ + l ′ ≤i j ′ A l ′ ,j ′The computational graph G of the Transformer represents the interactions between attention heads and MLPs. The nodes in G encompass the input embedding I, attention heads A l,j , MLPs M l , and the output node O, denoted as N = {I, A l,j , M l , O}. The edges in the model represent the connections between these nodes, E = {(n x , n y ), n x , n y ∈ N }. A circuit C is meticulously constructed to govern specific behaviors within the model, comprising a selection of nodes N C and edges E C that are critical to the successful execution of the tasks at hand, expressed as C =< N C , E C >.To identify circuits within a language model, a key approach is to examine the model's casual mediation by systematically altering the model's edges and nodes to observe the effects on performance [32, 34, 35] . The underlying principle is that critical edges or nodes are those whose removal results in a notable decline in the model's predictive capabilities. Since the edges in the model's computational graph represent the dependencies between nodes, we can simulate the absence of a particular node-to-node dependency by ablating an edge in the graph. For example, ablating an edge from A i ′ ,j ′ to A i,j involves replacing the contribution of A i ′ ,j ′ in the input to attention head A i,j with zero (in the case of zero ablation) or with the mean value of head A i ′ ,j ′ (in the case of mean ablation). The process of identifying critical edges or nodes through ablation can be broken down into the following steps: i) Overwrite the value of the edge (n x , n y ) with a corrupted value (either zero or mean ablation), ii) Perform a forward pass through the model with the altered graph, iii) Compare the output values of the modified model with those of the original model using a chosen metric S (Details in Eq.1 ). If the performance change is below a predefined threshold τ , we can consider the edge non-critical and remove it to obtain a new subgraph G/(n x , n y ). In addition to ablation-based methods, recent works have also explored the use of sparse auto-encoders [36, 37] to identify circuits within language models. This approach involves training an auto-encoder to learn a sparse representation of the model's internal structure, which can help reveal the underlying circuitry responsible for specific behaviors or functionalities.3 Knowledge Circuits Discovery in TransformersQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification:Section 8 Limitations and Appendix E.2.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
knowledge circuits construction: 
 Unlike previous work [12, 18] , which managed to find out the specific areas that store knowledge, we pay extra heed to the information flow that activates subsequent knowledge for answering questions.Similar to [38, 26] , we write language model as a graph consisting of the input, the output, attention heads, and MLPs by considering a "residual rewrite" of the model's computational structure. For example, this residual rewrite gives us a nearly-dense graph in GPT2-medium: one between every pair of (attention head, MLP, input, and output) nodes, except for attention heads in the same layer, which do not communicate with each other. In our paper, we concentrate on the task of answering factual open-domain questions, where the goal is to predict a target entity o given a subject-relation pair (s, r). A knowledge triplet k = (s, r, o) is often presented to the model in the form of a natural language prompt for next token prediction (e.g., "The official language of France is ____"). The model G is expected to generate the target entity, which is consistent with the language model's pretraining format. To identify the circuit that is critical for predicting the target entity o for a given subject-relation pair (s, r), we ablate each special edge e i = (n x , n y ) in the computation graph G. We then measure the impact of ablating the edge (zero ablation in our implementation) on the model's performance using the MatchNLL loss [32] for the target o:S(e i ) = log(G(o|(s, r))) -log(G/e i (o|(s, r)))If the score S(e i ) is less than the predefined threshold τ , we consider the edge to be non-critical and remove it from the computation graph, updating the temporary circuit C temp ← G/e i . We first sort the graph by topological rank following Conmy et al. [32] and traverse all edges in this manner, We derive a circuit C k that contributes to representing the knowledge necessary to answer the factual question:C k =< N k , E k > (2)Here, C k is the circuit for the knowledge triplet k, consisting of the nodes N k and edges E k that are essential for predicting the target entity o given the subject-relation pair (s, r).Once we have identified the knowledge circuit, we delve deeper into the specific roles and behaviors of each node and edge within the computation graph. Our goal is to comprehend the processing and contribution of each node n i to the functionality of the circuit. Drawing on the methodologies of previous studies [16, 39, 40] , we begin by applying layer normalization to the output of each node n i and then map it into the embedding space. This is achieved by multiplying the layer-normalized output by the unembedding matrix (W U ) of the language model: W U LN(n i ). This transformation allows us to inspect how each component writes information to the circuit and how it influences subsequent computational steps. By understanding the nodes' behavior in the circuit, we can better comprehend the circuit's structure and the key points where information is aggregated and disseminated. Implementations. We conduct experiments on GPT-style models, including GPT-2 medium and large. We also conduct primary experiments on TinyLLaMA [29] to validate the effectiveness of different architectures. We utilize the Automated Circuit Discovery [32] toolkit to build a circuit as an initiative of our analysis and leverage transformer lens [41] to further analyze the results. Specifically, we simply employ the MatchNLL [32] as the metric to detect the effect of the given node and edge and use zero ablation to knock out the specific computation node in the model's computation graph.A discovered knowledge circuit is deemed an accurate representation of a specific area within the transformer's knowledge storage, thus, it should be capable of representing the knowledge independently. Following [32] , we leverage the completeness of a circuit, which refers to its ability to independently reproduce the behavior or predictions of the full model for the relevant tasks. This property is assessed by examining whether the identified subgraph corresponds to the underlying algorithm implemented by the neural network. To evaluate completeness, we first construct the circuit using the validation data D val for a specific knowledge type and then test its performance on the test split D test in isolation. By doing so, we can observe any changes in performance compared to the original model. We use the Hit@10 metric to measure the rank of the target entity o among the top 10 predicted tokens:EQUATIONHere, |V | represents vocabulary size, and rank o is the rank of the target entity o in predictions.Dataset. In this work, we focus on the knowledge that is already stored in the language model. We utilize the dataset provided by LRE [42] and consider different kinds of knowledge, including linguistic, commonsense, fact, and bias. We evaluate whether the knowledge is present in the language model's parameters under zero-shot settings using the Hit@10 metric to sample knowledge from the validation set, which is used to construct the knowledge circuit. The data statistics are in Appendix A.maintain over 70% of its original performance. Additionally, we compute the random circuits by randomly deciding whether the edge should be removed and making sure the graph is connected. The random circuit is the same size as the circuit we discovered using our method. From the table, we can see that the random circuit failed to maintain the model's performance, which further enhanced the robustness and efficacy of our methods. One of the most fascinating observations is the performance improvement seen on several test datasets. For instance, the Landmark-country relation metric increases from 0.16 to 0.36. This suggests that the discovered knowledge circuits may encapsulate the relevant knowledge, and the model's performance on these tasks could have been hindered by noise from other components. We proceed to analyze the layer distribution of the original model G to understand the average percentage of nodes that are activated within the circuit for different knowledge domains. From Figure 2 , we observe that attention and MLPs are more active in the lower layers of the network, where the language model processes the input and extracts general information.To gain a more comprehensive view of the information processing, we compute the average rank o change of the target token in the D val across the layers and report the results in Figure 7 . This analysis reveals the phenomenon of early decoding [40] , suggesting that by the middle to the latest layers, the target entity is already present in the residual stream, and the subsequent layers in the Transformer are designed to increase the probability of the current token (See discussion in the running example). Special Components in Knowledge Circuits. From the discovered knowledge circuits, we can find several important attention heads that demonstrate specific behavior, including the mover head [31] , relation head [17, 43] and mixture head [17, 43] (more definitions in Appendix B.2). Mover Head [31, 27] focuses on the last token of the context and attends to the subject token, functioning as a mover to transfer information, while Relation Head [17] attends to the relation token in the context and produces some relation-related tokens that would guide the behavior of the following components. We think that these components would be accumulated by the MLP in the model, and the behavior of these special heads will be discussed in the running example part. We list some of these special components in Table 4 in Appendix. The different attention heads are responsible for expressing specific types of knowledge and may be activated by different facts. In our experiments with GPT-2 Medium and GPT-2 Large, we find that knowledge is distributed across several layers' attention heads and MLP matrices, suggesting that the target knowledge appears to have been accumulated throughout the GPT-2 model. Conversely, in TinyLLAMA, the special components are more concentrated. As depicted in Figure 7 , the rank of the target entity in TinyLLAMA experiences a sharp decline around several layers, whereas in the GPT2 model, the decline is more gradual. We hypothesize that this discrepancy may be attributed to the model's knowledge capacity [44] and warrants further investigation.A Running Example of Knowledge Circuit. We present a case and analyze the specific behaviors of components within the identified knowledge circuits. In particular, we find some special attention heads in the model such as the mover head and the relation head. We demonstrate the function of these heads in Figure 6 by ablating them from the circuit. Taking the factual knowledge "The official language of France is French" as an example, we visualize the knowledge circuit in Figure 1 . To express the information flow within the model more effectively, we have plotted the rank and probability of the target entity o at each layer when it is mapped into the embedding space, in Figure 3 . From this figure, we can see that after MLP 17, the target knowledge emerges as the top token in the residual stream, and after that layer, it undergoes an increased probability. The edges connected to MLP17 are (L14H13 → MLP17), (L14H7 → MLP17), and (L15H0 → MLP17) . Here, the L14H13 is a relation head that focuses on the relation token in the context. The output of this head is relationrelated tokens such as "language" and "Language". The attention head L14H7 is a mover head that moves the information from the subject position "France" to the last token. Previous work [31, 19] has introduced this mover head as an argument parser, which moves "France" to the last token, and the subsequent MLP conducts a function application to map "France" to "French". An intriguing observation is that we can find the output of this head already contains the target answer entity, which significantly contributes to the final output (L14H7 → Output). Also, we see the probability of the subject token in Figure 3 at the last token is nearly zero across these layers. Hence, instead of the argument parser function, we consider this mover head as an extract head proposed by Geva et al. [13] , which aims to extract the related-information from the subject token's position. In the subsequent knowledge editing experiments, we can observe changes in the behavior of these types of heads. Additionally, instead of extraction in the later layers proposed by Geva et al. [13] , we notice a gradual decrease in rank across all early-to-middle layers. The MLP17 combines information from previous tokens and integrates this information to prioritize the target token at the top rank.Interestingly, upon tracing the information flow to L14H7, we discovered that it is predominantly activated by L7H14, a relation head, and its output features several language tokens, such as "Arabic".We hypothesize that L7H14 may function as a signaling mechanism to activate the associated mover head, but this hypothesis necessitates further investigation to be confirmed. After MLP17, several attention heads, such as L18H14 (a relation head) and L20H6 (a mover head), collaborated to further enhance the final prediction of the target entity.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
knowledge circuits elucidate internal mechanisms for knowledge editing: 
 In this section, our objective is to evaluate the impact of previous knowledge editing methods and validate the effectiveness of knowledge circuits. We aim to understand why these methods may fail in certain cases and settings, which can also help deepen the understanding of the knowledge circuit.Single Factual Knowledge Editing. Here, we adopt the ROME method [18] and FT-M [24] , which aim to edit the MLP layers in the language model. The most important hyper-parameter in knowledge editing is the layer, as the same method's performance varies significantly via the layers. Here, we evaluate the performance of different editing layers and their effectiveness. We compare the knowledge circuits computed by the edited model with the original one, and we present results in Figure 4 and report details in Appendix D. As discussed in the previous part, the early-to-middle layers are the main part of aggregating the target entity o to the top rank. In the original model, the probability of the target entity "Intel" is nearly zero, and the model fails to elevate it to the top rank in the vocabulary. Editing the model with ROME and FT-M both give us the correct answer but we can view different scenarios for their knowledge circuits. For ROME, as the correct information is added to the subject position, we can recognize a behavior of the Mover Head shifts from copying to extracting the edited information from the subject position. This information gradually aggregates through the subsequent layers, and by layer 15, "Intel" emerges as the top-ranked entity with its probability increasing significantly. Specially, before editing, the mover head L15H3 attends to the "controller" token and returns "controller" as the output, while in the edited model, the attention head's output moves to the "Intel", which means the model gains the information at the subject space. For FT-M, the edited model tends to directly write the knowledge into the specific component, which would greatly dominate the following component in the model. As shown in Figure 4 , the output logits in MLP-0 for "Intel" are more than 10, and it emerges as the top rank in the residual stream directly. This phenomenon can be found in different knowledge types and layers and we report results in Appendix D.2. However, the added knowledge may have the risk of influencing unrelated knowledge. When we test another fact "Windows server", the model still tends to give us the "Intel" answer, demonstrating the overfitting problem. This finding supports previous analysis regarding the correlation between localization and editing [45] , suggesting that edits may not alter the storage but merely add signals into the knowledge circuits.Multi-hop Factual Knowledge Editing. Multi-hop knowledge editing poses a challenging scenario [20, 21, 46] , wherein we edit the model with new knowledge, yet the model struggles to perform reasoning using the edited information. We analyze multi-hop questions in language models [47, 48] L15H3 Original ModelWindows Server 2003 is created by Microsoft.Windows Server 2003 is created by Intel. Edit Case:Other Case:Platform Controller Hub is created by Intel.Other Case:Windows Server 2003 is created by Intel.Platform Controller Hub is created by Intel.Other Case:L15H3 1Figure 4 : Different behaviors when we edit the language model. In the original model, we can see the mover head L15H3 actually move the original token "Controller" and other information, while for ROME, we observe the mover head select the correct information "Intel", which means ROME successfully added the "Intel" to model. For the FT layer-0 editing, we can find this method directly write the edited knowledge into edited component. However, we find these two editing methods would affect other unrelated input "Windows server is created by?"to understand why current editing methods fail in these scenarios. For instance, given the fact (Thierry Mugle, "home country", France), we edit the fact to another country, such as (Thierry Mugle, "home country", France → China). We then assess the model's performance on questions based on the edited knowledge, including "The official currency of the home country of Thierry Mugle is" and "The capital city of the home country of Thierry Mugle is". While the unedited model could correctly answer these questions, we observe that the edited model would provide the answer "China" for subsequent hop reasoning. We find that the mover head in the original multi-hop reasoning circuit initially extracts the second-hop answer but, after editing, extracts "China", demonstrating that the edited information dominantly saturates and influences the circuit. Furthermore, we observe an intriguing phenomenon: even in the original model's multi-hop reasoning settings, it would directly provide the answer if we remove the context of the first-hop texts (Details in Appendix C.1). This further confirms the findings that the model relies on relational and subject-related information, regardless of grammatical adherence. 
knowledge circuits facilitate interpreting language model behaviors: 
 In this Section, our aim is to validate whether the identified knowledge circuits are actually utilized by the model when it employs knowledge. To address this, as shown in Figure 5 , we investigate three phenomena: hallucination, in-context learning, and reverse relations (Details in Appendix C.3).Factual Hallucination. If the knowledge is stored and expressed by the circuit we discovered, we aim to discover what happened when the model gave us the incorrect answer. We focus on factual hallucinations, which occur when the model provides an incorrect target entity for a given subject s and relation r. In our experiments (Figure 5 and Appendix C.2), we observe that the model fails to move the correct knowledge to the final token in the earlier layers. This failure is evident as the circuit lacks an effective mover head or the mover head selects incorrect information. For instance, in the prompt "The official currency of Malaysia is called", both the correct answer "Ringgit" and the incorrect one "Malaysian" are accumulated before layer 15. However, at layer 16, the mover head L15H10 extracts the erroneous information. Despite a rank drop of the true one in layers 20-22, this is insufficient to correct the previous mistake.In-Context Learning. Despite storing a vast amount of knowledge, a language model may still provide incorrect answers. However, with demonstrations or examples (based on RAG [49] ), it can quickly generate correct responses. To this end, we focus on the scenario where the model initially provides an incorrect answer but can then produce the correct response upon receiving the appropriate demonstration. We consider the original knowledge circuit and introduce a new knowledge circuit based on the demonstration. Our analysis reveals that, compared to the zero-shot knowledge circuit, several new attention heads appear in the computation graph when the demonstration is incorporated. We show the behavior of these attention heads in Figure 5 . We can see these heads mainly focus on the demonstration's context: "The comparative form of small is smaller" and works as the Induction Head [50] that look back over the sequence for previous instances of the current token and find the token that came after it last time. To better view the function of these heads, we conduct experiments by ablating the newly appeared attention head in the ICL circuit in Table 2 . We find that compared to the randomly selected attention head by ablating this attention, the probability drops significantly in the prediction, demonstrating the importance of these identified attention heads. These aligned with previous work where Todd et al. [51] have identified a concept known as the Function Vector, which represents the average of some key attention heads and provides the task learning ability. 
related work: 
 Knowledge Mechanism of Transformers. How the language model stores and utilizes knowledge is an ongoing research topic. Previous works find that the MLP in Transformers works as a keyvalue memory and stores enormous knowledge [12, 15, 14, 18] . As to the relation between entities, Hernandez et al. [42] observes that facts can be decoded linearly from the enriched residual stream of the subject by mapping the subject entity to the object entity. Instead of viewing the knowledge storage in isolation, Geva et al. [13] , Lv et al. [31] , Yu and Ananiadou [16] find the knowledge is accumulated during the layers. Regarding knowledge analysis, Bayazit et al. [52] also attempts to discover critical knowledge in language models. However, they only consider several layers in the Manipulate Language Models. Recently, many works aim to manipulate the language models to make the model aligned with world knowledge or social value norms, such as knowledge editing [20, 24] , machine unlearning [53, 54] and detoxification [55, 56] . Most of these works are elicited by previous knowledge mechanism findings such as knowledge neuron [57] . They modify the MLP in the LLM [18, 12] to change the model's behavior based on specific factual knowledge. However, recent works [58, 59] demonstrate the pivotal role of the attention part in knowledge representation.Hase et al. [45] also observe that the performance of editing within a layer may not reliably pinpoint the location of the fact. In this paper, we try to manipulate specific knowledge of language models via knowledge circuits, including both MLP and attention components across different layers. 
conclusion: 
 In this paper, we present a new perspective on knowledge storage based on circuit theory and conduct a preliminary analysis to demonstrate its effectiveness. We found that knowledge circuits in the model are not only responsible for expressing knowledge but can also guide behavior in different settings. We hope these findings can advance our understanding of the knowledge mechanisms of language models and provide insights for better designing and editing language models, enhancing knowledge, and improving reasoning to enhance factuality and alleviate hallucinations. In the future, we may further explore how these attention heads are formed during pretraining and activated. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [NA] Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use publicly available artifacts and show them in Section 3.3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this work, we employ the causal mediation method to automatically construct circuits tailored to specific knowledge domains. However, this circuit discovery-based patching approach is timeintensive. Contemporary research efforts have introduced more efficient methodologies for modeling information flow [60] [61] [62] . Additionally, alternative techniques for discovering circuits through mask training [63, 64] and Sparse Auto-Encoders [65, 36] have been proposed, highlighting diverse facets of circuit behavior within large language models (LLMs). We posit that the field of knowledge circuit discovery holds significant potential for advancement. Furthermore, recent studies [66] have developed 'circuit breakers' to manage representations associated with potentially harmful outputs. By concentrating on linguistic, factual, commonsense, and bias-related knowledge, we assert that our methodology can contribute to ensuring the safety and privacy of information, thereby fostering the development of trustworthy AI.To gain a clearer understanding of the knowledge aggregation phenomenon, we compute the rank of the target entity rank o within the vocabulary space |V | at the output of each layer. As depicted in Figure 7 , we observe that the model initially elevates the target entity to the top ranks of the vocabulary. Once the entity reaches the top of the vocabulary, subsequent layers continue to enhance its probability mass. These findings corroborate previous work by [68] [69] [70] , who note the substantial difference in logit entropy between layers, contributing to the model's improved prediction for the target entity. In our work, we aim to delve deeper into how the model, as well as specific components within it, give rise to these behaviors.When zooming into the discovered circuit, we can find several kinds of special attention heads, or MLPs, in the model that play a pivotal role in the final prediction, similar to what previous research has indicated [31] . Apart from mover head and relation head, there is another kind of head named Mix Head [17, 43] which would focus on both the relation token and the subject tokens. In our experiments, we found these heads usually work similarly to the mover head. In particular, mover heads contribute more to the subject-specific information, as ablating them would increase other relation-related tokens' probability. Moreover, if we ablate the relation head, we can find the model tends to generate some meaningless tokens instead of the relation information. From Figure 3 6, we can find ablating the mover head would increase the probability of "Italian", "English" and "Spanish", which are not subject-related. While ablating the relation head would lead to the increase of some meaningless words "a", and "that", which are not relation-related.Lv et al. [31] posits that these mover heads are responsible for extracting the "argument" from the context and passing it for further processing, such as function application. Consequently, Lv et al. [31] , Merullo et al. [19] suggests that the MLP within the language model performs a function The output of the model. Ablating the mover head would increase the probability of "Italian", "English" and "Spanish", which are not subject-related. While ablating the relation head would lead to the increase of some meaningless words "a", "that", which are not relation-related. Layer RankFigure 7 : The average rank of the target entity o for the D val in the vocabulary when mapping the output of each layer in the model to the embedding space. We can find that the in GPT2-Medium and GPT2-Large, the model would get the knowledge at middle-to-later layers. While for TinyLLAMA, the layer may be more later.application that transforms the subject into an object, with the subject's probability ranking higher than the object's. However, these findings are limited to specific cases, as the studies only examine two related tasks, such as capital city identification or color objection. Our analysis suggests that these conclusions may not be universally applicable to all knowledge domains and require further investigation. When we examine the ranks of subject and object entities, we rarely observe an overwhelming superiority at the last token position of the subject token and usually, the probability of the subject token at the last position is uniformly low. In our experiment, we view the model as a collaboration between different nodes in the identified circuit. They perform as different kinds of attention heads, like mover head and relation head. We list some heads that are responsible for the storage of different kinds of knowledge and relations in Table 4 .Component Reuse Phenomenon. Merullo et al. [27] have identified shared circuits for the IOI task and the Colored Objects task. We also observe this phenomenon in the factual recall task. As depicted in Table 4 , we can observe that for related relations such as "city_in_country", "name_birth_place", and "country_language", their circuits include both L21H12, which stores and maps country-related information. Additionally, we found that some relation heads are activated by different relations. For instance, in our experiments, the head 'L7H14' appears in the circuits of both "official_language" and "official_currency". We speculate that these reused heads, rather than task-specific heads, can be considered topic heads, as proposed by [71, 60] . We believe that further investigation into this distinction is warranted in future research. Figure 9 : The knowledge circuit from the "The official language of France is French" in GPT2-Medium.GPT2. We can also view some mover heads and relation heads in the circuit that would generate the target token as the output, such as L15H3 and L17H19.While previous work on knowledge storage suggests that knowledge may be localized in specific areas of the model, it is essential to ascertain whether the model actively employs this knowledge when encountering related contexts or if it relies on shortcuts.We consider scenarios where the model makes a correct prediction for all the multi-hop questions and single-hop questions. We found a circuit reuse phenomenon in the one-hop and multi-hop knowledge circuits. Here, we first compute the proportion of the nodes N single in the single-hop circuit C single that appears in N multiple the set of nodes in the multi-hop circuit C multiple .EQUATIONActually, there are two ways for the model to conduct multi-hop reasoning. As shown in the figure, the model can also answer the question by combining the two hop relations together (in the given case, combine "hometown" and "language" as "mother tongue") If the model is capable of combining two-hop relations in a more integrated or semantic way, such as inferring that the "mother tongue" is the language of one's hometown, this suggests a more complex reasoning process that goes beyond the simple overlap of nodes and edges. To capture this kind of reasoning, we assess the model's ability to integrate information from different hops. R integrated as the set of integrated relations (new paths created by combining information from different hops) We observe an overlap of these circuit nodes, indicating that the language model utilizes a large portion of the nodes in the original single hop's circuit, especially the mover head. From the overlap analysis, it seems the model utilizes single-hop information to conduct reasoning.We discover an intriguing phenomenon in GPT-2 and TinyLLAMA: the models can correctly answer first-hop knowledge and perform multi-hop reasoning based on it. However, interestingly, when we delete the first-hop knowledge while retaining only the second-hop relation and the first-hop subject, the models can still correctly answer the multi-hop question. Figure 10 illustrates a specific case in our findings. It further enhances our previous findings that the model actually conducts the factual recall with the relation head and the gathered information about the subject. Moreover, we found this phenomenon is hugely alleviated by the aligned model, which requires further investigation in the future.FranceThe official currency of the home country of the Thierry Mulger is the FrancThe official currency of Thierry Mulger is the Franc The results are presented in Figure 5 . We observe an interesting phenomenon: the correct answer and the wrong answer are both accumulated at the previous layer, but at some specific layers, the wrong answer is selected as the answer to extract. We hypothesize that there may be a circuit competition here proposed by [72] and we detect the behavior of the specific component between them.Reverse Curse [73] is an important issue in language models when models successfully give us the correct answer o for (s, r), but with a reverse relation r and o, the models fail to give us the correct subject s. In this part, we endeavor to investigate how the language model manages the reverse relation when they successfully store the knowledge. We first sample facts where the language model successfully predicts the given knowledge and the reversed fact. Then, we compute the overlap between these two circuits, C d and C r under node levels based on equation 4. We select the "superhero_person" relation to see the difference between these two circuits and test the node overlap of the two circuits in the model. We notice that the overlap between the two circuits is about 70%, indicating the language model may store the related information in the same place. We also found the activated mover heads for the two relationships to be identical.D.1 Method Implementation ROME As proposed by Meng et al. [18] , ROME views knowledge editing as a minimal optimization problem. ROME regards the MLP module as a simple key-value store. Specifically, the key represents a subject and the value encapsulates knowledge about that subject, the MLP can reestablish the association by retrieving the corresponding value for the key. To add a new key-value pair, ROME applies a rank-one modification to the MLP's weights, effectively "writing in" the new information directly. This method enables more direct and precise modification of the model's knowledge. The ROME method for model editing was conducted based on the EasyEdit [74] framework, utilizing the default parameters provided by EasyEdit. The experiments are performed on an A800 80G GPU, with approximately 8GB of memory consumption.FT-M For Fine-Tuning (FT-M), we follow Zhang et al. [24] . It trains the MLP layer using the crossentropy loss on the target answer while masking the original text. This approach aligns more closely with the traditional fine-tuning object. The FT-M method is conducted using the EasyEdit4 [74] framework, with the default parameters provided by EasyEdit. The experiments are also performed on an A800 80G GPU, with a memory consumption of approximately 10GB. When a circuit is established for a particular piece of knowledge, we can manipulate the model's computation by targeting critical points within the circuit. Li et al. [75] ablates a small number of important causal pathways by masking the edges in the circuit and making the model less toxic and safer, which proves the effectiveness of the circuit. As illustrated in figure 12 and figure 13 , we present the changes in the ranking of the predicted probabilities for the target new token when editing layer 6, 12, and 18 of the GPT-2 medium model using FT-M and ROME methods. When applying FT-M for model editing, it is evident that the rank of the target new token's probability sharply Hub was created by" with the target entity "Intel" shows that when editing the model using different layers, the fine-tuned settings allow the edited MLP to directly provide the edited information.Figure 12 : FT-M Rank Change Across Different Layers declines at the corresponding edited layer, resulting in a vertical line in the figure. This indicates that FT-M directly embeds the editing information into the model's information flow. Conversely, when using the ROME method for editing, this effect is mitigated. The predicted probability of the target new token reaches its peak only a few layers after the edited layer. This observation is consistent with our previous analysis in Section 5.E.1 More Related Work Knowledge in MLP Geva et al. [76] claim that MLP serves as key-value memories for knowledge in LMs. Geva et al. [77] further propose the knowledge neuron theory, suggesting that the key and observe that multiple distinct sets of KNs can store identical facts. Chen et al. [79] explore the structural and functional among neurons by neurological topology clustering method. Meng et al. [18] and Meng et al. [80] confirm that MLP modules do store factual knowledge and pioneering use of knowledge editing methods to modify outdated knowledge stored in language models. Anthropic recently introduces scaling monosemanticity 5 , which extracts highly abstract features that respond to and behaviorally cause abstract behaviors.Knowledge in Attention Heads Li et al. [58] reveal that some attention heads are capable of truthful answers. Wu et al. [81] investigate 4 model families, 6 model scales, and 3 types of finetuning, and find retrieval heads, which are responsible for retrieving relevant information from long context. Jin et al. [82] suggest that memory heads can retrieve knowledge from internal memory, while context heads can recall knowledge from external context. Todd et al. [83] use causal mediation analysis on a diverse range of in-context-learning and find some attention heads, dubbed function vectors, which trigger the ability of in-context-learning.Recent works emphasize the importance of connections of components among language models for knowledge representation and utilization. Geva et al. [13] describe factual recall by the following three steps: (1) subject enrichment in MLP sublayers, akin to ROME [18] , (2) propagation of relations to the END token, and (3) selective extraction of attributes by attention heads in later layers. Lv et al. [31] apply projection and intervention to explore mechanisms in factual recalls tasks and conclude that task-specific attention head may move the topic entity to the final position of the residual stream, while MLP conducts relation function.Circuit Circuit discovery also plays a significant role in analyzing the internal mechanisms of the entire model [25] . Specifically, a circuit, comprising components such as MLP and attention, is a subgraph of the computation graph. Conmy et al. [32] design an automated circuit discovery approach that implements the specified behavior. Wang et al. [26] explain the circuits for the Indirect Object Identification (IOI) task. They use causal interventions to discover circuits responsible for the flow of information. Instead the above task-specific circuit, Merullo et al. [27] presents evidence a circuit is shared by similar tasks in IOI and Color Object (CO) tasks. Dutta et al. [84] construct circuits using attention heads, and further observe that attention heads are pivotal to chain-of-thought reasoning, i.e., attention heads that move information along ontological relations exclusively appear in the initial half of the layers, while the tokens responsible for writing the answer predominantly appear in the later half of the layers. However, the above-mentioned circuit studies either focus solely on a single component (MLP or attention) or only explore IOI and CO tasks. IOI and CO tasks necessitate the model to search the preceding context for a matching token and then copy it into the next token prediction. Also, despite the success of previous circuit discovery, we can hardly make it into real usage. Hence, in this work, we attempt to analyze a knowledge circuit consisting of both MLP and attention components and investigate the effect of current editing methods on the circuit to shed light on the future.Tools There are also many tools that are designed to analyze the LM's behavior, such as Logit Lens [85] , Attention len [86] , Attribution lens [42] and transformer-lens [41] . NeuroX [87] implements various interpretation methods under a unified API and provides insights into how knowledge is structured in representations and discovers the role of neurons in LM. Transformer Debugger [88] is an interpretability tool provided by OpenAI, which deploys the GPT-4 and sparse auto-encoder to explain the language neurons and attention head. PatchScope [89] is a tool provided by Google that uses a new model to explain the hidden states in the original model.Despite of the attempt to combine the attention head and MLP to view the knowledge storage as a whole, this work operates with a relatively coarse granularity of circuits. For instance, the neurons within an MLP may necessitate a finer level of granularity to fully capture their behavior and contributions. Even though we now know these components work together to express the knowledge, why they are activated is still opaque. Our methodology employs the logit lens as a means to detect and analyze component information. However, this approach may encounter discrepancies between the middle layers and the output unembedding matrix. Such discrepancies can hinder a comprehensive and concrete analysis of the circuit components' behavior in the early layers. This limitation suggests the need for more robust techniques to bridge the gap between intermediate representations and final outputs. Recently, the Attention Lens method [86] has been proposed, which involves training a specific unembedding matrix to map each attention head into the vocabulary space. While this method is promising, it is also resource-intensive. Nevertheless, it represents a potential starting point for a deeper understanding of the knowledge circuits within neural models. Moreover, our research indicates that several mover heads are reused across different types of knowledge or relational contexts. The mechanisms by which these heads are activated and the conditions under which they operate require further exploration and may shed light on why neurons are sometimes "monosemantic" responding to a single feature, and sometimes "polysemantic" [90] responding to many unrelated features.NeurIPS Paper ChecklistKnowledge Circuits Unveil Implicit Neural Knowledge Representations Knowledge Circuits Evaluation. We report the results of GPT2-Medium in Table1, which indicates that with only less than 10% of the original knowledge circuit's subgraph, the model canhttps://github.com/ArthurConmy/Automatic-Circuit-DiscoveryCheckpoint: https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3Thttps://github.com/zjunlp/EasyEdithttps://transformer-circuits.pub/2024/scaling-monosemanticity/index.html 


Decision: This paper considers how circuits are instrumental in articulating specific knowledge in LLMs. Experiments on GPT2 and TinyLLAMA allowed observations of how certain model components collaboratively encode knowledge within the model. Further, this paper provides explanations of hallucination and in-context learning via the circuit perspective.

The reviewers consider this paper well-written. 

The mechanistic findings about knowledge editing techniques (e.g., the edited knowledge ‘dominantly saturates’ the subject mover head) are among the earliest of its kind. I see two other works of this kind [1,2] but they are contemporaneous and are on different granularity levels — [1] on neuron-level circuits, [2] on SAE-level circuits, and this paper is on module-level circuits.

[1] What do the circuits mean? https://arxiv.org/abs/2406.17241 

[2] Sparse feature circuits https://arxiv.org/abs/2403.19647 

While the reviewers unanimously agreed on positive scores after discussion, some useful points can help improve the paper. The reviewers recommended some additional elaborations on concepts adopted from past papers along the mech interp thread, e.g., the mover head, relation head, and mixture heads. 

Additionally, it would be great to further analyze how the revealed findings (which are somewhat subjective) can benefit the model development. To me, this is a good point but it is a point towards the interpretability research field in general.

Reviews: review: paper_summary: paper_summary: This paper presents a circuit-based analysis of factual recall and model editing in language models. The authors make two key claims about knowledge storage and retrieval: 1) Factual recall is compatible with circuit-based frameworks. 2) Traditional model editing methods influence behavior change within the model's "knowledge circuit." The paper compellingly advocates for studying knowledge storage and recall using the circuit framework, presenting preliminary but promising results.
strength_weakness: strength_weakness: ## General Comments

Despite the strengths and contributions highlighted, a significant shortcoming of this paper is the absence of a concrete, successful application of circuits for the tasks mentioned in the introduction and Section 6, such as model editing, hallucination detection, and in-context learning (ICL). This is why I described the review as "preliminary," even though the paper provides concrete experiments and evidence on the existence of a connection between factual knowledge recall and circuits.

Despite its shortcomings, this paper offers an intriguing and promising new approach to addressing the problem by building on Niu et al.'s [21] call to "...examine the entire decision-making circuit" when studying how knowledge is stored and recalled in language models (LMs) with supporting evidence and experiments. The authors highlight the limitations of the previous knowledge neuron theory of LMs, noting its inadequacies as a fundamental theory for interpreting the factual recall process in LMs. The community needs a new theoretical framework, and I believe this paper points us in the right direction.

### Knowledge Circuit Information Analysis

This section of the experiment aims to demonstrate Claim #1: that circuits can adequately explain the process of factual storage and recall. While the experimental results provide some preliminary support for this claim, the evidence is not as compelling as suggested in the introduction. Although the model shows initial success in using circuit-based interpretations for factual recall, some of the results are underwhelming. Notably, the relation "capital city" has a test set performance of 0%. Despite the authors' assertion that the standalone performance of the circuit is good, there is no baseline performance for comparison. Perhaps using random circuits as a baseline could provide a clearer benchmark.

### Related Work

The paper provides an adequate summary of prior work in Section 7 and the introduction; however, there is room for improvement. The introduction initially highlights the issue with previous knowledge editing methods at lines 40-42, noting that "different types of knowledge are often located in the same areas." While this is a valid issue, it is not the most critical one. The relevant work section (lines 336-339) offers a more comprehensive summary of the issues. Therefore, it would be beneficial to expand the introduction by incorporating a condensed summary from lines 336-339 to provide a clearer and more thorough context for the readers.
questions: questions: Please see the weaknesses section.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The authors propose utilizing techniques from mechanistic interpretability to explore the connections between model components involved in factual recall. They systematically ablate connections between model components in a reverse topological order, maintaining a list of the connections that most significantly recover loss on factual recall tasks. Several properties of the resultant "knowledge circuit" are measured, including its sufficiency for recovering model performance on factual recall.

The authors then demonstrate the roles of various components of the knowledge circuit, such as relation heads, mover heads, extract heads, and MLPs. They interpret the roles of these components by projecting their outputs directly into the logit space of the model. This approach is applied to understanding hallucinations and model editing failures.

The authors demonstrate how knowledge editing methods like ROME switch the role of the subject mover head to instead extract the final answer directly from the subject token rather than moving information about the subject.  The authors also manage to explain failures in multi-hop knowledge editing - the edited knowledge "dominantly saturates" the subject mover head preventing it from performing its natural function.  They also show how factual hallucination failures can be attributed to a mistake on the part of the subject mover head.
strength_weakness: strength_weakness: - The authors choose to perform ROME [1] edits at layer 0 of the model.  In contrast, the original ROME paper performs edits in the middle of the model.  The precise layer is chosen using a causal tracing algorithm.  This choice is not justified in the paper.  It is unclear what difference this makes in the results provided in this paper.
- In table 1, it is unclear why ablating everything but the circuit improves performance on some tasks, but decreases others.  For example, restricting the model to just the circuit increases the accuracy of "Country Language" but decreases the accuracy of "fruit inside color".

[1] Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov. "Locating and Editing Factual Associations in GPT." (2023).
questions: questions: - Why did the authors choose to perform ROME edits at layer 0 of the model, in contrast to the original ROME paper's approach of editing in the middle layers using a causal tracing algorithm, and what impact might this choice have on the results presented in the paper?
- In Table 1, what explains the inconsistent effects of ablating everything but the circuit on task performance, such as improving accuracy for "Country Language" while decreasing it for "fruit inside color"?
limitations: limitations: The authors discuss the limitations of their work, and how it can contribute to ensuring safety and privacy information to promote trustworthy AI.
paper_summary: paper_summary: This paper focus on the problem of interpreting the knowledge storage mechanism of the large language models. The authors proposed a new perspective, which uses the knowledge circuit to understand how the language model stores and expresses the knowledge. Knowledge circuit is a subgraph of the computation graph of the model. Specifically, the knowledge circuit of a given factual knowledge is defined as the smallest subgraph to recall it "from the knowledge stored in the model". They proposed a knowledge circuit construction method, where the nodes are ablated one-by-one to measure the impact of removing it from the computation graph. The resulted knowledge circuit will then be a necessary computation graph for maintaining the original inference result. Experiments are conducted across a variety of domains of factual recall tasks. To intuitively interpret the constructed knowledge circuits, they probes the information flow by projecting the latent representations into the space of token embeddings. Under this interpretation, several conclusions are drawn as follows.
- language model tends to aggregate knowledge in the earlier to middle layers and further enhances this information in the later layers.
- the knowledge editing method ROME tends to incorporate edited information primarily at the edited layer.
- when hallucination occurs, the language model fails to correctly transfer knowledge to the final token in the earlier layers.
- several new attention heads emerge in the knowledge circuit during in-context learning.
strength_weakness: strength_weakness: - Regarding the novelty of this work, reference [36] appears to be highly relevant, as it involves the construction of a flow graph and the presentation of semantic information flow. However, this reference is not discussed in the related work section.
- The robustness of the knowledge circuit construction method should be addressed. For instance, the results of the construction method should be unique or equivalent under different traversal orders. If not, the solution may not be well-defined.
- The interpretation based on the knowledge circuit lacks clarity. Although several conclusions are drawn in the paper, it is unclear if these conclusions align with existing objective observations on related phenomena, such as in-context learning, and whether they could aid in predicting the model's behavior. The effectiveness of an explanation method should be validated by deriving a conclusion or guidance that can be subsequently verified through observation. This aspect is not clearly demonstrated in the paper.
- It would be beneficial to include formal definitions of mover head, relation head, and mixture head, as these are crucial concepts in this paper.
questions: questions: - Please refer to the weakness part.
- According to Table 1, the performance drop of Circuit varies across different datasets. Is there any analysis of this phenomenon?
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduce a critical subgraph named Knowledge Circuit in the language model to understand how LLMs store and express the knowledge. Through extensive experiments and cases analysis, the authors show the significance of this proposed Knowledge Circuit, which can unveil implicit neural knowledge representations, elucidate internal mechanisms for knowledge editing, and facilitate interpreting language model behaviors.
strength_weakness: strength_weakness: 1. It would be better if the authors analyze more knowledge editing methods.

2. What is the average size of this subgraph? And what is the distribution of nodes?
questions: questions: Please see above Weaknesses.
limitations: limitations: None
paper_summary: paper_summary: This paper tries to study how language models (LMs) recall factual knowledge by finding circuits within the model internals in a mechanistic way, on a dataset of (subject, relation object)-like sentences. The discovered circuits help understand the mechanisms of LM's knowledge recall, and also enable some explorations on the impact of knowledge editing methods and analyze hallucinations and in-context learning (ICL).
strength_weakness: strength_weakness: - Overall, the findings in this paper are not particularly surprising, and mostly verify known mechanisms studied by prior investigations on knowledge recall in LMs. This could also be seen from the paper writing - in most (if not all) places where a discovered circuit component has a dedicated functionality, it is something already found in the literature (e.g., mover head, relation head, mixture head, etc.). Also given the simplicity of the task (recall the object entity given the subject and relation), it is not hard to anticipate these mechanisms in a transformer model in the first place.

- The introduced methodology is not very technically involved. The adopted circuit discovery method is a direct application of standard causal mediation analysis in the literature, and very much follows the framework of Conmy et al. as cited.

- The interpretations of the mechanisms are somewhat subjective and not rigorously justified. For example, there are usually direct claims about a certain head or a certain MLP layer functioning as such and such, without enough supporting evidence. It would be better to include more statistics to help convince the reader why things are the way they are described in the main text. For example, how to quantitatively define what a "mover head" is, and how did you get this from the statistical evidence?

- There are few in-depth discussions, potentially due to the simplicity of the task. But even within the task itself, the overall findings are quite fragile and it is very hard to get the bigger picture from all the presented findings. Ultimately, the important question is: are there issues with the current LM's mechanisms on factual recall, and how to address them? Take hallucination for example, is it always the case that "the model fails to move the correct knowledge to the final token in the earlier layers"? If this is the case, why does this happen, and is there a general method to resolve the issue besides checking every error example and fixing the problematic components?

- The writing overall could be improved. For example, "zero ablation" and "MatchNLL"  are repeatedly mentioned in Section 3.1 and 3.3. The caption around Figure 2 is not informative enough to understand the figure content (e.g., are these attention/mlp nodes?) and is too close to the main text.
questions: questions: See weaknesses.
limitations: limitations: Yes.

Comments: title: Response to rebuttal
Acknowledgement to the rebuttal
comment: We deeply appreciate your time and effort in reviewing our work and your support in raising your score. Your assessment has been invaluable in refining our work and clarifying the key aspects of our research. We are committed to refining our experimental approach and analysis in the revised manuscript to reflect the valuable feedback you have provided.

We sincerely thank you for your continued time and thoughtful consideration.

Warm regards!
I would like to thank the authors for their thoughtful response. Their willingness to improve the paper moves my opinion to the positive side. I hope they will incorporate the discussed improvements into the final version.
Dear reviewer 7LFR,

We extend our heartfelt gratitude for your invaluable feedback. We hope that our response will serve to address your concerns effectively.

- Our objective is to identify “knowledge circuits,” which are the sparse computational subgraphs within the Transformer model that encapsulate specific facets of its behavior related to factual recall. In light of the Transformer model’s understanding and the directed acyclic graph (DAG) nature of its computation, we have adopted a reverse-topological approach. This method, which involves tracing information in a top-down manner through the network, has been widely utilized in prior research [1-5] and has consistently demonstrated its reliability and efficacy.Regarding the DAG and activation patching, examining variables in reverse order significantly facilitates the isolation of each variable’s effects, thereby preventing confounding influences from upstream variables—a critical aspect when estimating direct causal effects. 
As we mentioned in our previous response, other order tends to maintain some redundant components and omit the later layers in the model, leading to an incomplete graph. It also requires more computation time (in our experiments about 5~8 times lower than the reverse-order).  

- To assess the validity of the constructed circuits, we have employed the circuit-standalone performance metric, a standard approach utilized in earlier studies [1,3-7]. Regarding the performance of Table1, we can see that the circuit-standalone can improve the performance on the test set, suggesting that the discovered knowledge circuits may encapsulate the relevant knowledge and the model’s performance on these tasks could have been hindered by noise from other unrelated components. 
To rigorously evaluate whether the constructed circuits genuinely represent the original “knowledge,” we have conducted a comparative analysis among the discovered circuit  $C$, random circuit $C_{random}$ and the complementary circuit $\mathcal{G}-C$ on the test sets. The random circuit matches the size of our discovered circuit, while the complementary circuit represents the graph from which the discovered circuit has been ablated.
|  |  $C_{random}$ | $C$ | $\mathcal{G}-C$ |
| --- | --- | --- | --- |
| Adj Antonym | 0 | 0.40 | 0.00 |
| object superclass | 0 | 0.52 | 0.14 |
| Landmark country | 0 |  0.36 | 0.10 |
| Person Native Language | 0 |  0.76 | 0.17 |
| name birthplace | 0 | 0.57 | 0.05 |

We can vew that our discovered circuit contribute most to the knowledge recall, indicating the circuit contains most of the nodes related to the knowledge recall.

We agree that the circuit-discovery method still have room for improvement and we discuss it in the limitations in the paper. 
In the recent days, some new works to improve the quality of the circuit emerging [6,8] and we still can see potential to update the circuit discovery through these works.
We believe this is an ongoing research endeavor and we want to underscore that the goal of our work is to find the knowledge circuit in LLM and adopt the circuit to analyze the reason behind of the shortcomings of the current knowledge editing methods and guide the further manipulation of the LM. Our work make a step here and we are committed to continuing this journey.

Once again, we thank you for your insightful advice and constructive feedback.

---
[1] Towards Automated Circuit Discovery for Mechanistic Interpretability.  (2023)

[2] Representation Engineering: A Top-Down Approach to AI Transparency. (2023)

[3] Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. (2023)

[4] Circuit component reuse across tasks in transformer language models. (2024)

[5] LLM Circuit Analyses Are Consistent Across Training and Scale. (2024)

[6] Finding Transformer Circuits with Edge Pruning. (2024)

[7] Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. (2024)

[8] Functional Faithfulness in the Wild: Circuit Discovery with Differentiable Computation Graph Pruning. (2024)
Many thanks to the authors for the detailed answers. I maintain the positive rating for the paper.
Thank you for taking the time to respond. I appreciate your thoughtful feedback, which has helped strengthen my analysis of the comparison between ROME editing and FT.  This has increased my confidence in the high quality of your work.
Thank you for providing a response to my comments and questions. It is very helpful. I maintain my positive review to the paper.
Thank you for your thoughtful feedback and for considering the additional information provided in our response. 
We appreciate your reference to Chughtai et al.'s work, “Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs” (Sumup for short). Both our works aims to discover the knowledge mechanism in language model. We agree that there are connections between the identified attention head by Chughtai et al. and our own findings. We will revise our manuscript to better acknowledge these connections and to ensure that we accurately represent the existing literature.

Here, we put an extra comparison between our two works:
- Our research focuses on identifying the neural circuit within the model that is responsible for knowledge retention. Our experiments show that the identified circuit significantly contributes to the model’s performance. By examining the components within the discovered subgraph, we have identified key attention heads and their interconnections that are crucial to the circuit’s function. In contrast, Sumup directly investigates the behavior of attention heads, with its primary contribution being the identification of these heads. Despite of their claim of the additive mechanism for the factual recall, they simply mentioned in Section 4 in their paper as a speculation. In our work, we can view these connection through the circuit we discovered. Additionally, the additive motif the Sumup paper proposed is not validated. Our experiments suggest that the behavior of the subject mover head changes when the connection to the relation head is removed, indicating potential causal relationships beyond simple additivity.

- Our work also adds value by offering a clearer explanation of the model’s operational behavior. Through the circuit perspective, we gain insights into the reasons behind failures in knowledge editing tasks involving locality and reasoning. Furthermore, examining these circuits allows us to understand the model’s behavior in scenarios involving hallucination and in-context learning. 

We are grateful for your willingness to slightly raise your evaluation based on the positive signals and promises from our response. Regarding the major weaknesses you identified, we understand your perspective, as the circuits we have discovered indeed hold great potential for further investigation. As you noted in your review, our analysis based on the discovered circuits is informative and could inspire further applications of circuit discovery. We believe our work represents a solid foundation in this area and are confident that the revisions we are undertaking will enhance the quality of our paper and more effectively address the concerns you have raised.

Thank you once again for your valuable input. We are committed to producing a high-quality, impactful research paper, and we appreciate your role in helping us achieve that goal.

Sincerely!
I would like to thank the authors for their efforts in addressing my concerns. However, a few issues still persist.     
Regarding the robustness of the knowledge circuit construction method, the authors acknowledge that the resulting circuits differ depending on the traversal order used. This raises concerns about the lack of rigorous definitions for the term "knowledge circuit" and the absence of an objective metric to evaluate the constructed circuits. The results in Table 1 indicate that there can be significant discrepancies between the decisions of the original model and those produced by the constructed circuit. This brings into question whether the constructed circuits truly align with the original "knowledge."
Thank you for the response, which addresses my concerns to a certain degree. Regarding "previous findings about the special attention head are intriguing but do not apply to the factual recall task", I believe this is not accurate. For example, "Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMs" by Chughtai et al (also cited as [15] in the paper here) investigates factual recall and finds "subject heads", "relation heads", 'mixed heads', etc. which overlap a lot with the findings here.

Overall, I believe the major weaknesses I mentioned (lack of novelty in techniques and findings, lack of deeper insights) still hold, but I'll slightly raise my evaluation due to the extra positive signals and promises from the response.