Title: Knowledge Circuits in Pretrained Transformers
Venue: NeurIPS 2024
Accepted: yes
Keywords: Knowledge Mechanism

Circuit Theory

Large Language Models
Abstract: The remarkable capabilities of modern large language models are rooted in their vast repositories of knowledge encoded within their parameters, enabling them to perceive the world and engage in reasoning. The inner workings of how these models store knowledge have long been a subject of intense interest and investigation among researchers. To date, most studies have concentrated on isolated components within these models, such as the Multilayer Perceptrons and attention head. In this paper, we delve into the computation graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. The experiments, conducted with GPT2 and TinyLLAMA, has allowed us to observe how certain information heads, relation heads, and Multilayer Perceptrons collaboratively encode knowledge within the model. Moreover, we evaluate the impact of current knowledge editing techniques on these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies. Finally, we utilize knowledge circuits to analyze and interpret language model behaviors such as hallucinations and in-context learning. We believe the knowledge circuit holds potential for advancing our understanding of Transformers and guiding the improved design of knowledge editing.

Knowledge Circuits in Pretrained Transformers

Abstract

The remarkable capabilities of modern large language models are rooted in their vast repositories of knowledge encoded within their parameters, enabling them to perceive the world and engage in reasoning. The inner workings of how these models store knowledge have long been a subject of intense interest and investigation among researchers. To date, most studies have concentrated on isolated components within these models, such as the Multilayer Perceptrons and attention head. In this paper, we delve into the computation graph of the language model to uncover the knowledge circuits that are instrumental in articulating specific knowledge. The experiments, conducted with GPT2 and TinyLLAMA, have allowed us to observe how certain information heads, relation heads, and Multilayer Perceptrons collaboratively encode knowledge within the model. Moreover, we evaluate the impact of current knowledge editing techniques on these knowledge circuits, providing deeper insights into the functioning and constraints of these editing methodologies. Finally, we utilize knowledge circuits to analyze and interpret language model behaviors such as hallucinations and in-context learning. We believe the knowledge circuits hold potential for advancing our understanding of Transformers and guiding the improved design of knowledge editing 1 .

1 Introduction

'Knowledge is power, and when embodied in the form of new technical inventions and mechanical discoveries it is the force that drives history. ' [1, 2], Bacon's words are vividly re-enacted in the era of Large Language Models (LLMs) [3, 4], as we witness their immense power in reshaping human society and redefining our understanding of machine intelligence. One thing that cannot be denied is that knowledge encapsulated within these models empowers their capabilities in reasoning, perceiving the world, and engaging in human-like communication. Nevertheless, these powerful models are not without their flaws. They still struggle with issues such as hallucinations [5-7], unsafe norms [8, 9], and offensive behaviors [10, 11] and these problems are exacerbated by the enigmatic internal mechanisms of knowledge storage within language models.

Recently, the research community has devoted significant efforts to unraveling the knowledge storage mechanisms of these models. Various studies [12-19] have been conducted to shed light on this intricate process, aiming to enhance our understanding and improve the safety and reliability of language models. The main finding in previous work is that knowledge may primarily stored in the Multilayer Perceptrons (MLPs) of Transformer-based language models. These MLPs function as a key-value neural memory, with knowledge being stored in what are termed 'knowledge neurons' (KN). Based on these findings, researchers conduct Knowledge Editing [18, 20] to update the language models' inaccurate facts, bias and unsafe content in their parametric space. Despite the

Output logits

Output logits

(a)  A simplified Circuits

(b)  Some Special Component's output

Residual Output: French

MLP19

MLP22

MLP18

L15H0

L14H7

MLP12

L7H14

MLP0

MLP1-9

Input Embed

The o

ffi

cial language of France is

L15H0

L14H7

L7H14

language

France

↓

The

The

S

The

The

Of

竹

language

official

languages

anguage

1.12

1.14

language

official

Arabic

dag

0.51

0.52

France

of

0.2

0.4

Languages

language

dialect

- 1.04

1.08

1.10

1.06

France

of

Agriculture

Franc

Tong

0.50

= 0.48

= 0.49

is

is

= 0.47

language

official

OF

The

official

Of

France

：

language

France

official

The

of

is

French

Marse

Franc

franc

Ré

0.640

= 0.635

0.650

0.655

0.645

language

France

official

The

of

is

0.4

Francois

Francois

Macron

franc

oire

0.90

0.86

0.84

0.82

0.92

0.88

initial success of these methods, there are still limitations, such as poor generalization, severe side effects, and failure to effectively utilize edited knowledge [20, 21], which motivate us to re-think previous approaches for interpreting knowledge storage in language models. Note that previous works treat the knowledge blocks as isolated components following the Restorative Theory [22], often focusing on identifying the specific blocks that store particular knowledge. Several works [23, 24] have proposed that different types of knowledge are often located in the same areas, suggesting that the current KN thesis may be an oversimplification.

To this end, instead of solely pinpointing tiny regions where the knowledge expressed can be localized, we aim to explore the cooperation between different components in Transformers like attention heads, MLPs, and embeddings, to understand how the language model stores and expresses the knowledge. Here, we introduce a new perspective: Knowledge Circuits , a critical subgraph in the language model to view the knowledge mechanism of Transformers. Note that Circuit, as a subgraph in the computation graph, has gained ever-growing attention in the mechanistic interpretability field [25]. Previous work [26, 27] has found several important circuits for specific tasks like Indirect Object Identification and Color Object Identification. These tasks necessitate the model to search the preceding context for a matching token and copy it into the next token prediction. In this work, we aim to construct knowledge circuits that require the model to utilize stored knowledge for making predictions. Our goal is to better unveil implicit neural knowledge representations, elucidate the internal mechanisms for knowledge editing, and interpret more complex behaviors of language models. Specifically, we leverage factual recall tasks and conduct experiments across various domains, including factual, social bias, linguistic, and commonsense knowledge. We utilize GPT-2 [28] and TinyLLAMA [29] to explore the potential knowledge representations and utilization mechanisms in these models. As shown in Figure 1 (a), we construct knowledge circuits associated with various expressions of knowledge using the existing knowledge stored in the language model. Through those discovered knowledge circuits, we find many interesting phenomena and conclusions as follows:

Knowledge circuits unveil implicit neural knowledge representations. We find that even when the knowledge circuits are used independently, the language model can recall related knowledge with a significant portion of its overall performance, demonstrating the effectiveness of those discovered knowledge representations (circuits). We also delve into specific pieces of knowledge and analyze the information flow within their respective circuits, indicating that the language model tends to aggregate knowledge in the earlier to middle layers and further enhances this information in the later layers. We further uncover several special components (e.g., mover heads and relation heads)

in transferring information to the final token position and capturing relational information from the context (Figure 1 (b)).

Knowledge circuits elucidate internal mechanisms for knowledge editing. We conduct experiments to evaluate the impact of current knowledge editing methods on the language models' original knowledge circuits. Empirically, we observe that ROME [18] tends to incorporate edited information primarily at the edited layer. Subsequent mover heads (Appendix B.2) then transport this information to the residual stream of the last token. Conversely, during fine-tuning, the edited token is directly integrated into the language model, exerting a dominant influence on subsequent predictions.

Knowledge circuits facilitate interpreting language model behaviors. We further utilize the knowledge circuits to interpret language model behaviors, such as hallucination and in-context learning . We observe that when hallucination occurs, the language model fails to correctly transfer knowledge to the final token in the earlier layers. This is evident as the knowledge circuit lacks an effective 'mover' head, or the mover head selects incorrect information. Additionally, we notice that several new attention heads emerge in the knowledge circuit during in-context learning.

2 Background: Circuit Theory

2.1 Preliminaries

In the context of neural network interpretability, a circuit can be conceptualized as a humaninterpretable subgraph that is dedicated to executing specific tasks within a neural network model [30, 26, 31-33]. When we visualize a neural network model as a connected directed acyclic graph (DAG), denoted as G , the individual nodes represent the various components involved in the forward pass, such as neurons, attention heads, and embeddings. The edges symbolize the interactions between these components, including residual connections, attention mechanisms, and projections. A circuit, represented as C ⊆ G , emerges as a significant subgraph of G that is responsible for particular behaviors or functionalities. In this paper, we focus on the Transformer decoder architecture to conduct our experiments. The residual stream of Transformers has been demonstrated to be a valuable tool for mechanistic interpretability in recent works [25, 16]. The Transformer architecture typically starts with token embeddings, followed by a sequence of 'residual blocks' and concludes with a token unembedding. Each residual block comprises an attention layer and an MLP layer, both of which 'read' their input from the residual stream (via a linear projection) and 'write' their output back to the residual stream through an additive projection. We can consider an attention head A l,j (the j th attention head in layer l ) as operating on the residual stream from the previous layer, R l -1 . Given that R 0 = I (where I represents the input embeddings), we can reinterpret attention head A l,j as processing the cumulative output of all previous attention heads and MLPs and input embedding, treating each node in the previous layers as separate input arguments. Similarly, an MLP node M l can be seen as operating on the cumulative output of all previous attention heads and MLPs and input embedding, and the output node O operates on the sum of the input embeddings and the outputs of all attention heads and MLPs. The following equations represent the residual connections in the Transformer model, where R l is the residual stream at layer l , and Input A l and Input l M are the inputs to the attention and MLP layers, respectively:

The computational graph G of the Transformer represents the interactions between attention heads and MLPs. The nodes in G encompass the input embedding I , attention heads A l,j , MLPs M l , and the output node O , denoted as N = { I, A l,j , M l , O } . The edges in the model represent the connections between these nodes, E = { ( n x , n y ) , n x , n y ∈ N } . A circuit C is meticulously constructed to govern specific behaviors within the model, comprising a selection of nodes N C and edges E C that are critical to the successful execution of the tasks at hand, expressed as C = < N C , E C > .

2.2 Circuit Discovery

To identify circuits within a language model, a key approach is to examine the model's casual mediation by systematically altering the model's edges and nodes to observe the effects on performance [32, 34, 35]. The underlying principle is that critical edges or nodes are those whose removal results in a notable decline in the model's predictive capabilities. Since the edges in the model's computational graph represent the dependencies between nodes, we can simulate the absence of a particular node-to-node dependency by ablating an edge in the graph. For example, ablating an edge from A i ′ ,j ′ to A i,j involves replacing the contribution of A i ′ ,j ′ in the input to attention head A i,j with zero (in the case of zero ablation) or with the mean value of head A i ′ ,j ′ (in the case of mean ablation). The process of identifying critical edges or nodes through ablation can be broken down into the following steps: i) Overwrite the value of the edge ( n x , n y ) with a corrupted value (either zero or mean ablation), ii) Perform a forward pass through the model with the altered graph, iii) Compare the output values of the modified model with those of the original model using a chosen metric S (Details in Eq.1 ). If the performance change is below a predefined threshold τ , we can consider the edge non-critical and remove it to obtain a new subgraph G / ( n x , n y ) . In addition to ablation-based methods, recent works have also explored the use of sparse auto-encoders [36, 37] to identify circuits within language models. This approach involves training an auto-encoder to learn a sparse representation of the model's internal structure, which can help reveal the underlying circuitry responsible for specific behaviors or functionalities.

3 Knowledge Circuits Discovery in Transformers

3.1 Knowledge Circuits Construction

Unlike previous work [12, 18], which managed to find out the specific areas that store knowledge, we pay extra heed to the information flow that activates subsequent knowledge for answering questions. Similar to [38, 26], we write language model as a graph consisting of the input, the output, attention heads, and MLPs by considering a 'residual rewrite' of the model's computational structure. For example, this residual rewrite gives us a nearly-dense graph in GPT2-medium: one between every pair of (attention head, MLP, input, and output) nodes, except for attention heads in the same layer, which do not communicate with each other. In our paper, we concentrate on the task of answering factual open-domain questions, where the goal is to predict a target entity o given a subject-relation pair ( s, r ) . A knowledge triplet k = ( s, r, o ) is often presented to the model in the form of a natural language prompt for next token prediction (e.g., 'The official language of France is ____' ). The model G is expected to generate the target entity, which is consistent with the language model's pretraining format. To identify the circuit that is critical for predicting the target entity o for a given subject-relation pair ( s, r ) , we ablate each special edge e i = ( n x , n y ) in the computation graph G . We then measure the impact of ablating the edge ( zero ablation in our implementation) on the model's performance using the MatchNLL loss [32] for the target o :

If the score S ( e i ) is less than the predefined threshold τ , we consider the edge to be non-critical and remove it from the computation graph, updating the temporary circuit C temp ←G /e i . We first sort the graph by topological rank following Conmy et al. [32] and traverse all edges in this manner, We derive a circuit C k that contributes to representing the knowledge necessary to answer the factual question:

Here, C k is the circuit for the knowledge triplet k , consisting of the nodes N k and edges E k that are essential for predicting the target entity o given the subject-relation pair ( s, r ) .

3.2 Knowledge Circuits Information Analysis

Once we have identified the knowledge circuit, we delve deeper into the specific roles and behaviors of each node and edge within the computation graph. Our goal is to comprehend the processing and contribution of each node n i to the functionality of the circuit. Drawing on the methodologies of previous studies [16, 39, 40], we begin by applying layer normalization to the output of each node n i and then map it into the embedding space. This is achieved by multiplying the layer-normalized output by the unembedding matrix ( W U ) of the language model: W U LN( n i ) . This transformation allows

us to inspect how each component writes information to the circuit and how it influences subsequent computational steps. By understanding the nodes' behavior in the circuit, we can better comprehend the circuit's structure and the key points where information is aggregated and disseminated.

3.3 Knowledge Circuits Experimental Settings

Implementations. We conduct experiments on GPT-style models, including GPT-2 medium and large. We also conduct primary experiments on TinyLLaMA [29] to validate the effectiveness of different architectures. We utilize the Automated Circuit Discovery [32] toolkit to build a circuit as an initiative of our analysis and leverage transformer lens [41] to further analyze the results. Specifically, we simply employ the MatchNLL [32] as the metric to detect the effect of the given node and edge and use zero ablation to knock out the specific computation node in the model's computation graph.

Metrics. A discovered knowledge circuit is deemed an accurate representation of a specific area within the transformer's knowledge storage, thus, it should be capable of representing the knowledge independently. Following [32], we leverage the completeness of a circuit, which refers to its ability to independently reproduce the behavior or predictions of the full model for the relevant tasks. This property is assessed by examining whether the identified subgraph corresponds to the underlying algorithm implemented by the neural network. To evaluate completeness, we first construct the circuit using the validation data D val for a specific knowledge type and then test its performance on the test split D test in isolation. By doing so, we can observe any changes in performance compared to the original model. We use the Hit@10 metric to measure the rank of the target entity o among the top 10 predicted tokens:

Here, | V | represents vocabulary size, and rank o is the rank of the target entity o in predictions.

Dataset. In this work, we focus on the knowledge that is already stored in the language model . We utilize the dataset provided by LRE [42] and consider different kinds of knowledge, including linguistic, commonsense, fact, and bias. We evaluate whether the knowledge is present in the language model's parameters under zero-shot settings using the Hit@10 metric to sample knowledge from the validation set, which is used to construct the knowledge circuit. The data statistics are in Appendix A.

4 Knowledge Circuits Unveil Implicit Neural Knowledge Representations

Knowledge Circuits Evaluation. We report the results of GPT2-Medium in Table 1, which indicates that with only less than 10% of the original knowledge circuit's subgraph, the model can

maintain over 70% of its original performance. Additionally, we compute the random circuits by randomly deciding whether the edge should be removed and making sure the graph is connected. The random circuit is the same size as the circuit we discovered using our method. From the table, we can see that the random circuit failed to maintain the model's performance, which further enhanced the robustness and efficacy of our methods. One of the most fascinating observations is the performance improvement seen on several test datasets . For instance, the Landmark-country relation metric increases from 0.16 to 0.36. This suggests that the discovered knowledge circuits may encapsulate the relevant knowledge, and the model's performance on these tasks could have been hindered by noise from other components. We proceed to analyze the layer distribution of the original model G to understand the average percentage of nodes that are activated within the circuit for different knowledge domains. From Figure 2, we observe that attention and MLPs are more active in the lower layers of the network, where the language model processes the input and extracts general information. To gain a more comprehensive view of the information processing, we compute the average rank o change of the target token in the D val across the layers and report the results in Figure 7. This analysis reveals the phenomenon of early decoding [40], suggesting that by the middle to the latest layers, the target entity is already present in the residual stream, and the subsequent layers in the Transformer are designed to increase the probability of the current token (See discussion in the running example).

0.0

0.2

0.4

0.6

0

5

10

15

20

25

Layer

Linguistic

0.0

0.2

0.4

0.6

Commonsense

0.00

0.25

0.50

0.75

circuit activation frequency

0

5

10

15

20

25

Layer

0.0

0.2

0.4

0.6

circuit activation frequency

Factual

Special Components in Knowledge Circuits. From the discovered knowledge circuits, we can find several important attention heads that demonstrate specific behavior, including the mover head [31], relation head [17, 43] and mixture head [17, 43] (more definitions in Appendix B.2). Mover Head [31, 27] focuses on the last token of the context and attends to the subject token, functioning as a mover to transfer information, while Relation Head [17] attends to the relation token in the context and produces some relation-related tokens that would guide the behavior of the following components. We think that these components would be accumulated by the MLP in the model, and the behavior of these special heads will be discussed in the running example part. We list some of these special components in Table 4 in Appendix. The different attention heads are responsible for expressing specific types of knowledge and may be activated by different facts. In our experiments with GPT-2 Medium and GPT-2 Large, we find that knowledge is distributed across several layers' attention heads and MLP matrices, suggesting that the target knowledge appears to have been accumulated throughout the GPT-2 model. Conversely, in TinyLLAMA, the special components are more concentrated. As depicted in Figure 7, the rank of the target entity in TinyLLAMA experiences a sharp decline around several layers, whereas in the GPT2 model, the decline is more gradual. We hypothesize that this discrepancy may be attributed to the model's knowledge capacity [44] and warrants further investigation.

A Running Example of Knowledge Circuit. We present a case and analyze the specific behaviors of components within the identified knowledge circuits. In particular, we find some special attention heads in the model such as the mover head and the relation head. We demonstrate the function of these heads in Figure 6 by ablating them from the circuit. Taking the factual knowledge 'The official language of France is French' as an example, we visualize the knowledge circuit in Figure 1. To express the information flow within the model more effectively, we have plotted the rank and probability of the target entity o at each layer when it is mapped into the embedding space, in Figure 3. From this figure, we can see that after MLP 17, the target knowledge emerges as the top token in the residual stream, and after that layer, it undergoes an increased probability. The edges connected to MLP17 are (L14H13 → MLP17), (L14H7 → MLP17), and (L15H0 → MLP17) . Here, the L14H13 is a relation head that focuses on the relation token in the context. The output of this head is relationrelated tokens such as 'language' and 'Language' . The attention head L14H7 is a mover head that moves the information from the subject position 'France' to the last token. Previous work [31, 19] has introduced this mover head as an argument parser , which moves 'France' to the last token, and the subsequent MLP conducts a function application to map 'France' to 'French' . An intriguing

observation is that we can find the output of this head already contains the target answer entity, which significantly contributes to the final output (L14H7 → Output). Also, we see the probability of the subject token in Figure 3 at the last token is nearly zero across these layers. Hence, instead of the argument parser function, we consider this mover head as an extract head proposed by Geva et al. [13], which aims to extract the related-information from the subject token's position. In the subsequent knowledge editing experiments, we can observe changes in the behavior of these types of heads.

Additionally, instead of extraction in the later layers proposed by Geva et al. [13], we notice a gradual decrease in rank across all early-to-middle layers. The MLP17 combines information from previous tokens and integrates this information to prioritize the target token at the top rank.

Interestingly, upon tracing the information flow to L14H7, we discovered that it is predominantly activated by L7H14, a relation head, and its output features several language tokens, such as 'Arabic' . We hypothesize that L7H14 may function as a signaling mechanism to activate the associated mover head, but this hypothesis necessitates further investigation to be confirmed. After MLP17, several attention heads, such as L18H14 (a relation head) and L20H6 (a mover head), collaborated to further enhance the final prediction of the target entity.

0

2

4

6

8

10

12

14

16

18

20

22

24

Layer

10

0

10

1

10

2

10

3

10

4

Rank (log scale)

Target Entity at Last Position

Target Entity at Subject Position

0.0

0.2

0.4

0.6

0.8

1.0

Probability

Prob. of Object Entity

Prob. of Subject Entity

5 Knowledge Circuits Elucidate Internal Mechanisms for Knowledge Editing

In this section, our objective is to evaluate the impact of previous knowledge editing methods and validate the effectiveness of knowledge circuits. We aim to understand why these methods may fail in certain cases and settings, which can also help deepen the understanding of the knowledge circuit.

Single Factual Knowledge Editing. Here, we adopt the ROME method [18] and FT-M [24], which aim to edit the MLP layers in the language model. The most important hyper-parameter in knowledge editing is the layer, as the same method's performance varies significantly via the layers. Here, we evaluate the performance of different editing layers and their effectiveness. We compare the knowledge circuits computed by the edited model with the original one, and we present results in Figure 4 and report details in Appendix D. As discussed in the previous part, the early-to-middle layers are the main part of aggregating the target entity o to the top rank. In the original model, the probability of the target entity 'Intel' is nearly zero, and the model fails to elevate it to the top rank in the vocabulary. Editing the model with ROME and FT-M both give us the correct answer but we can view different scenarios for their knowledge circuits. For ROME , as the correct information is added to the subject position, we can recognize a behavior of the Mover Head shifts from copying to extracting the edited information from the subject position . This information gradually aggregates through the subsequent layers, and by layer 15, 'Intel' emerges as the top-ranked entity with its probability increasing significantly. Specially, before editing, the mover head L15H3 attends to the 'controller' token and returns 'controller' as the output, while in the edited model, the attention head's output moves to the 'Intel' , which means the model gains the information at the subject space. For FT-M, the edited model tends to directly write the knowledge into the specific component, which would greatly dominate the following component in the model. As shown in Figure 4, the output logits in MLP-0 for 'Intel' are more than 10, and it emerges as the top rank in the residual stream directly. This phenomenon can be found in different knowledge types and layers and we report results in Appendix D.2. However, the added knowledge may have the risk of influencing unrelated knowledge. When we test another fact 'Windows server' , the model still tends to give us the 'Intel' answer, demonstrating the overfitting problem. This finding supports previous analysis regarding the correlation between localization and editing [45], suggesting that edits may not alter the storage but merely add signals into the knowledge circuits.

Multi-hop Factual Knowledge Editing. Multi-hop knowledge editing poses a challenging scenario [20, 21, 46], wherein we edit the model with new knowledge, yet the model struggles to perform reasoning using the edited information. We analyze multi-hop questions in language models [47, 48]

L15H3

Original Model

ROME-L0

Windows Server 2003

is created by

Microsoft.

MLP 0

Windows Server 2003

is created by

Intel.

FT-L0

Attention Pattern

Output logits

Case Output

L15H3

Hub is created by

Microsoft.

Output logits

Edit Case:

Other Case:

Hub is created by

Intel.

Edit Case:

Other Case:

Windows Server 2003

is created by

Intel.

Hub is created by

Intel.

Edit Case:

Other Case:

L15H3

Platform

Controller

Hub

Platform

Controller

0.1

St0

Controller

firmware

0.144

0.142

Hub

50°0

0.140

was

engineer

0.138

0.05

Hardware

0.136

created

=0.1

0.134

by

0.15

joy

0.132

Platform

Controller

created

Hub

was

by

Platform

Intel

0.90

Controller

0.2

Intel

Hub

0.1

0.88

motherboard

SeM

=0.1

IEEE

0.86

created

0.2

- 0.84

by

0.3

Hardware

Target Entity at Last Position

Prob. of Object Entity

Target Entity at Subject Position

-- Prob. of Subject Entity

0.8

Rank (log scale)

0.6

Probability

0.4

Layer

Target Entity at Last Position

Prob. of Object Entity

Target Entity at Subject Position

Prob. of Subject Entity

0.8

Rank (log scale)

0.6

Probability

0.4

0.2

Target Entity at Last Position

Prob. of Object Entity

Target Entity at Subject Position

·Prob. of Subject Entity

0.8

Rank (log scale)

0.6

Probability

0.4

0.2

Layer

awei

1.04

Intel

13.5

Samsung

1.02

Lux

13.0

Insight

12.5

Huawei

1.00

Bei

12.0

Intel

0.98

VERTISEMENT

11.5

Lenovo

= 0.96

Computing

Witness

11.0

10.5

to understand why current editing methods fail in these scenarios. For instance, given the fact (Thierry Mugle, 'home country' , France), we edit the fact to another country, such as (Thierry Mugle, 'home country' , France → China). We then assess the model's performance on questions based on the edited knowledge, including 'The official currency of the home country of Thierry Mugle is' and 'The capital city of the home country of Thierry Mugle is' . While the unedited model could correctly answer these questions, we observe that the edited model would provide the answer 'China' for subsequent hop reasoning. We find that the mover head in the original multi-hop reasoning circuit initially extracts the second-hop answer but, after editing, extracts 'China' , demonstrating that the edited information dominantly saturates and influences the circuit. Furthermore, we observe an intriguing phenomenon: even in the original model's multi-hop reasoning settings, it would directly provide the answer if we remove the context of the first-hop texts (Details in Appendix C.1). This further confirms the findings that the model relies on relational and subject-related information, regardless of grammatical adherence.

6 Knowledge Circuits Facilitate Interpreting Language Model Behaviors

In this Section, our aim is to validate whether the identified knowledge circuits are actually utilized by the model when it employs knowledge. To address this, as shown in Figure 5, we investigate three phenomena: hallucination, in-context learning, and reverse relations (Details in Appendix C.3).

Factual Hallucination. If the knowledge is stored and expressed by the circuit we discovered, we aim to discover what happened when the model gave us the incorrect answer. We focus on factual hallucinations, which occur when the model provides an incorrect target entity for a given subject s and relation r . In our experiments (Figure 5 and Appendix C.2), we observe that the model fails to move the correct knowledge to the final token in the earlier layers. This failure is evident as the

The

comparative

form

of

small

is

smaller

.

The

comparative

form

of

big

is

The

comparative

form

of

small

is

smaller

.

The

comparative

form

of

big

is

-0.5

0

0.5

Loading [MathJax]/extensions/MathMenu.js

L15H0

Attention Pattern

Output logits

Embed

Residual Output

L17H2

New attention head

Original circuit

L12H13

L14H11

L12H13

Attention Pattern

Output logits

The

comparative

form

of

small

is

smaller

.

The

comparative

form

of

big

is

The

comparative

form

of

small

is

smaller

.

The

comparative

form

of

big

is

-0.5

0

0.5

Loading [MathJax]/extensions/MathMenu.js

(a)  A hallucination Case

(b)  An in-context learning Case

The official currency of Malaysia is called the

Correct Entity

Prob.of Correct Entity

Wrong Entity

Prob. of Wrong Entity

0.8

10°

Rank (log scale)

9′0

10°

Probability

0.4

10²

0.2

10

0.0

offi

icial

curr

ency

Malay

Of

ysia

1415161718192021222324

The

Malays

official

Kuala

currency

Malaysia

1.7

Malaysian

of

Singapore

1.6

Malaysia

Brune

is

Indonesia

1.5

S'0-

Jakarta

called

Cheong

1.4

the

RM

0.54

lighter

0.53

cleaner

0.52

sharper

cooler

0.51

0.50

shorter

0.49

happier

- 0.48

fewer

0.47

circuit lacks an effective mover head or the mover head selects incorrect information. For instance, in the prompt 'The official currency of Malaysia is called' , both the correct answer 'Ringgit' and the incorrect one 'Malaysian' are accumulated before layer 15. However, at layer 16, the mover head L15H10 extracts the erroneous information. Despite a rank drop of the true one in layers 20-22, this is insufficient to correct the previous mistake.

In-Context Learning. Despite storing a vast amount of knowledge, a language model may still provide incorrect answers. However, with demonstrations or examples (based on RAG [49]), it can quickly generate correct responses. To this end, we focus on the scenario where the model initially provides an incorrect answer but can then produce the correct response upon receiving the appropriate demonstration. We consider the original knowledge circuit and introduce a new knowledge circuit based on the demonstration. Our analysis reveals that, compared to the zero-shot knowledge circuit, several new attention heads appear in the computation graph when the demonstration is incorporated. We show the behavior of these attention heads in Figure 5. We can see these heads mainly focus on the demonstration's context: 'The comparative form of small is smaller' and works as the Induction Head [50] that look back over the sequence for previous instances of the current token and find the token that came after it last time. To better view the function of these heads, we conduct experiments by ablating the newly appeared attention head in the ICL circuit in Table 2. We find that compared to the randomly selected attention head by ablating this attention, the probability drops significantly in the prediction, demonstrating the importance of these identified attention heads. These aligned with previous work where Todd et al. [51] have identified a concept known as the Function Vector, which represents the average of some key attention heads and provides the task learning ability.

7 Related Work

Knowledge Mechanism of Transformers. How the language model stores and utilizes knowledge is an ongoing research topic. Previous works find that the MLP in Transformers works as a keyvalue memory and stores enormous knowledge [12, 15, 14, 18]. As to the relation between entities, Hernandez et al. [42] observes that facts can be decoded linearly from the enriched residual stream of the subject by mapping the subject entity to the object entity. Instead of viewing the knowledge storage in isolation, Geva et al. [13], Lv et al. [31], Yu and Ananiadou [16] find the knowledge is accumulated during the layers. Regarding knowledge analysis, Bayazit et al. [52] also attempts to discover critical knowledge in language models. However, they only consider several layers in the

model and use the pruning method, which may overlook the connections between components. More related works can be found in Appendix E.1.

Manipulate Language Models. Recently, many works aim to manipulate the language models to make the model aligned with world knowledge or social value norms, such as knowledge editing [20, 24], machine unlearning [53, 54] and detoxification [55, 56]. Most of these works are elicited by previous knowledge mechanism findings such as knowledge neuron [57]. They modify the MLP in the LLM [18, 12] to change the model's behavior based on specific factual knowledge. However, recent works [58, 59] demonstrate the pivotal role of the attention part in knowledge representation. Hase et al. [45] also observe that the performance of editing within a layer may not reliably pinpoint the location of the fact. In this paper, we try to manipulate specific knowledge of language models via knowledge circuits, including both MLP and attention components across different layers.

8 Conclusion

In this paper, we present a new perspective on knowledge storage based on circuit theory and conduct a preliminary analysis to demonstrate its effectiveness. We found that knowledge circuits in the model are not only responsible for expressing knowledge but can also guide behavior in different settings. We hope these findings can advance our understanding of the knowledge mechanisms of language models and provide insights for better designing and editing language models, enhancing knowledge, and improving reasoning to enhance factuality and alleviate hallucinations.

Limitations and Broader Impacts

In this work, we employ the causal mediation method to automatically construct circuits tailored to specific knowledge domains. However, this circuit discovery-based patching approach is timeintensive. Contemporary research efforts have introduced more efficient methodologies for modeling information flow [60-62]. Additionally, alternative techniques for discovering circuits through mask training [63, 64] and Sparse Auto-Encoders [65, 36] have been proposed, highlighting diverse facets of circuit behavior within large language models (LLMs). We posit that the field of knowledge circuit discovery holds significant potential for advancement. Furthermore, recent studies [66] have developed 'circuit breakers' to manage representations associated with potentially harmful outputs. We hope that our approach can contribute to ensuring the safety and privacy of information, thereby fostering the development of trustworthy AI.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Yongjiang Talent Introduction Programme (2021A-156-G), CIPSC-SMP-Zhipu Large Model Cross-Disciplinary Fund, Ningbo Science and Technology Special Projects under Grant No. 2023Z212, Information Technology Center and State Key Lab of CAD&CG, Zhejiang University, NUS-NCS Joint Laboratory (A-0008542-00-00), and the Ministry of Education, Singapore, under the Academic Research Fund Tier 1 (FY2023) (Grant A-8001996-00-00). We gratefully acknowledge the support of Zhejiang University Education Foundation Qizhen Scholar Foundation.

References

Francis bacon. https://iep.utm.edu/francis-bacon/ .

Francis Bacon. The advancement of learning [1605]. In Primer of intellectual freedom , pages 172-192. Harvard University Press, 1949.

OpenAI and the Co-authors. Gpt-4 technical report, 2024.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR , abs/2303.18223, 2023. doi: 10.48550/ARXIV.2303.18223. URL https://doi.org/10.48550/arXiv.2303.18223 .

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023.

Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, and Yue Zhang. Survey on factuality in large language models: Knowledge, retrieval and domain-specificity, 2023.

Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Unified hallucination detection for multimodal large language models. CoRR , abs/2402.03190, 2024. doi: 10.48550/ARXIV.2402.03190. URL https://doi.org/10.48550/arXiv.2402.03190 .

Helena Bonaldi, Yi-Ling Chung, Gavin Abercrombie, and Marco Guerini. Nlp for counterspeech against hate: A survey and how-to guide, 2024.

Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang, John Mitchell, Kai Shu, Kaidi Xu, Kai-Wei Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, and Yue Zhao. Trustllm: Trustworthiness in large language models. CoRR , abs/2401.05561, 2024. doi: 10.48550/ARXIV.2401.05561. URL https://doi.org/10.48550/arXiv.2401.05561 .

Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models with multiple choice questions. CoRR , abs/2309.07045, 2023. doi: 10.48550/ARXIV. 2309.07045. URL https://doi.org/10.48550/arXiv.2309.07045 .

Aiqi Jiang and Arkaitz Zubiaga. Cross-lingual offensive language detection: A systematic review of datasets, transfer approaches and challenges, 2024.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.581.

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12216-12235, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.751.

Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 30-45, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.emnlp-main.3.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wentau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5484-5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.446.

Zeping Yu and Sophia Ananiadou. Locating factual knowledge in large language models: Exploring the residual stream and analyzing subvalues in vocabulary space, 2024.

Bilal Chughtai, Alan Cooney, and Neel Nanda. Summing up the facts: Additive mechanisms behind factual recall in llms, 2024.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems , 36, 2022.

Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic, 2024.

Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. Editing large language models: Problems, methods, and opportunities. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 10222-10240, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.632.

Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the Ripple Effects of Knowledge Editing in Language Models. Transactions of the Association for Computational Linguistics , 12:283-298, 04 2024. ISSN 2307-387X. doi: 10.1162/tacl_a_00644. URL https://doi.org/10.1162/tacl_a_00644 .

Belinda Hopkins. Restorative theory in practice: Insights into what works and why . Jessica Kingsley Publishers, 2015.

Jingcheng Niu, Andrew Liu, Zining Zhu, and Gerald Penn. What does the knowledge neuron thesis have to do with knowledge? In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=2HJRwwbV3G .

Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. A comprehensive study of knowledge editing for large language models. arXiv preprint arXiv:2401.01286 , 2024.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations , 2023. URL https: //openreview.net/forum?id=NpsVSN6o4ul .

Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=fpoAYV6Wsk .

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill , 2020. doi: 10.23915/distill.00024.001. https://distill.pub/2020/circuits/zoom-in.

Ang Lv, Kaiyi Zhang, Yuhan Chen, Yulong Wang, Lifeng Liu, Ji-Rong Wen, Jian Xie, and Rui Yan. Interpreting key mechanisms of factual recall in transformer-based language models, 2024.

Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 16318-16352. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/ file/34e1dbe95d34d7ebaf99b9bcaeb5b2be-Paper-Conference.pdf .

Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety - a review, 2024.

Judea Pearl. Direct and indirect effects. In Probabilistic and causal inference: the works of Judea Pearl , pages 373-392. 2022.

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems , 33:12388-12401, 2020.

Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt, 2024.

Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600 , 2023.

Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023.

Shahar Katz and Yonatan Belinkov. VISIT: Visualizing and interpreting the semantic information flow of transformers. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 14094-14113, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-emnlp.939. URL https://aclanthology.org/2023.findings-emnlp.939 .

nostalgebraist. interpreting GPT: the logit lens. 2020. URL https://www.lesswrong.com/ posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens .

Neel Nanda and Joseph Bloom. Transformerlens. https://github.com/neelnanda-io/ TransformerLens , 2022.

Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Linearity of relation decoding in transformer language models. In Proceedings of the 2024 International Conference on Learning Representations , 2024.

Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R. Costa-jussà. A primer on the inner workings of transformer-based language models, 2024.

Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws. 2024.

Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id=EldbUlZtbd .

Zexuan Zhong, Zhengxuan Wu, Christopher Manning, Christopher Potts, and Danqi Chen. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 15686-15702, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.971.

Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning?, 2024.

Tianjie Ju, Yijin Chen, Xinwei Yuan, Zhuosheng Zhang, Wei Du, Yubin Zheng, and Gongshen Liu. Investigating multi-hop factual shortcuts in knowledge editing of large language models, 2024.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. CoRR , abs/2312.10997, 2023. doi: 10.48550/ARXIV.2312.10997. URL https://doi.org/10.48550/arXiv.2312.10997 .

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-inductionheads/index.html.

Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C Wallace, and David Bau. LLMs represent contextual tasks as compact function vectors. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id= AwyxtyMwaG .

Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. Discovering knowledge-critical subnetworks in pretrained language models, 2023.

Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for llms: Tasks, methods, and challenges, 2023.

Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12041-12052, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.738. URL https://aclanthology.org/2023.emnlp-main.738 .

Xinshuo Hu, Dongfang Li, Baotian Hu, Zihao Zheng, Zhenyu Liu, and Min Zhang. Separate the wheat from the chaff: Model deficiency unlearning via parameter-efficient module operation. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, ThirtyEighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada , pages 18252-18260. AAAI Press, 2024. doi: 10.1609/AAAI.V38I16.29784. URL https://doi.org/10.1609/aaai.v38i16.29784 .

Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing, 2024.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 8493-8502. Association for Computational Linguistics, 2022.

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inferencetime intervention: Eliciting truthful answers from a language model. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/ forum?id=aLLuYpn83y .

Mansi Sakarvadia, Aswathy Ajith, Arham Khan, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Memory injections: Correcting multi-hop reasoning failures during inference in transformer-based language models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages 342-356, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.blackboxnlp-1.26.

Javier Ferrando and Elena Voita. Information flow routes: Automatically interpreting language models at scale. arXiv preprint arXiv:2403.00824 , 2024.

Aaquib Syed, Can Rager, and Arthur Conmy. Attribution patching outperforms automated circuit discovery. In NeurIPS Workshop on Attributing Model Behavior at Scale , 2023. URL https://openreview.net/forum?id=tiLbFR4bJW .

Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling , 2024. URL https://openreview.net/forum?id=TZ0CCGDcuT .

Adithya Bhaskar, Alexander Wettig, Dan Friedman, and Danqi Chen. Finding transformer circuits with edge pruning. arXiv preprint arXiv:2406.16778 , 2024.

Lei Yu, Jingcheng Niu, Zining Zhu, and Gerald Penn. Functional faithfulness in the wild: Circuit discovery with differentiable computation graph pruning. arXiv preprint arXiv:2407.03779 , 2024.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , 2023. https://transformercircuits.pub/2023/monosemantic-features/index.html.

Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with short circuiting. arXiv preprint arXiv:2406.04313 , 2024.

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 4895-4901, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. emnlp-main.298. URL https://aclanthology.org/2023.emnlp-main.298 .

Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, and Junxian He. In-context sharpness as alerts: An inner representation perspective for hallucination mitigation, 2024.

Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https: //openreview.net/forum?id=Th6NyL07na .

Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710 , 2024.

Beren and Sid Black. The singular value decompositions of transformer weight matrices are highly interpretable. https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/ the-singular-value-decompositions-of-transformer-weight , 2022.

Junlin Zhang. Parametric reflection of the world: Why can gpt generate intelligence through next token prediction. https://zhuanlan.zhihu.com/p/632795115 , 2023.

Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: LLMs trained on 'a is b' fail to learn 'b is a'. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=GPKTIktA0k .

Peng Wang, Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, et al. Easyedit: An easy-to-use knowledge editing framework for large language models. arXiv preprint arXiv:2308.07269 , 2023.

Maximilian Li, Xander Davies, and Max Nadeau. Circuit breaking: Removing model behaviors with targeted ablation. Workshop on Challenges in Deployable Generative AI at International Conference on Machine Learning , 2023.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 5484-5495. Association for Computational Linguistics, 2021. doi: 10.18653/V1/2021.EMNLP-MAIN.446. URL https://doi.org/10.18653/v1/2021. emnlp-main.446 .

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 30-45. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.EMNLP-MAIN.3. URL https: //doi.org/10.18653/v1/2022.emnlp-main.3 .

Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada , pages 17817-17825. AAAI Press, 2024. doi: 10.1609/AAAI.V38I16.29735. URL https://doi.org/10.1609/aaai.v38i16.29735 .

Yuheng Chen, Pengfei Cao, Yubo Chen, Yining Wang, Shengping Liu, Kang Liu, and Jun Zhao. The da vinci code of large pre-trained language models: Deciphering degenerate knowledge neurons. CoRR , abs/2402.13731, 2024. doi: 10.48550/ARXIV.2402.13731. URL https://doi.org/10.48550/arXiv.2402.13731 .

Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. Massediting memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023. URL https://openreview.net/pdf?id=MkbcAHIYgyS .

Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality, 2024.

Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. CoRR , abs/2402.18154, 2024. doi: 10.48550/ARXIV.2402.18154. URL https://doi.org/10.48550/arXiv.2402.18154 .

Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. CoRR , abs/2310.15213, 2023. doi: 10.48550/ ARXIV.2310.15213. URL https://doi.org/10.48550/arXiv.2310.15213 .

Subhabrata Dutta, Joykirat Singh, Soumen Chakrabarti, and Tanmoy Chakraborty. How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning. CoRR , abs/2402.18312, 2024. doi: 10.48550/ARXIV.2402.18312. URL https://doi.org/10. 48550/arXiv.2402.18312 .

nostalgebraist. interpreting gpt: the logit lens. https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens , 2020.

Mert Yuksekgonul, Varun Chandrasekaran, Erik Jones, Suriya Gunasekar, Ranjita Naik, Hamid Palangi, Ece Kamar, and Besmira Nushi. Attention satisfies: A constraint-satisfaction lens on factual errors of language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=gfFVATffPd .

Fahim Dalvi, Hassan Sajjad, and Nadir Durrani. Neurox library for neuron analysis of deep NLP models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL 2023, Toronto, Canada, July 10-12, 2023 , pages 226-234. Association for Computational Linguistics, 2023.

Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders. Transformer debugger. https://github.com/openai/transformer-debugger , 2024.

Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models, 2024.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. CoRR , abs/2209.10652, 2022. doi: 10.48550/ARXIV.2209.10652. URL https://doi.org/10.48550/arXiv.2209.10652 .

A Implementation Details

Hyper-parameter. The primary hyperparameter for constructing a circuit is the threshold τ used to detect performance drops. Setting τ too high may result in an incomplete circuit, while setting it too low can introduce numerous unnecessary nodes. In our experiment, we test τ values from the set {0.02, 0.01, 0.005} to determine the appropriate circuit size for different types of knowledge.

Implementation. We utilize ACDC 2 to construct circuits that encode specific knowledge representations. Additionally, we re-implement the code to compute the relevant dataset and impact metrics for the knowledge used. Since TinyLLAMA 3 incorporates the Grouped Query Attention Mechanism [67], we interleave and repeat the key and value pairs to analyze the specific behavior of each attention head. We use the NVIDIA-A800 (40GB) to conduct our experiments. It took about 1-2 days to compute the circuit for the knowledge type in GPT2-medium.

Dataset Details. All the data used in our paper is sourced from Hernandez et al. [42], with the detailed information provided in Table 6. In the original setting, they use the data for few-shot settings, but in our experiments, we consider zero-shot knowledge storage, so here we sample the data using the Hit@10 to detect whether the model understands knowledge for the given prompt based on the s and o . We sample the test

set in a 1:1 ratio with the validation set to ensure a balanced evaluation.

B More Experiment Results

B.1 Rank Change Across Layers

To gain a clearer understanding of the knowledge aggregation phenomenon, we compute the rank of the target entity rank o within the vocabulary space | V | at the output of each layer. As depicted in Figure 7, we observe that the model initially elevates the target entity to the top ranks of the vocabulary. Once the entity reaches the top of the vocabulary, subsequent layers continue to enhance its probability mass. These findings corroborate previous work by [68-70], who note the substantial difference in logit entropy between layers, contributing to the model's improved prediction for the target entity. In our work, we aim to delve deeper into how the model, as well as specific components within it, give rise to these behaviors.

B.2 Special Components in Knowledge Circuit

When zooming into the discovered circuit, we can find several kinds of special attention heads, or MLPs, in the model that play a pivotal role in the final prediction, similar to what previous research has indicated [31]. Apart from mover head and relation head, there is another kind of head named Mix Head [17, 43] which would focus on both the relation token and the subject tokens. In our experiments, we found these heads usually work similarly to the mover head. In particular, mover heads contribute more to the subject-specific information, as ablating them would increase other relation-related tokens' probability. Moreover, if we ablate the relation head, we can find the model tends to generate some meaningless tokens instead of the relation information. From Figure 3 6, we can find ablating the mover head would increase the probability of "Italian", "English" and "Spanish", which are not subject-related. While ablating the relation head would lead to the increase of some meaningless words "a", and "that", which are not relation-related.

Lv et al. [31] posits that these mover heads are responsible for extracting the 'argument' from the context and passing it for further processing, such as function application. Consequently, Lv et al. [31], Merullo et al. [19] suggests that the MLP within the language model performs a function

Appendix

(a) original output

(b) ablating mover head

(c) ablating relation head

French

English

0.7

the

0.6

Spanish

0.5

spoken

0.4

Latin

0.3

e

0.2

Italian

-0.1

Franc

F

French

English

0.35

Spanish

0.30

the

0.25

Italian

0.20

Portuguese

0.15

F

0.10

Catalan

0.05

II

German

French

the

0.16

II

0.14

0.12

a

0.10

not

0.08

France

English

0.06

0.04

to

-0.02

now

that

1

6

11

16

21

10

0

10

1

10

2

10

3

10

4

GPT2-Medium

name_birthplace

1

6

11

16

21

adj_antonym

1

6

11

16

21

work_location

1

6

11

16

21

landmark_in_country

1

6

11

16

21

product_by_company

1

6

11

16

21

person_plays_instrument

1

7

13

19

25

31

10

0

10

1

10

2

10

3

10

4

GPT2-Large

1

7

13

19

25

31

1

7

13

19

25

31

1

7

13

19

25

31

1

7

13

19

25

31

1

7

13

19

25

31

1

5

9

13

17

21

Layer

10

2

10

3

10

4

TinyLLAMA

1

5

9

13

17

21

Layer

1

5

9

13

17

21

Layer

1

5

9

13

17

21

Layer

1

5

9

13

17

21

Layer

1

5

9

13

17

21

Layer

Rank

application that transforms the subject into an object, with the subject's probability ranking higher than the object's. However, these findings are limited to specific cases, as the studies only examine two related tasks, such as capital city identification or color objection. Our analysis suggests that these conclusions may not be universally applicable to all knowledge domains and require further investigation. When we examine the ranks of subject and object entities, we rarely observe an overwhelming superiority at the last token position of the subject token and usually, the probability of the subject token at the last position is uniformly low. In our experiment, we view the model as a collaboration between different nodes in the identified circuit. They perform as different kinds of

Attention Pattern

Output logits

Residual Output: French

L15H11

L17H19

L9H3

MLP0-2

Attention Pattern

Output logits

(a)  A simplified Knowledge Circuit

(b)  Some Component's output

(c) The rank and probability of the entity in the residual stream

Input Embed

Output logits

L9H3

L15H11

MLP16

MLP13

MLP9

Attention Pattern

L17H19

mother

tongue

mother

tongue

The

046

The

The

English

mother

tongue

of

France

French

0.62

0.60

mother

tongue

of

MaTepu

0.105

=0.104

Th

ier

France

0.58

0.56

Th

ier

ra

= 0.102

0.103

ry

M

0.52

0.54

A

M

ammer

0.101

6n

0.50

6n

yiOHanb

0.100

le

le

is

is

mother

tongue

The

The

Target Entity at Last Position

Target Entity a Sebject Position

Prob. of Subjet Entity

Prob. of Object Entity

1.0

mother

tongue

French

0.700

0.8

of

France

0.675

Rank (log scale)

Th

0.650

ler

France

0.625

0

ry

Paris

0.600

6n

M

le

franc

= 0.575

0.550

10

0.2

is

Layer

17

21

22

0.0

attention heads, like mover head and relation head . We list some heads that are responsible for the storage of different kinds of knowledge and relations in Table 4.

Component Reuse Phenomenon. Merullo et al. [27] have identified shared circuits for the IOI task and the Colored Objects task. We also observe this phenomenon in the factual recall task. As depicted in Table 4, we can observe that for related relations such as 'city_in_country', 'name_birth_place', and 'country_language', their circuits include both L21H12, which stores and maps country-related information. Additionally, we found that some relation heads are activated by different relations. For instance, in our experiments, the head 'L7H14' appears in the circuits of both 'official_language' and 'official_currency'. We speculate that these reused heads, rather than task-specific heads, can be considered topic heads, as proposed by [71, 60]. We believe that further investigation into this distinction is warranted in future research.

B.3 A Case on TinyLLAMA

In this part, we demonstrate one circuit we found in the TinyLAMA in Figure 8. Actually, in TinyLLAMA, the attention heads bearing specific behaviors in the later layers is usually less than

<m22>

<resid_post>

<a20.6>

<m17>

<m19>

<m16>

<a15.0>

<a14.7>

<m14>

<m13>

<m20>

<a7.14>

<m18>

<m9>

<m8>

<m12>

<a18.14>

<m0>

<m11>

<m4>

<m2>

<m3>

<m1>

<m5>

<m7>

<a1.0>

<a14.13>

<a5.15>

<a3.4>

<a2.9>

<a2.3>

<a2.1>

<a1.15>

<a1.12>

<a1.8>

<a1.4>

<a1.1>

<a11.9>

<a10.11>

<a2.2>

<a9.13>

<a6.5>

<a3.9>

<a2.0>

<a8.12>

<a6.1>

<a1.6>

<a6.10>

<a5.13>

<a4.1>

<a4.4>

<a1.14>

<a1.11>

<a1.10>

<a2.11>

<a2.7>

<a1.13>

<a1.9>

<a1.2>

<a13.7>

<m6>

embed

<a1.5>

<a0.14>

<a0.13>

<a0.11>

<a0.10>

<a0.9>

<a0.8>

<a0.7>

<a0.5>

<a0.4>

<a0.2>

GPT2. We can also view some mover heads and relation heads in the circuit that would generate the target token as the output, such as L15H3 and L17H19.

C More Circuit Utilization Analysis

While previous work on knowledge storage suggests that knowledge may be localized in specific areas of the model, it is essential to ascertain whether the model actively employs this knowledge when encountering related contexts or if it relies on shortcuts.

C.1 Multi-hop Factual Knowledge Editing

We consider scenarios where the model makes a correct prediction for all the multi-hop questions and single-hop questions. We found a circuit reuse phenomenon in the one-hop and multi-hop knowledge circuits. Here, we first compute the proportion of the nodes N single in the single-hop circuit C single that appears in N multiple the set of nodes in the multi-hop circuit C multiple .

Actually, there are two ways for the model to conduct multi-hop reasoning. As shown in the figure, the model can also answer the question by combining the two hop relations together (in the given case, combine 'hometown' and 'language' as 'mother tongue') If the model is capable of combining two-hop relations in a more integrated or semantic way, such as inferring that the 'mother tongue' is the language of one's hometown, this suggests a more complex reasoning process that goes beyond the simple overlap of nodes and edges. To capture this kind of reasoning, we assess the model's ability to integrate information from different hops. R integrated as the set of integrated relations (new paths created by combining information from different hops)

We observe an overlap of these circuit nodes, indicating that the language model utilizes a large portion of the nodes in the original single hop's circuit, especially the mover head. From the overlap analysis, it seems the model utilizes single-hop information to conduct reasoning. We discover an intriguing phenomenon in GPT-

2 and TinyLLAMA: the models can correctly answer first-hop knowledge and perform multi-hop reasoning based on it. However, interestingly, when we delete the first-hop knowledge while retaining only the second-hop relation and the first-hop subject, the models can still correctly answer the multi-hop question. Figure 10 illustrates a specific case in our findings. It further enhances our previous findings that the model actually conducts the factual recall with the relation head and the gathered information about the subject. Moreover, we found this phenomenon is hugely alleviated by the aligned model, which requires further investigation in the future.

Grammatical errors

The home country of the Thierry Mulger is

France

The official currency of the home country of

the Thierry Mulger is the

Franc

the

Franc

Without first-hop

factual knowledge

Multi-hop knowledge

Corrupted knowledge

C.2 Hallucination

The results are presented in Figure 5. We observe an interesting phenomenon: the correct answer and the wrong answer are both accumulated at the previous layer, but at some specific layers, the wrong answer is selected as the answer to extract. We hypothesize that there may be a circuit competition here proposed by [72] and we detect the behavior of the specific component between them.

C.3 Reverse Relation

Reverse Curse [73] is an important issue in language models when models successfully give us the correct answer o for ( s, r ) , but with a reverse relation ˆ r and o , the models fail to give us the correct subject s . In this part, we endeavor to investigate how the language model manages the reverse relation when they successfully store the knowledge. We first sample facts where the language model successfully predicts the given knowledge and the reversed fact. Then, we compute the overlap between these two circuits, C d and C r under node levels based on equation 4. We select the 'superhero_person' relation to see the difference between these two circuits and test the node overlap of the two circuits in the model. We notice that the overlap between the two circuits is about 70%, indicating the language model may store the related information in the same place. We also found the activated mover heads for the two relationships to be identical.

D Edit Experiments

D.1 Method Implementation

ROME As proposed by Meng et al. [18], ROME views knowledge editing as a minimal optimization problem. ROME regards the MLP module as a simple key-value store. Specifically, the key represents a subject and the value encapsulates knowledge about that subject, the MLP can reestablish the association by retrieving the corresponding value for the key. To add a new key-value pair, ROME applies a rank-one modification to the MLP's weights, effectively 'writing in' the new information directly. This method enables more direct and precise modification of the model's knowledge. The ROME method for model editing was conducted based on the EasyEdit[74] framework, utilizing the default parameters provided by EasyEdit. The experiments are performed on an A800 80G GPU, with approximately 8GB of memory consumption.

FT-M For Fine-Tuning (FT-M), we follow Zhang et al. [24]. It trains the MLP layer using the crossentropy loss on the target answer while masking the original text. This approach aligns more closely with the traditional fine-tuning object. The FT-M method is conducted using the EasyEdit 4 [74] framework, with the default parameters provided by EasyEdit. The experiments are also performed on an A800 80G GPU, with a memory consumption of approximately 10GB.

D.2 Edit Cases on FT-M and ROME

When a circuit is established for a particular piece of knowledge, we can manipulate the model's computation by targeting critical points within the circuit. Li et al. [75] ablates a small number of important causal pathways by masking the edges in the circuit and making the model less toxic and safer, which proves the effectiveness of the circuit. As illustrated in figure 12 and figure 13, we present the changes in the ranking of the predicted probabilities for the target new token when editing layer 6, 12, and 18 of the GPT-2 medium model using FT-M and ROME methods. When applying FT-M for model editing, it is evident that the rank of the target new token's probability sharply

(a) Edit Layer 0 with FT

(b) Edit Layer 6 with FT

(c) Edit Layer 12 with FT

(d) Edit Layer 23 with FT

embed

<a0.7>

<mO

<a1.5>

<m1>

<m12>

<m13>

<resid_post>

embed

<a0.13>

<a0.11>

<a0.7>

<a0.2>

<a0.0>

embed

<m0>

<a0.8>

<a0.7>

<a0.4>

<a0.2>

<a0.1>

<82.14>

<82.5>

<a1.14>

<a1.6>

<a1.0

<m0x

<m6>

<resid_post>

<m8>

<m7>

<resid_post>

ca0.13s

<80.11s

80.10

a0.7s

002

4

ca0.15

<80.16s

87.9-

a6.15=

"83.4s

"al.1ts

sm15=

cm13

<a9.3

mfts

al.1

caf1.4

<87.2s

<a7.2>

23

City-in-Country

Country-Language

Landmark-in-Country

Person-Father

Star-Constellation

Superhero-Person

10

104

104

104

10

104

10

10

103

10

103

103

102

102

102

10

102

10

10

10

10

10

10

100

10

15

20

100

10

15

100

10

100

100

100

10

10

104

104

104

104

0

103

103

103

103

103

10²

102

10²

102

10²

102

10

101

10

10

10

10

100

10

15

20

100

15

100

10

15

100

15

100

10

15

20

100

20

15

104

104

104

104

104

10²

103

103

103

10

103

102

102

102

102

10²

102

10

10

10

10

10

101

100

10

15

20

100

10

15

20

10

15

20

100

10

15

20

100

10

15

20

5

10

15

20

Layer

Layer

Layer

Layer

Layer

Layer

declines at the corresponding edited layer, resulting in a vertical line in the figure. This indicates that FT-M directly embeds the editing information into the model's information flow. Conversely, when using the ROME method for editing, this effect is mitigated. The predicted probability of the target new token reaches its peak only a few layers after the edited layer. This observation is consistent with our previous analysis in Section 5.

E More related Work and Discussion

E.1 More Related Work

Knowledge in MLP Geva et al. [76] claim that MLP serves as key-value memories for knowledge in LMs. Geva et al. [77] further propose the knowledge neuron theory, suggesting that the key and

City-in-Country

Country-Language

Landmark-in-Country

Person-Father

Star-Constellation

Superhero-Person

104

104

104

104

104

103

103

103

10

103

103

Layer

102

102

102

102

102

101

10

10

10

0T

10

15

20

100

0

5

10

20

100

5

15

20

100

10

20

100

5101520

100

0

104

104

104

104

104

104

Layer 12

103

103

103

103

103

10

102

102

10²

102

102

101

101

101

10

10

101

10

15

20

100

10

15

20

10°

10

20

100

10

0

20

100

10

15

104

104

104

104

103

103

103

103

103

103

102

102

102

102

10²

102

101

10

10

10

10

10

10

Layer

15

20

100

10

Layer

20

口

10

Layer

15

20

100

10

Layer

15

20

100

0

10

Layer

15

20

100

10

Layer

20

value vectors in MLPs encode factual knowledge. Based on the above findings, Chen et al. [78] observe that multiple distinct sets of KNs can store identical facts. Chen et al. [79] explore the structural and functional among neurons by neurological topology clustering method. Meng et al. [18] and Meng et al. [80] confirm that MLP modules do store factual knowledge and pioneering use of knowledge editing methods to modify outdated knowledge stored in language models. Anthropic recently introduces scaling monosemanticity 5 , which extracts highly abstract features that respond to and behaviorally cause abstract behaviors.

Knowledge in Attention Heads Li et al. [58] reveal that some attention heads are capable of truthful answers. Wu et al. [81] investigate 4 model families, 6 model scales, and 3 types of finetuning, and find retrieval heads, which are responsible for retrieving relevant information from long context. Jin et al. [82] suggest that memory heads can retrieve knowledge from internal memory, while context heads can recall knowledge from external context. Todd et al. [83] use causal mediation analysis on a diverse range of in-context-learning and find some attention heads, dubbed function vectors, which trigger the ability of in-context-learning.

Knowledge in Hybrid Components Recent works emphasize the importance of connections of components among language models for knowledge representation and utilization. Geva et al. [13] describe factual recall by the following three steps: (1) subject enrichment in MLP sublayers, akin to ROME[18], (2) propagation of relations to the END token, and (3) selective extraction of attributes by attention heads in later layers. Lv et al. [31] apply projection and intervention to explore mechanisms in factual recalls tasks and conclude that task-specific attention head may move the topic entity to the final position of the residual stream, while MLP conducts relation function.

Circuit Circuit discovery also plays a significant role in analyzing the internal mechanisms of the entire model [25]. Specifically, a circuit, comprising components such as MLP and attention, is a subgraph of the computation graph. Conmy et al. [32] design an automated circuit discovery approach that implements the specified behavior. Wang et al. [26] explain the circuits for the Indirect Object Identification (IOI) task. They use causal interventions to discover circuits responsible for the flow of information. Instead the above task-specific circuit, Merullo et al. [27] presents evidence a circuit is shared by similar tasks in IOI and Color Object (CO) tasks. Dutta et al. [84] construct circuits using attention heads, and further observe that attention heads are pivotal to chain-of-thought reasoning, i.e., attention heads that move information along ontological relations exclusively appear in the initial half of the layers, while the tokens responsible for writing the answer predominantly appear in the later half of the layers. However, the above-mentioned circuit studies either focus solely

on a single component (MLP or attention) or only explore IOI and CO tasks. IOI and CO tasks necessitate the model to search the preceding context for a matching token and then copy it into the next token prediction. Also, despite the success of previous circuit discovery, we can hardly make it into real usage. Hence, in this work, we attempt to analyze a knowledge circuit consisting of both MLP and attention components and investigate the effect of current editing methods on the circuit to shed light on the future.

Tools There are also many tools that are designed to analyze the LM's behavior, such as Logit Lens [85], Attention len [86], Attribution lens [42] and transformer-lens [41]. NeuroX [87] implements various interpretation methods under a unified API and provides insights into how knowledge is structured in representations and discovers the role of neurons in LM. Transformer Debugger [88] is an interpretability tool provided by OpenAI, which deploys the GPT-4 and sparse auto-encoder to explain the language neurons and attention head. PatchScope [89] is a tool provided by Google that uses a new model to explain the hidden states in the original model.

E.2 Limitation and Future Discussion

Despite of the attempt to combine the attention head and MLP to view the knowledge storage as a whole, this work operates with a relatively coarse granularity of circuits. For instance, the neurons within an MLP may necessitate a finer level of granularity to fully capture their behavior and contributions. Even though we now know these components work together to express the knowledge, why they are activated is still opaque. Our methodology employs the logit lens as a means to detect and analyze component information. However, this approach may encounter discrepancies between the middle layers and the output unembedding matrix. Such discrepancies can hinder a comprehensive and concrete analysis of the circuit components' behavior in the early layers. This limitation suggests the need for more robust techniques to bridge the gap between intermediate representations and final outputs. Recently, the Attention Lens method [86] has been proposed, which involves training a specific unembedding matrix to map each attention head into the vocabulary space. While this method is promising, it is also resource-intensive. Nevertheless, it represents a potential starting point for a deeper understanding of the knowledge circuits within neural models. Moreover, our research indicates that several mover heads are reused across different types of knowledge or relational contexts. The mechanisms by which these heads are activated and the conditions under which they operate require further exploration and may shed light on why neurons are sometimes 'monosemantic' responding to a single feature, and sometimes 'polysemantic' [90] responding to many unrelated features.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Abstract and Section 1 Introduction

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:Section 8 Limitations and Appendix E.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: No theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We report the setup throughout the paper as well as in the Subsection 3.3 and Appendix A

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We use publicly available datasets (Appendix A), Code and Data are also provided in supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We report the setup throughout the paper as well as in the Subsection 3.3 and Appendix A

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: [No]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Appendix A

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer:[Yes]

Justification: We use publicly standard datasets that do not contain information about individual people or offensive context to our knowledge. Ethical considerations are discussed in Section 8.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In Section 8

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use publicly available artifacts and show them in Section 3.3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tries to study how language models (LMs) recall factual knowledge by finding circuits within the model internals in a mechanistic way, on a dataset of (subject, relation object)-like sentences. The discovered circuits help understand the mechanisms of LM's knowledge recall, and also enable some explorations on the impact of knowledge editing methods and analyze hallucinations and in-context learning (ICL).\n\nSTRENGTHS:\n- Discovering the internal mechanisms of LMs' factual recall is an important problem for better understanding and improving LM's knowledge-related abilities. The methodologies and the discovered circuits presented in the paper are interesting and may be useful for the community.\n- The analysis based on the discovered circuits is informative and could inspire further applications of circuit discovery.\n\nWEAKNESSES:\n- Overall, the findings in this paper are not particularly surprising, and mostly verify known mechanisms studied by prior investigations on knowledge recall in LMs. This could also be seen from the paper writing - in most (if not all) places where a discovered circuit component has a dedicated functionality, it is something already found in the literature (e.g., mover head, relation head, mixture head, etc.). Also given the simplicity of the task (recall the object entity given the subject and relation), it is not hard to anticipate these mechanisms in a transformer model in the first place.\n\n- The introduced methodology is not very technically involved. The adopted circuit discovery method is a direct application of standard causal mediation analysis in the literature, and very much follows the framework of Conmy et al. as cited.\n\n- The interpretations of the mechanisms are somewhat subjective and not rigorously justified. For example, there are usually direct claims about a certain head or a certain MLP layer functioning as such and such, without enough supporting evidence. It would be better to include more statistics to help convince the reader why things are the way they are described in the main text. For example, how to quantitatively define what a \"mover head\" is, and how did you get this from the statistical evidence?\n\n- There are few in-depth discussions, potentially due to the simplicity of the task. But even within the task itself, the overall findings are quite fragile and it is very hard to get the bigger picture from all the presented findings. Ultimately, the important question is: are there issues with the current LM's mechanisms on factual recall, and how to address them? Take hallucination for example, is it always the case that \"the model fails to move the correct knowledge to the final token in the earlier layers\"? If this is the case, why does this happen, and is there a general method to resolve the issue besides checking every error example and fixing the problematic components?\n\n- The writing overall could be improved. For example, \"zero ablation\" and \"MatchNLL\"  are repeatedly mentioned in Section 3.1 and 3.3. The caption around Figure 2 is not informative enough to understand the figure content (e.g., are these attention/mlp nodes?) and is too close to the main text.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-02"}

comments: SUMMARY:
The paper introduce a critical subgraph named Knowledge Circuit in the language model to understand how LLMs store and express the knowledge. Through extensive experiments and cases analysis, the authors show the significance of this proposed Knowledge Circuit, which can unveil implicit neural knowledge representations, elucidate internal mechanisms for knowledge editing, and facilitate interpreting language model behaviors.

STRENGTHS:
1. The authors approach the knowledge mechanism of LLMs from the Knowledge Circuit perspective, which has not been explored in this field and has good interpretability.

2. The authors conduct extensive experimental analysis, and find that the Knowledge Circuit is beneficial to unveiling implicit neural knowledge representations, understanding knowledge editing mechanism, and interpreting language model behaviors.

3. The paper is written-well, with a direct motivation and a well organized structure.

WEAKNESSES:
1. It would be better if the authors analyze more knowledge editing methods.

2. What is the average size of this subgraph? And what is the distribution of nodes?

QUESTIONS:
Please see above Weaknesses.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on the problem of interpreting the knowledge storage mechanism of the large language models. The authors proposed a new perspective, which uses the knowledge circuit to understand how the language model stores and expresses the knowledge. Knowledge circuit is a subgraph of the computation graph of the model. Specifically, the knowledge circuit of a given factual knowledge is defined as the smallest subgraph to recall it \"from the knowledge stored in the model\". They proposed a knowledge circuit construction method, where the nodes are ablated one-by-one to measure the impact of removing it from the computation graph. The resulted knowledge circuit will then be a necessary computation graph for maintaining the original inference result. Experiments are conducted across a variety of domains of factual recall tasks. To intuitively interpret the constructed knowledge circuits, they probes the information flow by projecting the latent representations into the space of token embeddings. Under this interpretation, several conclusions are drawn as follows.\n- language model tends to aggregate knowledge in the earlier to middle layers and further enhances this information in the later layers.\n- the knowledge editing method ROME tends to incorporate edited information primarily at the edited layer.\n- when hallucination occurs, the language model fails to correctly transfer knowledge to the final token in the earlier layers.\n- several new attention heads emerge in the knowledge circuit during in-context learning.\n\nSTRENGTHS:\n- It is in general well formatted and easy to follow.\n- Experimental details are well included.\n\nWEAKNESSES:\n- Regarding the novelty of this work, reference [36] appears to be highly relevant, as it involves the construction of a flow graph and the presentation of semantic information flow. However, this reference is not discussed in the related work section.\n- The robustness of the knowledge circuit construction method should be addressed. For instance, the results of the construction method should be unique or equivalent under different traversal orders. If not, the solution may not be well-defined.\n- The interpretation based on the knowledge circuit lacks clarity. Although several conclusions are drawn in the paper, it is unclear if these conclusions align with existing objective observations on related phenomena, such as in-context learning, and whether they could aid in predicting the model's behavior. The effectiveness of an explanation method should be validated by deriving a conclusion or guidance that can be subsequently verified through observation. This aspect is not clearly demonstrated in the paper.\n- It would be beneficial to include formal definitions of mover head, relation head, and mixture head, as these are crucial concepts in this paper.\n\nQUESTIONS:\n- Please refer to the weakness part.\n- According to Table 1, the performance drop of Circuit varies across different datasets. Is there any analysis of this phenomenon?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose utilizing techniques from mechanistic interpretability to explore the connections between model components involved in factual recall. They systematically ablate connections between model components in a reverse topological order, maintaining a list of the connections that most significantly recover loss on factual recall tasks. Several properties of the resultant \"knowledge circuit\" are measured, including its sufficiency for recovering model performance on factual recall.\n\nThe authors then demonstrate the roles of various components of the knowledge circuit, such as relation heads, mover heads, extract heads, and MLPs. They interpret the roles of these components by projecting their outputs directly into the logit space of the model. This approach is applied to understanding hallucinations and model editing failures.\n\nThe authors demonstrate how knowledge editing methods like ROME switch the role of the subject mover head to instead extract the final answer directly from the subject token rather than moving information about the subject.  The authors also manage to explain failures in multi-hop knowledge editing - the edited knowledge \"dominantly saturates\" the subject mover head preventing it from performing its natural function.  They also show how factual hallucination failures can be attributed to a mistake on the part of the subject mover head.\n\nSTRENGTHS:\n- As far as I am aware, this paper is the first to provide a mechanistic interpretation of the impact of knowledge editing techniques on a model.  The authors demonstrate how knowledge editing methods like ROME switch the role of the subject mover head to instead extract the final answer directly from the subject token rather than moving information about the subject.  The authors also manage to explain failures in multi-hop knowledge editing - the edited knowledge \"dominantly saturates\" the subject mover head preventing it from performing its natural function.  This work will be quite relevant to other researchers thinking about improving model editing techniques.\n- The authors provide a new perspective on the origins of hallucinations in the model. They attribute hallucination failures to the inability of the subject mover head to move the correct answer to the final token position. This mechanistic understanding offers valuable insights into why hallucinations occur and how they might be mitigated.\n- The authors contextualize their findings amongst other papers that attempt to mechanistically interpret factual recall in models.  The authors also reproduce a claim mentioned in several previous works, involving components in the early layers of a model being more important for knowledge retrieval (Figure 2).\n\nWEAKNESSES:\n- The authors choose to perform ROME [1] edits at layer 0 of the model.  In contrast, the original ROME paper performs edits in the middle of the model.  The precise layer is chosen using a causal tracing algorithm.  This choice is not justified in the paper.  It is unclear what difference this makes in the results provided in this paper.\n- In table 1, it is unclear why ablating everything but the circuit improves performance on some tasks, but decreases others.  For example, restricting the model to just the circuit increases the accuracy of \"Country Language\" but decreases the accuracy of \"fruit inside color\".\n\n[1] Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov. \"Locating and Editing Factual Associations in GPT.\" (2023).\n\nQUESTIONS:\n- Why did the authors choose to perform ROME edits at layer 0 of the model, in contrast to the original ROME paper's approach of editing in the middle layers using a causal tracing algorithm, and what impact might this choice have on the results presented in the paper?\n- In Table 1, what explains the inconsistent effects of ablating everything but the circuit on task performance, such as improving accuracy for \"Country Language\" while decreasing it for \"fruit inside color\"?\n\nLIMITATIONS:\nThe authors discuss the limitations of their work, and how it can contribute to ensuring safety and privacy information to promote trustworthy AI.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a circuit-based analysis of factual recall and model editing in language models. The authors make two key claims about knowledge storage and retrieval: 1) Factual recall is compatible with circuit-based frameworks. 2) Traditional model editing methods influence behavior change within the model's \"knowledge circuit.\" The paper compellingly advocates for studying knowledge storage and recall using the circuit framework, presenting preliminary but promising results.\n\nSTRENGTHS:\nThis paper is well-written, and the experiments are presented with clarity. While some of the results are still preliminary, the paper provides a novel perspective on interpreting and understanding the subject.\n\nWEAKNESSES:\n## General Comments\n\nDespite the strengths and contributions highlighted, a significant shortcoming of this paper is the absence of a concrete, successful application of circuits for the tasks mentioned in the introduction and Section 6, such as model editing, hallucination detection, and in-context learning (ICL). This is why I described the review as \"preliminary,\" even though the paper provides concrete experiments and evidence on the existence of a connection between factual knowledge recall and circuits.\n\nDespite its shortcomings, this paper offers an intriguing and promising new approach to addressing the problem by building on Niu et al.'s [21] call to \"...examine the entire decision-making circuit\" when studying how knowledge is stored and recalled in language models (LMs) with supporting evidence and experiments. The authors highlight the limitations of the previous knowledge neuron theory of LMs, noting its inadequacies as a fundamental theory for interpreting the factual recall process in LMs. The community needs a new theoretical framework, and I believe this paper points us in the right direction.\n\n### Knowledge Circuit Information Analysis\n\nThis section of the experiment aims to demonstrate Claim #1: that circuits can adequately explain the process of factual storage and recall. While the experimental results provide some preliminary support for this claim, the evidence is not as compelling as suggested in the introduction. Although the model shows initial success in using circuit-based interpretations for factual recall, some of the results are underwhelming. Notably, the relation \"capital city\" has a test set performance of 0%. Despite the authors' assertion that the standalone performance of the circuit is good, there is no baseline performance for comparison. Perhaps using random circuits as a baseline could provide a clearer benchmark.\n\n### Related Work\n\nThe paper provides an adequate summary of prior work in Section 7 and the introduction; however, there is room for improvement. The introduction initially highlights the issue with previous knowledge editing methods at lines 40-42, noting that \"different types of knowledge are often located in the same areas.\" While this is a valid issue, it is not the most critical one. The relevant work section (lines 336-339) offers a more comprehensive summary of the issues. Therefore, it would be beneficial to expand the introduction by incorporating a condensed summary from lines 336-339 to provide a clearer and more thorough context for the readers.\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}