Tracing and Dissecting How LLMs Recall Factual Knowledge for Real World Questions
Authors: Yiqun Wang, Sile Hu, Chaoqun Wan, Yonggang Zhang, Xiang Tian, Yaowu Chen, Xu Shen, Jieping Ye
Accepted at: acl25

Tracing and Dissecting How LLMs Recall Factual Knowledge for Real World Questions

Yiqun Wang1,2*, Chaoqun Wan2, Sile Hu2, Yonggang Zhang3, Xiang Tian1†, Yaowu Chen1, Xu Shen2‡, Jieping Ye2,
1Zhejiang University, 2Alibaba Cloud, 3Hong Kong Baptist University,
{11915039, tianx}@zju.edu.cn, shenxu.sx@alibaba-inc.com

Abstract

Recent advancements in large language models (LLMs) have shown promising ability to perform commonsense reasoning, bringing machines closer to human-like understanding. However, deciphering the internal reasoning processes of LLMs remains challenging due to the complex interdependencies among generated tokens, especially in practical question-answering. In this study, we introduce a two-dimensional analysis framework—comprising token back-tracing and individual token decoding—to uncover how LLMs conduct factual knowledge recall. Through explanatory analysis of three typical reasoning datasets, we identify a consistent three-phase pattern: Subject Augmentation and Broadcasting, Object Retrieval and Reranking, and Conclusion Fusion and Generation. Our findings reveal that LLMs do not lack relevant knowledge but struggle to select the most accurate information based on context during the retrieval and rerank phase. Leveraging these findings, we apply representation engineering and selective fine-tuning to target specific modules responsible for retrieval and rerank errors. Experimental results show large improvements in response accuracy for both in-domain and out-of-domain settings, validating the rationality of the interpreting result.

1 Introduction

Recent progress in large language models (LLMs) have pushed machines closer to achieving human-like capabilities (Krause and Stolzenburg, 2023; Zhou et al., 2020). These models can not only comprehend user queries, but also perform commonsense reasoning based on factual knowledge. As a result, uncovering these abilities has become a focal point of interest. It is crucial for interpreting model behavior and analyzing unexpected errors (e.g., reversal curse (Berglund et al., 2023)), ultimately overcoming the limitations of LLMs.

Research on interpreting LLMs (Geva et al., 2023; Wang et al., 2024; Dai et al., 2022; Xie et al., 2024) often simplifies reasoning by focusing on factual triplets like “Ganesha is a Hindu god”. These studies examine how models derive the object (“Hindu”) from the subject (“Ganesha”) as well as the relation (“is”). However, in real-world scenarios, model must go beyond these triplets to understand the question, select relevant facts, and synthesize information to provide an answer. For example, when asked, “Is Ganesha associated with Thor?”, the model must comprehend the context, recognize that “Ganesha is a Hindu god” from all Ganesha-related facts, and conclude they are not related. In contrast, for the question, “Does Ganesha look like a tiger?”, the model in turn focuses on appearance-related facts, such as “Ganesha is depicted with an elephant head”. Understanding how models select appropriate factual knowledge and leverage it to reach conclusions is crucial for comprehending the overall reasoning process. This holistic approach extends beyond simple triplet analysis and can better reflect the complexity of real-world reasoning tasks.

In this study, we aim to decipher the commonsense reasoning process within the response of LLMs. Specifically, we focus on real-world scenarios where models typically generate complex and multi-token rationales before producing the final answer. The challenge lies in the dense inter-

![Model inner reasoning process on commonsense reasoning tasks.](page_1012_370_388_186.png)

Figure 1: Model inner reasoning process on commonsense reasoning tasks.
connectivity of token generation, where each generated token is influenced by multiple preceding ones, leading to a recursive analytical complexity. Existing interpretability tools cannot be directly applied to analyze this complexity. To decipher the multi-token generation process, we design a new framework by breaking down the analysis into two dimensions: token back-tracing and individual token decoding. Token back-tracing starts from the final answer and traces back to the original question. It identifies intermediate key tokens with significant direct impact through causal analysis. This reveals a chain of crucial information transfers between tokens, as shown in Fig. 1. For individual token decoding, following Wang et al. (2023), we adopt an “explain then verify” strategy. We first identify and decode the semantic information within the key modules. Then these key modules are knocked out to verify the reliability of results.

The interpreting analysis of three typical reasoning datasets revealed a consistent pattern in models’ commonsense reasoning. The process unfolds in three stages: 1) Subject augmentation and broadcast: the model first generates extensive subject-related information through attention heads and MLP, and broadcasts it to subsequent key positions (e.g., sentence endings); 2) Object retrieval and rerank: the model retrieves the previously generated subject information with attention heads and reorders it using MLPs when predicting attributes; and 3) Conclusion fusion and generation: the attributes are further transported to the conclusion through heads and generate corresponding conclusions, ultimately forming the answer. Based on this pattern, we further analyzed the failure cases of current models. One key finding is that LLMs are not unaware of relevant facts, but rather struggle to select the most accurate fact during retrieval and rerank based on contextual cues. This motivated us to develop a direct application of interpretability findings: by identifying specific modules through explanatory localization, we employed selective fine-tuning and representation engineering to optimize the attribute retrieval and rerank. Results show significant improvement in model performance, simultaneously validating the rationality of the interpretability results.

We summarize our contributions as follows: (1) We introduce an effective interpreting framework that combines token back-tracing with individual token decoding to understand how LLMs reason across multiple tokens. (2) We break down how language models perform commonsense reasoning into human-understandable steps: LLMs first augment related facts and broadcast the information into the proceeding key positions, subsequently retrieving and re-ranking these facts to predict correct object, and finally fusing and generating conclusions. (3) Using the interpreting result, we identify that on commonsense reasoning tasks, LLMs often fail to retrieve and rerank correct facts, leading to erroneous reasoning or conclusions. By selectively fine-tuning key heads and MLPs, the performance of reasoning is enhanced, especially for out-of-domain samples. It validates the reliability of the interpreting results.

2 Related Works

2.1 Mechanistic Interpretability

Mechanistic interpretability in LLMs aims to understand model behavior by reverse-engineering the internal computational processes. Many interpretability tools have been developed to analyze language models. Logit attribution projects internal vectors into vocabulary space to interpret encoded information, and has been successfully applied in multiple studies to reveal various interpretability findings (Geva et al., 2021b, 2022; Dar et al., 2023; Belrose et al., 2023). Activation patching employs causal interventions on internal model components using corrupted inputs (Meng et al., 2022; Wang et al., 2023; Goldowsky-Dill et al., 2023; Conmy et al., 2023). This approach identifies critical modules and computational circuits by analyzing changes in model predictions. It has been effectively used to identify task-specific modules across various LLM studies (Lieberum et al., 2023; Zhang et al., 2024; Chen et al., 2024; Hanna et al., 2023). Sparse autoencoders decompose internal features into interpretable combinations (Bricken et al., 2023; Templeton et al., 2024; Lieberum et al., 2024; Gao et al., 2024), while knockout techniques verify component importance by analyzing prediction changes after component removal (Wang et al., 2023; Olsson et al., 2022). In our work, we adapt these tools to interpret the model’s reasoning process. A detailed comparison of existing interpretability tools and our selection criteria is presented in §A.1.

2.2 Mechanism of Factual Knowledge Recall

Numerous studies have employed interpretability tools to investigate model mechanisms in reasoning
<table>
  <tr>
    <th>Data Type</th>
    <th>Input Case</th>
    <th>Output</th>
    <th>Related Research</th>
  </tr>
  <tr>
    <td>Single-hop knowledge recall</td>
    <td>The singer of ‘Superstition’ is __</td>
    <td>Stevie Wonder (single token)</td>
    <td>Geva et al. (2023); Dai et al. (2022); Yu and Ananiadou (2024)</td>
  </tr>
  <tr>
    <td>Multi-hop knowledge recall</td>
    <td>The mother of the singer of ‘Superstition’ is __</td>
    <td>Lula (single token)</td>
    <td>Yang et al. (2024); Biran et al. (2024)</td>
  </tr>
  <tr>
    <td>Real world QA with Rationale</td>
    <td>Q: Can Harry Potter book a flight on Asiana Airlines?<br>A:</td>
    <td>Harry Potter is a fictional character. Fictional characters cannot book flights. Thus, Harry Potter cannot book a flight on Asiana Airlines. So the answer is no. (complex multi-token)</td>
    <td>None</td>
  </tr>
</table>

Table 1: Comparison of factual knowledge recall tasks. Prior studies focused on single-token answer generation, while the mechanism of how LLMs generate final answers through step-by-step reasoning in real-world queries remains largely unexplored.

tasks. Geva et al. (2023) explored factual knowledge recall, finding that subject information is enriched in the subject token in early layers, while relation information is passed to the final token, which then uses attention heads to extract the corresponding attribute from the subject representation. Building on this, Wang et al. (2024); Dai et al. (2022); Yu and Ananiadou (2024); Geva et al. (2022) further identified MLP neurons involved in factual knowledge recall and demonstrated how modulating their activations can control model behavior. Additionally, works such as Yu et al. (2024); Ortu et al. (2024); Yu et al. (2023); Xie et al. (2024) analyzed the balance between retrieved knowledge and parametric memory. These studies largely focus on elementary retrieval tasks, such as recalling a single fact \( o \) within a triplet (\( s, r, o \)). In this study, we focus on interpreting the model’s reasoning process in more complicated reasoning tasks.

3 Methods

As shown in Tab. 1, previous studies on factual recall have primarily focused on single-token generation processes (Geva et al., 2023; Yang et al., 2024; Dai et al., 2022) on single-hop and multi-hop factual knowledge recall tasks. However, in real-world knowledge question answering scenarios (Geva et al., 2021a), language models typically generate complex, multi-token responses. Yet current interpretability tools, designed primarily for single-token generation analysis, are insufficient for explaining the intricate token interactions within multi-token responses.

3.1 Preliminary

In our experiments, we uncovered several key token positions in the reasoning process through back tracing: Subject, Object, Answer. These tokens are observed special in experiments and therefore highlighted for better comprehension. (1) **Subject** (\( S \)): The subject of inquiry in the question, represented as a concept node in a knowledge graph (Speer et al., 2017), denoting entity, idea, or object in commonsense reasoning (e.g., “*Harry Potter*” in Fig. 2). (2) **Object** (\( O \)): The object, paired with \( S \) contains some factual knowledge, is also a concept node. These objects, according to their relevance in the context of the question, can be categorized into predicted objects \( O_p \) (e.g., “*Harry Potter is a ‘fictional character’*”) and candidate objects \( O_c \) (e.g., “*Harry Potter is a ‘wizard’*”). (3) **Answer** (\( A \)): The answer to the question, which varies based on the type of question. It may be a binary judgment (e.g., “yes/no”) or a selection (e.g., “*(2) Kayla*”). We denote the correct and false answers as \( A_t \) and \( A_f \).

Furthermore, through back-tracing, we identified several positions that entail reasoning-related information: (4) **Reasoning conjunctive adverb** (\( R \)): we find conjunctive adverbs that connect reasoning steps (e.g., “*Thus*”) encodes rich information related to the answer. (5) **Conclusion** (\( C \)): terms that convey the affirmative or negating essence of the conclusion sentence, clarifying the stance to the question. (e.g., “*cannot*” in “*Thus, Harry Potter cannot book a flight on Asiana Airlines.*”) (6) **Question end** (\( Q_e \)): we find abundant subject-related information encoded at the end of the question.

3.2 Methodology

As illustrated in Fig. 2, the interpretation process is divided into two orthogonal pipelines. 1) **Token back-tracing**: The horizontal pipeline traces the path of tokens from the end to the start. Through causal back-tracing, tokens that are strongly corre-
Figure 2: (a) Two dimensions of our interpretation framework: i) Tracing back (horizontal): we use activation patching to identify the head with causal effect and trace the origin of the information iteratively. ii) Decoding concept knowledge (vertical): we use logit attribution to identify the key module for generating concepts during reasoning at the key position and decode the semantic information within it. (b) Example of Tracing Back: We first identify the key attention heads responsible for generating the attribute “fictional”, then trace back to the source information (“Harry Potter”) by analyzing the tokens these heads attend to. (c) Example of Decoding Concept Knowledge: We locate modules specifically contributing to the predicted attribute (“fictional”) by comparing prediction probabilities between the predicted and candidate attributes (e.g., “wizard”) in MLP outputs. We then use SAE to identify and interpret important features in MLP, revealing semantic patterns related to fictional content.

lated with a target token can be effectively identified, allowing us to focus on the most relevant information flows rather than exhaustively analyzing the dense connections across all tokens. This approach helps identify the key relationships between tokens, thereby pinpointing the crucial positions of key tokens in commonsense reasoning (as defined in §3.1). 2) Decode parametric concept or attribute: The second pipeline, shown vertically, analyzes the patterns within LLMs when generating a specific token, including inner behaviors and activation characteristics. It explains the behavior of modules (i.e., residual blocks, attention heads, and MLPs) by evaluating the information related to the target content (e.g., \( A_t, A_f, O_p, \) and \( O_c \)) within modules’ output. Subsequently, it decodes the semantic information and patterns encoded in these modules into human-understandable formats.

Instantiation of tracing token-to-token path. We employ activation patching (Wang et al., 2023) for causal back-tracing. This method, which originates from causal mediation analysis (Vig et al., 2020), enables us to identify significant attention heads through direct effect analysis (as shown in the right side of Fig. 2). We made several improvements to the original method: (1) Developing a refined metric to reduce noise in key module identification. (2) Extending the analysis from final logits to the middle layer outputs. (3) Automating the generation of counterfactual data to obtain large-scale results (See §A.2 for details). In our implementation, we identify heads with the Top-5 direct effect as key contributors to token generation. By analyzing the attention patterns in these important heads, we select the top 2 previous tokens with the highest attention scores as being correlated with the current token, serving as the basis for further tracing and analysis. For instance, as show in Fig. 2 (b), we identified the attention head responsible for generating \( O \) (“fictional”). Subsequently, by analyzing the attention score distribution (i.e. the last row of the attention score matrix), the most correlated token \( S \) (“Harry Potter”) is located. This process is then iteratively applied to discover the complete transition path across tokens.

Instantiation of decoding parametric concept or attribute. We use logit attribution (nostalgiaerist, 2021) to interpret the module behavior across layers. The method projects hidden states into the vocabulary space using the model’s pretrained unembedding matrix and obtains its distribution on the vocabulary space. Therefore, the method reveals the information contained in current hidden states and explains the contribution of specific heads or MLPs or residual blocks to the predicted token. To address the false identification issue, we calculate the softmax probability of the multiple tokens (\( O_p, O_c, A_t, \) or \( A_f \)) after projection. This improvement ensures the identified key modules contribute specifically to target prediction (e.g., \( O_p \)) rather than all related tokens (e.g., both \( O_p \) and \( O_c \)). The probabilities across layers will form the curves (see
line plot in Fig. 2 (c) for illustration), indicating the module’s inner reasoning process.

To validate the interpreting results obtained by logit attribution, for MLP, we adopt Sparse Autoencoder (SAE) (Templeton et al., 2024) to decode the semantic information embedded in the parameters and activations. (e.g., information related to “fictional characters” is decoded in MLP of deep layer when predicting “fictional” as shown in Fig. 2 (c)). More details are introduced in §A.2.2. Regarding attention heads, we use logit attribution to decode the semantic information. We project the outputs of the heads into the vocabulary space and examine the top-20 tokens in the head’s output distribution to decode the semantic information. To validate the functional roles of these key components, we followed Wang et al. (2023) by knocking out these modules to observe the influence on output.

Method selection and improvement details. Our interpretability framework leverages a diverse set of analytical tools, each selected to address specific aspects of the explanation process. Tab. 5 provides a concise overview of our tool selection rationale, while a more comprehensive analysis and comparison of existing interpretability tools is detailed in §A.1. Additionally, we have refined these tools with detailed improvements described in §A.2.

4 Experiments

4.1 Experiments Overview

Consider a question: “Q: Can Harry Potter book a flight on Asiana Airlines?” and Gemma2-9B’s output is “Harry Potter is a fictional character. Fictional characters cannot book flights. Thus, Harry Potter cannot book a flight on Asiana Airlines. So the answer is no.”. Through extensive experimental results, we find the models’ internal reasoning process consists of three distinct stages: (1) Subject Augmentation and Broadcast, at subject token position (\( S \), “Harry Potter”), the model extends from the subject to augment relevant object (e.g., “Wizard” and “fictional”). (2) Object Retrieval and Rerank, when predicting object token (\( O_p \), “fictional”) attention is responsible for retrieving related objects while MLP layers rank the most appropriate one as output. (3) Conclusion Fusion and Generation, when predicting conclusion (\( C \), “cannot”) and answer (\( A \), “no”) tokens, the model integrates the previous information and generates the final answer through attention heads and MLPs.

Given our analytical pipeline’s back-tracing nature, we present the results in reverse chronological order to align with the original investigation procedure. We begin tracing back from \( \mathcal{A} \rightarrow \mathcal{C} \rightarrow \mathcal{O} \), and decoding to find the conclusion fusion and generation (§4.3). Diving deeper into \( \mathcal{O} \), we further observe object retrieval and rerank (§4.4). Further tracing the origin of \( \mathcal{O} \) leads us to \( \mathcal{S} \), uncovering subject augmentation and broadcast (§4.5). Additionally, We extend our investigation across different models and datasets in §4.6.

4.2 Experiments Settings

Models. We conducted experiments on two popular open-sourced models, Gemma2-9B (Team et al., 2024) and Llama2-7B (Touvron et al., 2023). The results in this section primarily focus on Gemma2-9B, as Sparse Autoencoders (SAEs) have been trained for all its layers (including residual and MLP layers) (Lieberum et al., 2024), enabling comprehensive validation of our analyses. Gemma2-9B consists of 42 layers, Llama2-7B consists of 32 layers. See Appendix A.8 for results on Llama2-7B.

Datasets. We selected three widely used commonsense reasoning benchmark datasets: StrategyQA (Geva et al., 2021a), CommonsenseQA (Talmor et al., 2018), and SocialIQA (Sap et al., 2019). These three datasets evaluate distinct reasoning capabilities of language models (see Tab. 12 for details). Given that StrategyQA presents more sophisticated reasoning challenges, we primarily present the StrategyQA results in the main text. Detailed results for the other two datasets are presented in § A.6.

Settings. Following the experimental protocols in Geva et al. (2023); Lieberum et al. (2023), we randomly sampled 1,000 instances from each dataset for our experiments. All figures presented in this paper are averaged results across this sample size, ensuring statistical reliability while maintaining computational feasibility. Detailed analysis of result stability across various sizes of samples is presented in Appendix A.4. To elicit the step-by-step rationale, we adopt the few-shot CoT prompts from (Wei et al., 2022; Li et al., 2024).

4.3 Conclusion Fusion and Generation

We start from decoding the information of \( \mathcal{A}_l \) and \( \mathcal{A}_f \) (i.e. “yes” and “no”) in residual blocks, attention, and MLP layers at the position of predicting \( \mathcal{A} \) as shown in Figure 3a. The curves of residual blocks depicts how the model predicts \( \mathcal{A} \) across layers while curves of attention and MLP layers de-
Figure 3: (a) Logit attribution of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at predicting \( \mathcal{A} \) on StrategyQA. (b) Logit attribution of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at predicting \( \mathcal{O} \) on StrategyQA.

<table>
  <tr>
    <th>Head</th>
    <th>Top tokens in projection</th>
  </tr>
  <tr>
    <td>28.06</td>
    <td>yes, yeah, no, nil, Yes</td>
  </tr>
  <tr>
    <td>32.07</td>
    <td>Noah, node, Noah, no, Nora</td>
  </tr>
  <tr>
    <td>34.09</td>
    <td>denying, denied, denial, deny</td>
  </tr>
  <tr>
    <td>35.14</td>
    <td>ye, Ye, Yea, YE, yes, YES, Yeh</td>
  </tr>
</table>

Table 2: Top-scoring tokens in the key attention heads output when predicting \( \mathcal{A} \) on StrategyQA.

<table>
  <tr>
    <th>Layer</th>
    <th>ID</th>
    <th>Feature Explanation</th>
  </tr>
  <tr>
    <td>27</td>
    <td>76551</td>
    <td>questions and answers related to decision-making and assessments.</td>
  </tr>
  <tr>
    <td>30</td>
    <td>21336</td>
    <td>affirmative and negative responses to questions.</td>
  </tr>
  <tr>
    <td>38</td>
    <td>101266</td>
    <td>answers presented in a structured format, particularly in multiple-choice or quiz contexts.</td>
  </tr>
</table>

Table 3: Top-scoring features decoded by SAE in the output of the key MLP layers when predicting \( \mathcal{A} \) on StrategyQA.

pict the module contribution to the \( \mathcal{A}_t \) and \( \mathcal{A}_f \). The prediction of \( \mathcal{A} \) can be divided into three stages: (i) **Stage 1** (\( l = 0 - 24 \)): Little to no answer-related information is present in residual blocks, attention and MLP layers, indicating the model is still processing the input. (ii) **Stage 2** (\( l = 25 - 33 \)): Information related to the answer increases, yet the probabilities for \( \mathcal{A}_t \) and \( \mathcal{A}_f \) are close across residual blocks. Within the modules, attention heads begin to convey answer-related information from layer 25 and the MLP follows to encode this information from layer 26. Notably, the outputs of attention and MLP show similar information for both \( \mathcal{A}_t \) and \( \mathcal{A}_f \). In this stage, the model starts to generate an answer but has not yet identified the correct one. (iii) **Stage 3**: By layer 34, the model distinguishes the correct answer \( \mathcal{A}_t \), with its probability sharply rising and the \( \mathcal{A}_f \)'s probability decreasing. At the same layer, the attention output sharply spikes for \( \mathcal{A}_t \) (probability near 1.0), while the MLP output is much lower (\( \approx 0.1 \)). Afterward, the outputs of MLPs further increase \( \mathcal{A}_t \)'s probability (\( l = 37 - 38 \)), leading to the final prediction. In conclusion, attention is responsible for fusing related information, while the MLP enhances the probability of the correct answer, contributing to generating the final answer.

We further investigated the semantic information encoded in the outputs of MLP and attention heads for verification. In attention heads, we found that in stage 2 and 3, the key heads encoded information related to both \( \mathcal{A}_t \) and \( \mathcal{A}_f \) (see the outputs of heads in Tab. 2). Meanwhile, numerous features related to decision-making (see Tab. 2) are identified in MLPs. These findings provide additional evidence supporting the critical role of the MLP and Attention layer in the answer generation process.

Finally, we applied activation patching to identify key heads and trace the information for generating \( \mathcal{A} \). Tracing the information flow, the path began at the conclusion \( S \), progressed to the reasoning conjunctive adverb \( \mathcal{R} \), and finally arrived at object \( \mathcal{O} \). In the process, we discovered that \( \mathcal{R} \) acts as anchors for the fusion and transport of conclusion-related information in the reasoning process. For a detailed examination of the trace from \( \mathcal{A} \) to \( \mathcal{O} \), and an in-depth analysis of answer-related information at \( \mathcal{R} \), please refer to §A.5.

4.4 Object Retrieval and Rerank

The object information \( \mathcal{O} \) decoded in the outputs of the residual block, attention layers, and MLP layers is shown in Fig. 3b. We examined the predicted object \( \mathcal{O}_p \) and candidate object \( \mathcal{O}_c \) probability within these modules. For the residual block, the object information emerges at around layer 26. However, \( \mathcal{O}_p \) is not dominant in the first place, as the probabilities of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) increase alternately. For attention, \( \mathcal{O}_p \) and \( \mathcal{O}_c \) interleave, with neither showing explicit dominance throughout the whole layers. On the contrary, MLP shows obvious preference for \( \mathcal{O}_p \), where correct object information is prominent across almost all layers. Notably, at
layer 37, \( \mathcal{O}_p \) is clearly dominant, while \( \mathcal{O}_c \) remains minimal. This sharp spike aligns with a key transition point in the curve of residual block. From these observations, it seems that 1) both \( \mathcal{O}_p \) and \( \mathcal{O}_c \) are integrated during the process of object token generation. 2) The attention heads initially retrieve the information for both \( \mathcal{O}_p \) and \( \mathcal{O}_c \), while MLPs subsequently rerank \( \mathcal{O}_p \) to the top position.

To validate our finding, we look into the output of attention heads and MLP. As shown in Tab. 15, attention heads encode a rich set of attribute information relevant to the subject (e.g., “British”, “wizard”, “book”, and etc). Meanwhile, in Tab. 13, the features in MLP decoded by SAE are strongly related to “identity and character” (Tab. 13). These features are highly correlated to \( \mathcal{O}_p \), but none of them is related to \( \mathcal{O}_c \). These results validate the retrieving function for the attention head and the reranking function for the MLP.

Finally, we utilize activation patching to identify the heads with causal effect (see Fig. 10a) and find these heads focus on two critical token positions, \( \mathcal{S} \) and end of question. Therefore, we trace back to \( \mathcal{S} \) and \( \mathcal{Q}_e \) to investigate the origin of \( \mathcal{O} \).

Figure 4a illustrates the information of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) decoded in the outputs. Notably, we observe that: 1) Residual block contains obvious information regarding both \( \mathcal{O}_p \) and \( \mathcal{O}_c \) across various layers, with \( \mathcal{O}_c \) being more prominent at the end. 2) Another two curves show that both attention heads and MLPs have a large influence on \( \mathcal{O}_p \) and \( \mathcal{O}_c \). To further decode information, we identifyies that MLPs in layers 7 and 32 encode abundant features related to \( \mathcal{O} \) (see Tab. 14). Meanwhile, Probing also reveals that heads in layers 29 and 39 rank the \( \mathcal{O}_c \) at top. In addition to diminishing the impact of the information from any previous token, we also examine the three corresponding curves at the position before \( \mathcal{S} \) (for instance, “Question: [Can Harry Potter]”). The results (green line in Fig. 4a) reveal that the information regarding \( \mathcal{O} \) is virtually zero. It indicates that the emergence of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) is indeed contingent upon the appearance of \( \mathcal{C} \) and is independent of any previous tokens. In conclusion, both the MLP and heads play essential roles in assisting the model to associate and extend from \( \mathcal{S} \) to related \( \mathcal{O}_p \) and \( \mathcal{O}_c \). We refer to this stage, along with the contributions of the MLP and heads, as **subject augmentation**.

Regarding the question’s end position (\( \mathcal{Q}_e \)), Fig. 4b also presents the three corresponding curves. (1) In the residual, both \( \mathcal{O}_p \) and \( \mathcal{O}_c \) appear across multiple layers. On the contrary to the concept token position, \( \mathcal{O}_p \) has a greater presence than \( \mathcal{O}_c \). (2) The curves for the MLP and heads also encapsulate information about both \( \mathcal{O}_p \) and \( \mathcal{O}_c \), and further enhance the importance of \( \mathcal{O}_p \). It indicates that even at unrelated token positions, the \( \mathcal{O} \) corresponding to the \( \mathcal{S} \) (or the knowledge they encompass) can be broadcast. The original order of \( \mathcal{O} \) may be broadcast based on the current context, ultimately influencing the generation of \( \mathcal{O}_p \). We term this stage as **subject broadcasting**.

![Logit attribution of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at \( \mathcal{S} \) in StrategyQA. (b) Logit attribution of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at the end of question in StrategyQA.](page_186_1042_1077_377.png)

Figure 4: (a) Logit attribution of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at \( \mathcal{S} \) in StrategyQA. (b) Logit attribution of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at the end of question in StrategyQA.

4.5 Subject Augmentation and Broadcast

Generally, in commonsense reasoning datasets, the \( \mathcal{S} \) always appears in both the question and the rationale. Through analysis, we observe that the \( \mathcal{S} \) in the rationale can also be back-traced to the \( \mathcal{S} \) in the question. Therefore, we treat the position of \( \mathcal{S} \) in the question as a focal point for deeper analysis.

4.6 Verification and generalization of findings

To validate our interpretations, we conducted knockout analysis on key model components. Fig. 6 demonstrates the impact of sequentially knocking out the top-10 attention heads on predicted token probabilities when the model outputs \( \mathcal{O}_p \) (Fig. 6a) and \( \mathcal{A}_t \) (Fig. 6b). Our findings reveal substantial negative effects, with probability decrements ranging from 30% to 60%. In contrast, randomly knocking out 10 heads showed a negligible impact on the results, with maximum degradation of merely 2%. Additionally, we conducted knock-
Figure 5: The distribution of the four types of errors encountered by Gemma2-9B on StrategyQA. 1) Reference Error: The model retrieves irrelevant or wrong attributes. 2) Logic Error: incomplete reasoning steps. 3) Conclusion Error: reaches an incorrect answer, but based on correct rationale. 4) Concept Error: incorrectly identifies the target concept for analysis.

Figure 6: Change in the rate of predicted object (\( O_p \)) (a) and correct answer (\( A_t \)) (b) when knocking out the top-10 corresponding key attention heads identified by activation patching.

out on the MLPs responsible for object reranking (MLP 37 for Gemma2-9B and MLP 20 for Llama2-7B). The results indicate significant drops in the probability of \( O_p \), with reductions of 42% and 52% respectively. These results provide strong causal evidence supporting our identified key modules.

Further analysis of Gemma2-9B’s reasoning process on CommonsenseQA and SocialIQA (\S A.6) using 1,000 samples revealed similar patterns of object retrieval, reranking, and conclusion generation. However, subject augmentation was less prominent in these datasets, presumably due to the explicit provision of factual knowledge within the question context. These findings were also replicated using the Llama2-7B model across all three datasets (detailed results in \S A.8).

5 Application of Interpreting Results

5.1 Failure Case Analysis

In this section, we analyze the failure of LLMs in commonsense reasoning (\S 5.1) and then introduce two applications of interpreting results to enhance the model’s reasoning capability (\S 5.2 and \S 5.3).

We manually analyze all failure cases (167) of Gemma2-9B on StrategyQA training set. The results reveal four error types (Fig. 5): 1) Reference Errors: retrieving irrelevant or incorrect objects; 2) Logic Errors: insufficient knowledge to support conclusions; 3) Conclusion Errors: wrong answers despite correct reasoning; and 4) Concept Errors: misidentification of target concepts to analyze. Reference Errors dominate at 74% of all cases. Further probing reveals that these errors primarily stem from object reranking issues rather than knowledge gaps (see \S A.7 for details), as correct objects typically appear within the model’s top-5 predicted tokens. Based on this finding, we propose enhancing commonsense reasoning by using selective supervised fine-tuning and representation engineering.

5.2 Selective Supervised Fine-tuning

Zhang et al. (2024); Chen et al. (2024) proposed a method to enhance model’s capability through updating a small set of parameters. Specifically, given a sequence of attention heads and MLPs ordered by their significance, denoted as \( (MLP_{l_1}, Head_{l_2,h_2}), (Head_{l_3,h_3}), \ldots \), where \( l_i \) represents the layer index and \( h_i \) represents the head index of the \( i^{th} \) ranked head, only parameters of top \( K \) heads and top \( M \) MLPs are exclusively updated during fine-tuning. Following the same setting, we selectively fine-tune the top 32 Attention heads (for knowledge retrieval, i.e., red squares in Fig. 21a) and top 1 MLP layers (for knowledge reranking, i.e., peak in Fig. 3b MLP). Considering the generalization, we introduce another commonsense reasoning test dataset, WinoGrande (Sakaguchi et al., 2021). See \S A.9 for more detailed experiment settings.

**Experiment Results.** The comparative results between SSFT and SFT are presented in Table 4. For the experiments of Gemma2-9B on StrategyQA, both SSFT and SFT improved performance,
<table>
  <tr>
    <th rowspan="2">Models</th>
    <th colspan="2">ID Task</th>
    <th colspan="3">OOD Task</th>
  </tr>
  <tr>
    <th>Strategy</th>
    <th>CSQA</th>
    <th>SIQA</th>
    <th>Wino</th>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>70.7</td>
    <td>75.7</td>
    <td>73.0</td>
    <td>61.2</td>
  </tr>
  <tr>
    <td>+ SFT (9B)</td>
    <td>79.0</td>
    <td>74.3</td>
    <td>70.9</td>
    <td>60.3</td>
  </tr>
  <tr>
    <td>+ SSFT (0.3B)</td>
    <td>80.3</td>
    <td>76.2</td>
    <td>74.0</td>
    <td>65.2</td>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>62.5</td>
    <td>68.3</td>
    <td>67.9</td>
    <td>55.5</td>
  </tr>
  <tr>
    <td>+ SFT (7B)</td>
    <td>77.3</td>
    <td>54.8</td>
    <td>59.0</td>
    <td>52.7</td>
  </tr>
  <tr>
    <td>+ SSFT (0.2B)</td>
    <td>78.5</td>
    <td>64.1</td>
    <td>63.2</td>
    <td>61.1</td>
  </tr>
</table>

Table 4: Results on four commonsense reasoning tasks (i.e., StrategyQA, CSQA, Winogrande, and SocialIQA) before and after tuning on the StrategyQA dataset.

achieving gains of +8.3% and +9.6%, respectively. While SFT shows a comparable enhancement for the StrategyQA task, it adversely affected performance on OOD tasks, with an average decrease of −1.5%. In contrast, SSFT continued to bolster the model’s reasoning ability across all OOD commonsense reasoning tasks, improving the performance by an average of +2.6%. These findings suggest that selectively fine-tuning a small fraction of key components for commonsense reasoning can boost performance on ID tasks while maintaining generalizability, highlighting the effectiveness of our previous exploration. A similar trend was observed in the Llama2-7B results. Through mechanism analysis of the model before and after SSFT, we further validate that SSFT enhances the model’s knowledge retrieval and reranking capabilities. (See Fig. 23). Additionally, we further validate the effectiveness of SSFT through training on two other datasets (Tab. 16 and 17).

5.3 Representation Engineering

Representation engineering, which adjusts the model’s internal hidden states to influence its behavior, has proven to be an effective method for modulating model performance (Zou et al., 2023). Following the approach outlined in Xiao et al. (2024); Templeton et al. (2024), we correct the model’s erroneous behavior using:

\[
\tilde{h}_l = h_l + k x_t,
\]

where \( h_l \) represents the original output of residual block at layer \( l \), \( x_t \) is the feature direction corresponding to the correct knowledge identified using SAE, \( k \) is the steering magnitude which we set 5. See §A.3 for more details.

**Experiment Results.** We utilize representation engineering to correct the model’s (Gemma2-9B) failure in recalling correct object. For example, in question “Would Persephone be a suitable consultant to a landscape architect?”. The model initially defaults to identifying “Persephone as the Greek goddess of the underworld”, leading to an incorrect assessment. The correct reference is “Persephone is the Greek goddess of spring”. By introducing feature directions related to deities or nature into the residual block at layer 37 (object retrieval), we strengthened the model’s tendency to associate “Persephone” with “spring”. This tendency can largely contribute to the correct answer, and rectify the model’s response. As a result, 93% failure cases can be rectified, illustrating the rationality of the identified interpreting results.

6 Conclusion

In conclusion, our research sheds light on the intricate dynamics of commonsense reasoning within LLMs, revealing a structured process that parallels human cognitive reasoning. By meticulously analyzing the hidden states across various transformer layers and token positions, we identified a multi-faceted mechanism that integrates knowledge augmentation, retrieval, and answer generation—essentially resembling a retrieval-augmented generation framework. Our findings underscore the pivotal roles played by both attention heads and MLPs in the manifestation of factual knowledge, highlighting a dual approach to knowledge processing. Furthermore, our experiments demonstrated that while LLMs often possess relevant factual knowledge, they frequently struggle to retrieve the correct information during inference. Through selective fine-tuning of key components, we achieved notable enhancements in reasoning performance across diverse contexts, indicating that targeted adjustments can effectively optimize the reasoning capabilities of LLMs.

Limitations

While our study provides valuable insights into LLMs’ factual knowledge recall mechanisms, several limitations warrant discussion. First, our analysis primarily focuses on commonsense reasoning tasks involving factual knowledge recall, and the identified patterns may not fully generalize to other types of reasoning tasks. Future research could extend this framework to investigate more diverse reasoning scenarios. Second, our backtracing methodology operates under the assumption that only sparse connections between reason-
ing tokens are significant, which may potentially overlook some subtle information flows within the network. Finally, although we propose functional interpretations of different modules based on observed patterns, we acknowledge the inherent challenge of mapping complex neural mechanisms to human-understandable explanations, potentially leading to oversimplification of the actual processes involved.

Acknowledgements

This work was supported in part by the Zhejiang Provincial Natural Science Foundation of China under Grant LDT23F01013F01 and in part by the Fundamental Research Funds for the Central Universities.

References

Nora Belrose, Zach Furman, Logan Smith, and et al. 2023. Eliciting latent predictions from transformers with the tuned lens. CoRR, abs/2303.08112.

Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The reversal curse: LMs trained on “a is b” fail to learn “b is a”. In The Twelfth International Conference on Learning Representations.

Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. 2024. Hopping too late: Exploring the limitations of large language models on multi-hop queries. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14113–14130, Miami, Florida, USA. Association for Computational Linguistics.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html.

Wei Chen, Zhen Huang, Liang Xie, Binbin Lin, Houqiang Li, Le Lu, Xinmei Tian, Deng Cai, Yonggang Zhang, Wenxiao Wan, et al. 2024. From yes-men to truth-tellers: Addressing sycophancy in large language models with pinpoint tuning. arXiv preprint arXiv:2409.01658.

Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. 2023. Towards automated circuit discovery for mechanistic interpretability. In Advances in Neural Information Processing Systems, volume 36, pages 16318–16352. Curran Associates, Inc.

Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493–8502, Dublin, Ireland. Association for Computational Linguistics.

Guy Dar, Mor Geva, Ankit Gupta, and et al. 2023. Analyzing transformers in embedding space. In ACL, pages 16124–16170. Association for Computational Linguistics.

Yao Fu, Hao Peng, Litou Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, pages 10421–10430. PMLR.

Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. 2024. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093.

Mor Geva, Jasmin Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216–12235, Singapore. Association for Computational Linguistics.

Mor Geva, Avi Caciularu, Kevin Ro Wang, and et al. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In EMNLP, pages 30–45. Association for Computational Linguistics.

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021a. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361.

Mor Geva, Roei Schuster, Jonathan Berant, and et al. 2021b. Transformer feed-forward layers are key-value memories. In EMNLP, pages 5484–5495. Association for Computational Linguistics.

Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. 2023. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969.

Michael Hanna, Ollie Liu, and Alexandre Variengien. 2023. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. arXiv preprint arXiv:2305.00586.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. arXiv preprint arXiv:2210.11610.

Stefanie Krause and Frieder Stolzenburg. 2023. Commonsense reasoning and explainable artificial intelligence using large language models. In European Conference on Artificial Intelligence, pages 302–319. Springer.

Jiachun Li, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Daojian Zeng, Kang Liu, and Jun Zhao. 2024. Focus on your question! interpreting and mitigating toxic CoT problems in commonsense reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9206–9230, Bangkok, Thailand. Association for Computational Linguistics.

Tom Lieberum, Matthew Rahtz, J´anos Kram´ar, G. Irving, Rohin Shah, and Vladimir Mikulik. 2023. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. ArXiv, abs/2307.09458.

Tom Lieberum, Senthooan Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, J´anos Kram´ar, Anca Dragan, Rohin Shah, and Neel Nanda. 2024. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Neural Information Processing Systems.

nostalgebraist. 2021. Interpreting gpt: The logit lens.

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. https://distill.pub/2020/circuits/zoom-in.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895.

Francesco Ortu, Zhijjing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Sch¨olkopf. 2024. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8420–8436, Bangkok, Thailand. Association for Computational Linguistics.

Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. 2024. A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.

Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´e, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykhov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart M. Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In NeurIPS.

Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, and et al. 2023. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In ICLR.

Yifei Wang, Yuheng Chen, Wanting Wen, Yu Sheng, Linjing Li, and Daniel Dajun Zeng. 2024. Unveiling factual recall behaviors of large language models through knowledge neurons. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7388–7402, Miami, Florida, USA. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.

Yuxin Xiao, Chaoqun Wan, Yonggang Zhang, Wenxiao Wang, Binbin Lin, Xiaofei He, Xu Shen, and Jieping Ye. 2024. Enhancing multiple dimensions of trustworthiness in llms via sparse activation control. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth International Conference on Learning Representations.

Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. 2024. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10210–10229, Bangkok, Thailand. Association for Computational Linguistics.

Lei Yu, Meng Cao, Jackie CK Cheung, and Yue Dong. 2024. Mechanistic understanding and mitigation of language model non-factual hallucinations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7943–7956, Miami, Florida, USA. Association for Computational Linguistics.

Qinan Yu, Jack Merullo, and Ellie Pavlick. 2023. Characterizing mechanisms for factual recall in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9924–9959.

Zeping Yu and Sophia Ananiadou. 2024. Neuron-level knowledge attribution in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3267–3280, Miami, Florida, USA. Association for Computational Linguistics.

Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. 2024. Interpreting and improving large language models in arithmetic calculation. arXiv preprint arXiv:2409.01659.

Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. 2020. Evaluating commonsense in pretrained language models. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9733–9740.

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.
A Appendix

A.1 Discussion about Interpreting Techniques
In this section, we provide a comprehensive review of existing interpretation tools and present our rationale for selecting specific approaches to analyze factual recall in real-world question answering systems. We conclude with a critical discussion of the limitations inherent in these analytical frameworks. The section concludes with a critical examination of these interpretative tools’ limitations, particularly focusing on their underlying simplifying assumptions and their implications.

Existing tools: (1) Logit attribution (or logit lens) projects intermediate activations to vocabulary space, enabling researchers to understand how predictions evolve across layers. (2) Probing techniques employ shallow classifiers to detect specific information encoded in model activations, though they only reveal correlational rather than causal relationships. (3) Sparse Autoencoders (SAEs) help discover independent features from superposed representations by mapping activations to a higher-dimensional sparse space. (4) Visualization tools aid in hypothesis generation and qualitative analysis, particularly for understanding attention patterns and neuron activations. (5) Automated feature explanation leverage LLMs themselves to automatically generate and validate feature labels, reducing the need for manual annotation while providing quantitative measures of explanation quality. (6) Knockout: To identify crucial model components, knockout/ablation methods systematically remove or modify specific parts while observing behavioral changes. (7) Activation patching, helps locate important components and connections within model circuits by comparing clean and corrupted runs. We recommend reading paper (Rai et al., 2024) for a comprehensive overview of mechanistic interpretability in LLMs. (8) Information flow analysis, based on Taylor expansion, provides insights into how information propagates between tokens through attention mechanisms, though its correlational nature limits causal interpretability.

This paper focuses on understanding the internal mechanisms of how the model performs commonsense reasoning. Specifically, we studied how the model generates the final answers using complex rationales. We break down this problem into three sub-problems: 1) Locating the key token positions in the rationale; 2) For each key token, identifying the key components during its generation; 3) Analyzing the behavior of the key components. To address each of these problems, we select distinct tools from the candidate tools above. Tab. 5 provides justifications for our choices.

Simplifying assumptions of the tools While our interpretability analysis yields valuable insights, it is important to acknowledge the inherent limitations of the tools employed. These tools necessarily operate under certain simplifying assumptions. In Tab. 6, we systematically summarize the assumptions and reliability of these tools. These simplifying assumptions, while potentially not capturing all nuances of LLM reasoning, provide a structured framework for investigating specific aspects of rationale generation within LLMs. Based on this, the analysis results are reliable, especially when combined with the verification method outlined in Wang et al. (2023); Lieberum et al. (2023). The only problem may lie in the lack of comprehensiveness in interpreting the whole reasoning procedure.

In summary, based on the assumptions of circuits, similar embedding spaces, and sparse feature representation, we utilized these interpretability tools to explain the mechanism within the reasoning process. Indeed, some potential complex mechanisms should remain uncovered, necessitating the design of more suitable tools and methods. Addressing this will be a focus of our future work.

A.2 Improvements to the Interpreting Tools
Due to the complex challenge (multi-token rationale and dense token connection) of interpreting the reasoning mechanisms, directly applying these tools to interpret the commonsense reasoning process is infeasible. Therefore we made many improvements to existing interpreting tools. In Tab. 7, we outline the key problems we encountered with these tools and the modest improvements we made to address them. Specifically, our detailed improvement to activation patching and SAE are depicted in §A.2.1 and §A.2.2 respectively.

A.2.1 Activation Patching Details
Counterfactual data generation We use GPT-4 to assist in automatically generating the counterfactual data required for activation patching, with the prompt shown in Fig. 7 and an example in Tab. 9. Additionally, we implement a post-processing step: if the predicted token for the counterfactual data matches the prediction for the data under investigation (which would fail to perturb the model’s
Table 5: Analysis of interpretability tools selection for different investigation problems

<table>
  <tr>
    <th>Problems</th>
    <th>Candidate Tools</th>
    <th>Selected Tools</th>
    <th>Justification for not Using Other Tools</th>
    <th>Justification for Using the Selected Tools</th>
  </tr>
  <tr>
    <td>Locate the key token positions in the rationale of commonsense reasoning</td>
    <td>Activation patching, information flow</td>
    <td>Activation patching</td>
    <td><b>Not information flow:</b> i) Lack of causality: Information Flow applies Taylor expansion to assess the importance of connections between tokens, which lacks causal interpretation. <b>Partial interpretability:</b> our preliminary experiments indicate that the key heads identified using the Information Flow do not offer good interpretability. Furthermore, knocking out these heads has no significant impact on the model’s predictions.</td>
    <td><b>Use activation patching:</b> i) Causality: activation patching pinpoints the key attention heads using causal analysis by directly modifying activations and observing changes in the model’s output. <b>Partial interpretability:</b> Many works have successfully interpreted the model’s specific behavior using activation patching, such as mathematical calculation (Zhang et al., 2024), multi-choice question answering (Lieberum et al., 2023), and sycophancy (Chen et al., 2024)</td>
  </tr>
  <tr>
    <td>Identify the key components for commonsense reasoning</td>
    <td>Probing, logit attribution, activation patching</td>
    <td>Logit attribution</td>
    <td><b>Not probing:</b> i) Hard to Probe Diverse Knowledge: probing involves an external classifier, and the outputs of knowledge recall tasks are difficult to categorize due to the diversity of knowledge. ii) Correlation not causality: Probing passively analyzes correlation rather than causal relationships. <b>Not activation patching:</b> high cost (5 minutes to analyze a sample on a TB model using a single A100 GPU.)</td>
    <td><b>Use logit attribution:</b> high efficiency: logit attribution has proven to be an efficient tool to identify and analyze the key components within LLMs in many works (Wang et al., 2023; Lieberum et al., 2023; Zhang et al., 2024; Vu et al., 2023).</td>
  </tr>
  <tr>
    <td>Analyze the behavior of the key components in commonsense reasoning</td>
    <td>Logit attribution, automated feature explanation, SAE, knockout</td>
    <td>Logit attribution, SAE, knockout</td>
    <td><b>Not neuron activation pattern explanation:</b> i) polysemanticity: neurons may not be easily explainable because they often encapsulate multiple features and can be polysemantic. ii) <b>Interpretability illusion:</b> neuron analysis often focuses on top-activating dataset examples, which may not reflect the interpretability by neglecting the neuron’s varied behaviors across different activation levels</td>
    <td><b>Use the combination of three tools:</b> Logit attribution is an efficient analyzing tool but may cause false interpretation results. Therefore we introduce SAE and knockout, which are used to verify the correctness of the key components. We use SAE to decode semantic information in the neuron and verify whether the semantic consistency. We use knockout to verify that the identified module can actually affect the generated result. If the results are not consistent, the identified module from the Logit attribution will be discarded. Combining these two tools can further improve the reliability of the identified modules and the analysis of their behavior by removing noisy components.</td>
  </tr>
</table>

behavior). GPT-4 is prompted to regenerate the counterfactual data.

We conduct experiments to compare the performance of “GPT-4” and “human”. In our study, we recruited ten graduate students pursuing master’s degrees in NLP as participants. Five students were assigned to manually generate counterfactual data, while the remaining participants were tasked with comparing these human annotations against GPT-4-generated counterparts. The evaluation focused on determining which method could effectively perturb model behavior with minimal alterations to the original input. Overall, the results (Table 8) demonstrate that GPT-4 is highly accepted by human evaluators, with the combination of “GPT wins” and “Ties” exceeding 80%, underscoring its robust reliability. These indicate that <i>GPT-4’s outputs are almost consistent with those generated by humans</i>.

<b>Activation patching metric</b> We design a special metric to evaluate the causal effect: predicted token’s probability divided by the sum of probabilities of the top \( k \) tokens (\( k = 10 \)). For example, when LLM is generating the next token for input “<i>Harry Potter is a</i>”, the top \( k \) predicting tokens include “fictional, wizard, British, ...”. Then the metric is:

\[
\frac{\text{prob}(fictional)}{\text{prob}(fictional) + \text{prob}(wizard) + \text{prob}(...)}.
\]

While directly using the logit change of “fictional” leads to the identification of irrelevant modules that increase all the logits of “fictional, wizard, British, ...”.

<b>Trace the information source within the middle layers</b> While standard activation patching reveals causal effects by modifying network activations and observing changes in the model’s output, we identified a limitation during our experiments. We discovered that answer-relevant information (e.g., “no”) emerges strongly in intermediate layers at reasoning adverb positions \( \mathcal{R} \) (e.g., “Thus”), yet becomes almost imperceptible in the final layer output (as shown in Fig. 14b). This phenomenon renders traditional activation patching ineffective for tracing information sources within middle layers, as it relies solely on observing the model’s final output. To address this limitation, we pro-
<table>
  <tr>
    <th>Tools</th>
    <th>Simplified Assumption</th>
    <th>Reliability of the Assumption</th>
  </tr>
  <tr>
    <td>Activation Patching</td>
    <td><b>Analogous circuits assumption:</b> Model can be viewed as a computational graph M where nodes are terms in its forward pass (neurons, attention heads, embeddings, etc.) and edges are the interactions between those terms (residual connections, attention, projections, etc.), a circuit C is a subgraph of M responsible for some behavior. <i>Analogous circuits appear across different models and tasks, which suggests that neural networks tend to converge on similar mechanisms for solving similar problems.</i></td>
    <td>
      <b>1. Analogous features across different vision models:</b> Certain low-level features, such as Gabor filters and curve detectors, reliably appear in early layers across multiple vision models (e.g., AlexNet, InceptionV1, VGG19, ResNet) trained on different datasets like ImageNet and Places365 (Olah et al., 2020). This suggests that neural networks converge on similar representations for visual tasks. <b>2. Task-specific circuits identified in LLMs:</b> Many works have identified the circuit for different tasks (e.g. Indirect object identification (Wang et al., 2023), multiple-choice question answering (Lieberum et al., 2023) and greater-than computation (Hanna et al., 2023)), which indicates analogous circuits performing certain tasks appear across different models and tasks.<br>
      <b>3. Function-specific components identified in LLMs:</b> Research has identified elements like induction heads, which are specialized mechanisms that enable models that perform specific functions such as copying patterns from prior sequences (Olsson et al., 2022) and neurons correlate with specific grammatical features are discovered in (Geva et al., 2022).
    </td>
  </tr>
  <tr>
    <td>Logit Attribution</td>
    <td><b>Projection assumption:</b> the outputs of each module in the model can be projected into the vocabulary space via the unembedding matrix to encode the semantic information within hidden states.</td>
    <td>A bunch of work has proved the feasibility of understanding the hidden state and weights within transformer-based LLMs through projection to the vocabulary space (Wang et al., 2023; Lieberum et al., 2023; Geva et al., 2022).</td>
  </tr>
  <tr>
    <td>Sparse Autoencoder (SAE)</td>
    <td><b>Sparse representation assumption:</b> The hidden state within models can be efficiently represented by a small number of interpretable salient monosemantic features</td>
    <td>The SAE is based on sparse dictionary learning. Many works (Lieberum et al., 2024; Gao et al., 2024) have trained SAE to decompose and explain the hidden state within LLMs and prove SAE to be an efficient interpreting tool.</td>
  </tr>
</table>

Table 6: Comparison of Different Tools and Their Assumptions

<table>
  <tr>
    <th>Interpreting stage</th>
    <th>Sub steps</th>
    <th>Interpreting Tools</th>
    <th>Specific Problems</th>
    <th>Our Modest Improvement</th>
    <th>Detailed illustrations</th>
  </tr>
  <tr>
    <td>Locate the key token positions</td>
    <td>Generate \( x_r, x_c \) pairs</td>
    <td>Activation patching</td>
    <td>High labor cost to scale the results</td>
    <td>Automate this process using GPT-4 to get scalable results</td>
    <td>When explaining how model generates "fictional (character)" given "Harry Potter is a", we prompt GPT-4 to generate counterfactual data</td>
  </tr>
  <tr>
    <td>Locate the key token positions</td>
    <td>Compute logit change of key tokens</td>
    <td>Activation patching</td>
    <td>Noisy metric</td>
    <td>Design a special metric to evaluate the causal effect</td>
    <td>Using the same case above, the top k predicting tokens include "fictional, wizard, British..."</td>
  </tr>
  <tr>
    <td>Locate the key token positions</td>
    <td>Trace the information source within the middle layers</td>
    <td>Activation patching</td>
    <td>Unable to trace the source of middle layer information</td>
    <td>Design a metric to examine perturbations effect</td>
    <td>We identify rich answer-related information responsible for generating conclusion token</td>
  </tr>
  <tr>
    <td>Identify the key components</td>
    <td>Choose the target tokens for observation</td>
    <td>Logit attribution</td>
    <td>False identification of the key modules</td>
    <td>Introduce probabilities of candidate tokens as comparison</td>
    <td>Using the same case above, we identify the modules where logit attribution of predicted token is high</td>
  </tr>
  <tr>
    <td>Analyze the behavior of the key components</td>
    <td>Verify the reliability of key components</td>
    <td>Logit attribution</td>
    <td>Projection assumption failure</td>
    <td>Use SAE to decode semantic information</td>
    <td>Use the same case above, we use logit attribution to evaluate each modules' contribution</td>
  </tr>
  <tr>
    <td>Analyze the behavior of the key components</td>
    <td>SAE Training</td>
    <td>SAE</td>
    <td>High computation cost</td>
    <td>Logit attribution is first used to identify key layers</td>
    <td>For the LLaMA model, training SAE for MLPs of every layer requires substantial resources</td>
  </tr>
  <tr>
    <td>Analyze the behavior of the key components</td>
    <td>Evaluating the relevance between SAE features and reasoning task</td>
    <td>SAE</td>
    <td>High labor cost to select the feature</td>
    <td>Use GPT-4 to automatically analyze correlation</td>
    <td>When using SAE decomposing the output of key MLP, we use GPT-4 to select object-related features</td>
  </tr>
</table>

Table 7: Interpreting Tools and Improvements
pose an enhanced activation patching framework: when investigating information sources at layer \( k \), we iteratively corrupt the output of each attention head from layers 0 through \( k \) using activations from counterfactual data. By measuring the detrimental effects on target token probabilities in the layer \( k \) residual block output, we can identify key attention heads and effectively map the information flow across intermediate layers.

A.2.2 SAE Details

Mechanism of SAE: Based on dictionary learning, SAE translates the hidden states of LLMs into several interpretable pieces, or termed features. These features are activated on sparse token sequences with specific patterns, and most can be interpreted by GPT-4 (Lieberum et al., 2024) into concrete semantic descriptions.

Strategic Training of SAE: While Google has publicly released SAE checkpoints for all layers of Gemma-7B, such resources remain unavailable for the LLaMA-2 model family. Consequently, we undertook the task of training our own SAE models. Given the substantial computational requirements, we strategically limited our training to specific MLP layers (layers 16 and 20) that are crucial for object reranking.

The training code for our Sparse Autoencoder (SAE) builds upon the open-source implementation provided by OpenAI (https://github.com/openai/sparse_autoencoder), which employs a Top-K activation function to maintain sparse latent representations. Our training configuration utilized 2 billion tokens from the Pile dataset, structured in 64-token sequences. The SAE architecture incorporates 512,000 latent variables with a Top-K activation parameter of 32. We implemented a distributed training setup with tensor parallelism of 2 and data parallelism of 8, processing batches of 131,072 tokens. The learning rate was set to 1.24e-4, determined through scaling laws derived from the GPT-2 architecture. The entire training process consisted of a single epoch. The computational requirements were still substantial: generating MLP outputs for the LLaMA2-7B model across 2 billion tokens consumed approximately 5 hours on 64 A100 GPUs, while the subsequent SAE training phase required an additional six hours utilizing 16 A100 GPUs.

SAE feature relevance evaluation: We primarily use SAE to investigate the information contained in the MLP and residual block outputs at the concept token position. Specifically, we selected the top 64 activated features (Top-64) based on SAE activations. Since these features include a substantial number of general-purpose activations (e.g., those representing syntax, specific words, etc.), we employed GPT to automatically analyze whether these activated features are related to the concept. The prompt used for this analysis is provided in Fig. 8.

<table>
  <tr>
    <th>GPT-4 Wins</th>
    <th>Human Wins</th>
    <th>Ties</th>
  </tr>
  <tr>
    <td>8%</td>
    <td>12%</td>
    <td>80%</td>
  </tr>
</table>

Table 8: Comparison of differences between GPT-4 and human annotations for counterfactual data generation.

A.3 Details of Representation Engineering

Representation engineering serves as a downstream application of our interpretability results, primarily to verify their reliability. This technique enables behavioral adjustments of the model through targeted modifications of internal representations. For instance, Templeton et al. (2024) demonstrated how introducing a security-related feature into the model’s middle layer residual stream can guide it toward generating safer content. Our experimental protocol consists of four key steps: (1) Object Identification: Leveraging the ground truth rationales provided in the StrategyQA dataset, we employ GPT-4 to detect cases of incorrect object retrieval by the model. In instances where errors are identified, we determine the correct objects that should have been retrieved. (2) Layer Selection: We target the MLP layer that exhibits significant contribution to the retrieval of predicted objects (\( O_p \)). Specifically, we focus on layer 36, which corresponds to the peak responsibility for object reranking, as shown in Fig. 3b. (3) SAE Feature Selection: We decompose the hidden state at the factual knowledge prediction position using SAE. To identify steering-relevant features, we employ GPT-4 for automated assessment of feature relevance to the correct factual knowledge (detailed methodology in A.2.2), selecting the most pertinent feature for modification. (4) Magnitude Calibration: Through grid search across a range of 1-10, we empirically determine the optimal perturbation magnitude, settling on \( k = 5 \) for our interventions.
Prompt Template for Counterfactual Data Generation

<Inputs><topic> The particular topic being studied</topic>
<input_sentence> The original sentence provided for analysis</input_sentence>
<predicted_content> The specific words reflecting model behavior</predicted_content>
<first_word_predicted> The first word initially predicted by the model</first_word_predicted></Inputs>

<Instructions Structure>
1. Instruct the assistant to begin by analyzing the original input sentence and why it leads to the specific predicted word.
2. Guide the assistant to think about changes that could alter the model’s prediction.
3. Instruct the assistant to modify the original sentence so that the model’s prediction changes.
4. Request the assistant to modify the original sentence so that the model’s prediction changes.
5. Instruct the assistant to explain the modification’s rationale, focusing on why the modified sentence now influences a different predicted outcome.
6. Ensure the output is formatted in the specified JSON structure.
</Instructions Structure>

<Instructions>
Your task is to analyze and modify a sentence to influence the predictive behavior of a language model. You will be given a topic, an input sentence, the specific words predicted by the model, and the model’s first predicted word.

Here is the topic and input sentence to modify: <topic>($TOPIC)</topic> <input_sentence>{$INPUT_SENTENCE}</input_sentence>

Here are the words generated by model given the input sentence: <predicted_content>{$PREDICTED_CONTENT}</predicted_content>

Here is the first predicted word:
<first_word_predicted>{$FIRST_WORD_PREDICTED}</first_word_predicted>

Follow these steps carefully to complete the task:

1. **Analyze the Original Prediction**: Start by understanding the **input sentence** and why it leads the model to predict the **first_word_predicted** as the output under the specific **topic**. Consider the context, tone, or structure of the sentence that prompts this specific prediction.
2. **Plan the Modification**: Think about how you could change the **input_sentence** minimally (by changing only 3-4 words) to alter the model’s behavior so that it no longer predicts the original word or instead predicts a word with an opposite meaning. It’s acceptable to change some of the sentence’s meaning if it helps influence the output.
3. **Prepare Analysis and Model Input**:
   - Write the **reason for the original prediction** based on your analysis in Step 1.
   - Rewrite the **input_sentence** in a modified form that will change or flip the model’s predicted word.
   - Explain your **reason for the modification**, focusing on how the changes you made will influence the model to predict a different word.
4. **Output the Final Results**: Format your response in JSON, as shown below:

```json
{
    "Reason for original prediction": "Explain why the original input caused the model to predict the initial word.",
    "Modified input": "Write the modified sentence here.",
    "Reason for modification": "Explain why the modified input will lead to a different prediction from the model."
}
```
Make sure each section is clear and precise. End your response with this JSON structure.
</Instructions>

Figure 7: Prompt for using GPT-4 to generate counterfactual data in activation patching.

A.4 Stability Analysis of Interpretability Results

To validate the robustness of our findings, we conducted comprehensive scaling experiments. Take the logit attribution of the answer generation stage in StrategyQA as an example. We examined different sample sizes (50, 100, and 1000 instances) and included an additional random resampling of 1000 instances. As illustrated in Fig. 11, the progression patterns of correct and false answers across different model components (Residual block, Attention, and MLP) remain remarkably consistent regardless of sample size. Specifically, the characteristic peaks in the attention mechanism around layer 34 and the distinctive MLP activation patterns in the later layers (30-40) are preserved across all sample sizes. The resampled 1000-instance experiment further corroborates these findings, exhibiting nearly identical behavioral patterns to the original 1000-instance sample. This consistency across different sample sizes and random resampling strongly suggests that our choice of 1000 instances provides a reliable representation of the model’s behavior patterns. Moreover, the clear separation between correct and false answer trajectories remains stable across all experimental conditions, indicating that our interpretability findings are not artifacts of sample size but rather reflect genuine computational patterns within the model. While we demonstrate this stability using the answer generation phase, similar consistency is observed in other reasoning stages.

A.5 Tracing from Answer \( \mathcal{A} \) to Object \( \mathcal{O} \)

We found that the attention heads responsible for generating \( \mathcal{A} \) primarily focus on the conclusion token \( \mathcal{C} \), as demonstrated by the pattern of head 25.08 in Tab. 10. Therefore, we traced back to the \( \mathcal{C} \). Fig. 14a shows the probabilities of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) in the residual block, attention, and MLP outputs
Table 9: Example of probing data \( X_r \) and counterfactual data \( X_c \) generated by GPT-4. Counterfactual data change the model (Gemma2-9B) prediction behavior by applying minimal change to the probing data.

<table>
  <tr>
    <th>Data</th>
    <th>Model Input</th>
    <th>Model Predict</th>
  </tr>
  <tr>
    <td>\( X_r \)</td>
    <td>Question: Kendall opened their mouth to speak and what came out shocked everyone. How would you describe Kendall? (1) a very quiet person (2) a very passive person (3) a very aggressive and talkative person Answer: Kendall opened their mouth to speak and what came out shocked everyone. Thus, Kendall is a very __</td>
    <td>aggressive</td>
  </tr>
  <tr>
    <td>\( X_c \)</td>
    <td>Question: Kendall opened their mouth to speak and what came out <b>was softer than expected</b>. How would you describe Kendall? (1) a very quiet person (2) a very passive person (3) a very aggressive and talkative person Answer: Kendall opened their mouth to speak and what came out <b>was softer than expected</b>. Thus, Kendall is a very __</td>
    <td>quiet</td>
  </tr>
</table>

at the conclusion token position. It is evident that the model distinguishes the correct answer \( \mathcal{A}_t \) in the deep layers, with both the attention and MLP outputs containing substantial information related to \( \mathcal{A}_t \).

Next, we identified the heads for generating \( \mathcal{C} \) using activation patching and discovered that the key attention heads primarily focus on the reasoning conjunctive adverb \( \mathcal{R} \) (i.e., “<i>Thus</i>” in head 31.03 pattern in Tab. 10). We also observed that the attention head outputs contain information related to the correct answer \( \mathcal{A}_t \), such as “yes,” “indeed,” and “true.” Based on these findings, we conducted further probing at \( \mathcal{R} \) to trace the origin of \( \mathcal{A}_t \).

Through decoding information of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at \( \mathcal{R} \) (Fig. 14b), we find that deep layers (30 – 34) already encode rich information related to the correct answer \( \mathcal{A}_t \). To trace the origin of the answer-related information, we employed a modified activation patching to identify the key Attention heads. Specifically, we iteratively corrupted the output of each attention head from layer 0 – 30 using the activation in counterfactual data, then identified the key attention heads that have a significant negative influence on the probability of \( \mathcal{A}_t \) in residual block (layer 30) output. Three key Attention heads (25.7, 25.8 and 25.9) are identified that primarily focus on the position of attribute \( \mathcal{A} \) (e.g., “company”). From the observation above, we conclude a key finding: <i>reasoning conjunctive adverbs serve as an anchor for gathering and transferring conclusion-related information in reasoning process.</i> Therefore, our investigation continuously traces back to the position of object \( \mathcal{O} \) prediction.

A.6 Results on CommonsenseQA and SocialIQA

We further apply our interpreting method to CommonsenseQA (Talmor et al., 2018) and SocialIQA (Sap et al., 2019) and find the model’s reasoning process within these two datasets consists of <b>attribute retrieval, attribute rerank, and answer generation</b> as shown in Fig. 9. Similarly, we start by decoding the probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at the position of predicting answer \( \mathcal{A}_t \). The decoding curve of CommonsenseQA is in Fig. 20b and SocialIQA result is in Fig. 22b. It is observed that the information trend in residual block, Attention, and MLP is similar across the two datasets. Specifically, the probability of \( \mathcal{A}_t \) increases significantly at layer 30, while Attention output encodes \( \mathcal{A}_t \) related information before layer 30 and \( \mathcal{A}_t \) relate information emerges in MLP at layer around 32. Therefore, we conclude the answer generation process as follows: attention is responsible for copying and generating \( \mathcal{A}_t \) related information and MLP is responsible for augmenting this information. Through backtracing, we identified the key heads for generating the correct answer (see key head distribution in Fig. 19b and 21b). As shown in Tab. 11, we find the head output encodes rich information related to the correct answer and mainly attends to the object in rationale and choices in question. Therefore, we first trace back to the position of \( \mathcal{C} \).

Since both datasets are in the form of multiple-choice questions, the answer (object) is already provided as one of the options. Therefore, we treat the correct answer as the predicted object \( \mathcal{O}_p \) and the other options as candidate objects \( \mathcal{O}_c \). The logit attribution curves for \( \mathcal{A}_t \) and \( \mathcal{A}_f \) are shown in Fig. 20a and 22a for CommonsenseQA and SocialIQA respectively. As shown in the figure, the attention output contains both \( \mathcal{O}_p \) and \( \mathcal{O}_c \), while
Prompt Template for SAE feature filtering

Instruction: evaluate the relevance of a feature that activates on certain texts to the concept of '{concept}'.

You will be provided with a possible explanation of the feature, a set of texts where the feature has been activated, along with the most activated word(s) in each text.

<$<Example>$>
Concept: 'Environmental Protection'

Possible explanation of the feature: feature identifies texts related to protecting the natural environment.

Activated texts and most activated word:
- must take action to reduce carbon emissions and combat climate change. | most activated word: emissions
- Deforestation is a major threat to biodiversity and attributes to global warming. | most activated word: deforestation
<$<Example>$>

<$<Expected Output>$>
The feature is highly relevant to the concept of environmental protection as it identifies texts discussing environmental issues and solutions.
Relevance Score: 10
<$<Expected Output>$>

Based on the given explanation of the feature and the activated texts, please rate the relevance of the feature to the concept of {concept} on the scale below:
- 0: Not at all relevant, the feature is not related to the concept.
- 5: Neutral, the feature is not directly related to the concept but share some common traits with the concept, e.g. apple and banana are both fruits.
- 10: Very relevant, the feature is directly related to the concept.

Please conclude your response in the following format:
'Relevance Score: [SCORE]', where [SCORE] is an integer between 0 and 10.

Here is the concept: {concept} and the explanation of the feature:

Concept: {concept}

Possible explanation of the feature: {explanation}

Activated texts and most activated word: {texts}
</Instructions>

Figure 8: Prompt for using GPT-4 to evaluate the relevance between feature and task.

the MLP output only contains the \( \mathcal{O}_p \). This finding aligns with our previous discovery on StrategyQA regarding the object retrieval and reranking mechanism: attention heads first aggregate all relevant objects, and then the MLP ranks these objects based on their relevance, selecting the \( \mathcal{O}_p \) for the final output. These results further validate the generalizability of our approach and findings.

Finally, we used activation patching to identify the key attention heads responsible for generating \( \mathcal{O}_p \). The distribution of important heads is shown in Fig. 19a and 21a. We found that the key heads primarily focus on the options in the question (see head pattern of 34.14 in Tab. 11), which serve as the source for all objects. With this, the complete reasoning process is concluded.

A.7 Results of Logit Attribution on Failure Cases

Our interpretability framework was applied to analyze cases of Reference Errors, where the model retrieves irrelevant or incorrect objects. As shown in Fig. 13, several noteworthy patterns emerge: First, although the model ultimately outputs incorrect objects, information related to the correct answer remains present in the final layers. This observation is also supported by our examination of the model’s top-5 token predictions, which consistently include the correct object among the candidates, albeit not as the primary prediction. Second, we observe a distinct pattern in the information flow: Layer 35’s attention mechanisms exhibit strong signals related to the correct object (a pattern absent in earlier layers), while Layer 32’s MLP shows pronounced activation patterns associated with incorrect object. This pattern diverges from our previously observed object recall process in successful cases, where attention heads first gather relevant object information (object retrieval), and MLPs subsequently rerank the most pertinent object for output (object rerank).

Based on these observations, we hypothesize that Reference Errors stem from two primary mechanisms: (1) Insufficient information gathering by attention mechanisms in the middle-to-late layers (25–35), leading to incomplete object collection. (2) Incorrect reranking by MLP layers, which erroneously prioritizes irrelevant objects over pertinent ones. This mechanistic understanding informed our fine-tuning strategy, where we simultaneously target both attention heads and MLP layers for optimization.

A.8 Experiment Results on Llama2-7B

On Llama2-7B, we apply the same method to interpret the reasoning process in StrategyQA (Fig. 12), CommonsenseQA (Fig. 17), and SocialIQA (Fig. 22). Three phases of reasoning, i.e. subject augmentation and broadcast, object retrieval and rerank, conclusion fusion and generation are observed on StrategyQA. Similarly, object retrieval and rerank and conclusion generation are observed on CommonsenseQA and SocialIQA.

Taking StrategyQA as an example, the reasoning process can be divided into three distinct phases: (1) Subject Augmentation and Broadcast. At the subject token position (S), the shallow MLPs
Figure 9: Model inner reasoning process on CommonsenseQA.

![Model inner reasoning process on CommonsenseQA](page_184_120_1080_370.png)

(a) Trace back at \( \mathcal{O} \) (to \( \mathcal{S} \))   (b) Trace back at \( \mathcal{R} \) (to \( \mathcal{O} \))   (c) Trace back at \( \mathcal{C} \) (to \( \mathcal{R} \))   (d) Trace back at \( \mathcal{A} \) (to \( \mathcal{C} \))

Figure 10: Distribution of key heads during tracing back at different token positions on StrategyQA (averaged on 100 samples). The red squares indicate heads that have a significant positive impact on predicting the output token.

![Distribution of key heads during tracing back at different token positions on StrategyQA](page_184_563_1080_370.png)

(layers 6-8) extend from the subject to augment relevant object information. Subsequently, this augmented information is propagated to the question’s end position through attention mechanisms in the middle-to-late layers (20-26). (Fig. 12a and 12b)
(2) Object Retrieval and Rerank: During the prediction of object tokens (\( O_p \)), attention mechanisms (layers 19-25) are responsible for retrieving related objects, while MLPs (layers 19-22) rank and select the most appropriate one for output. (Fig. 12c) (3) Conclusion Fusion and Generation: Prior to answer prediction, the correct answer information is prepared in the middle layers (around layer 20) at the Reasoning adverb position (\( \mathcal{R} \)). This information is then transported to the Conclusion position (\( \mathcal{C} \)) through mid-layer attention mechanisms (layers 19-25), where mid-layer MLPs (layers 19-25) enhance the correct answer information to generate the conclusion. Finally, at the answer generation position, attention (layers 19-25) transfers the correct answer information from the conclusion position to produce the final answer output (\( \mathcal{A} \)). (Fig. 12e and 12f)

A.9 Details of SSFT

In selective supervised fine-tuning, a sequence of attention heads and MLPs ordered by their significance will be given, denoted as \((MLP.l_1), (Head.l_2.h_2), (Head.l_3.h_3), \ldots,\) where \( l_i \) represents the layer index and \( h_i \) represents the head index of the \( i^{th} \) ranked head, only parameters of top \( K \) heads and top \( M \) MLPs are exclusively updated during fine-tuning. We optimize both the corresponding input mapping matrix \( \{W_{l_1}^{h_1}, W_{l_2}^{h_2}, ..., W_{l_K}^{h_K}\} \) and the output mapping matrix \( \{O_{l_1}^{h_1}, O_{l_2}^{h_2}, ..., O_{l_K}^{h_K}\} \) in top \( K \) heads simultaneously. For the selected MLP layer, we update all parameters in this layer.

For the format of training data, following Fu et al. (2023) and Huang et al. (2022), each sample in our training data is organized with the format of “{Few-shot CoT prompt} Q: {Question}. A: {Rationale}”. We train the model using a learning rate of \( 5e-5 \) and a batch size of 32 for 2 epochs. For supervised fine-tuning, a learning rate of \( 1e-5 \) is utilized, while all other configurations remain consistent with SSFT training. Experiments are conducted on 8 NVIDIA A100 (80GB) GPUs. Given that both SocialIQA and CommonsenseQA datasets only provide answer labels without explicit reasoning rationales, we employed GPT-4 to synthesize reasoning rationales using the few-shot exemplars from Li et al. (2024). To ensure the quality of the synthesized rationales, we implemented a filtering mechanism that verifies the consistency between the answers within these generated rationales and the original labels.
<table>
  <tr>
    <th>Pos.</th>
    <th>Head</th>
    <th>Attention score</th>
    <th>Projection</th>
  </tr>
  <tr>
    <td>\( \mathcal{O} \)</td>
    <td>25.02</td>
    <td>
      Q: Is Canon Inc. a Kabushiki gaisha? newline-A: Canon Inc. is a Japanese company. Japanese companies are Kabushiki gaisha. Thus, Canon Inc. is a Kabushiki gaisha.
    </td>
    <td>Japan, Japanese, Jepang, Japón, japan, Tokyo</td>
  </tr>
  <tr>
    <td>\( \mathcal{R} \)</td>
    <td>25.08</td>
    <td>
      Q: Is Canon Inc. a Japanese company? newline-A: Canon Inc. is a Japanese company. Japanese companies are Kabushiki gaisha. Thus,
    </td>
    <td>confirmation, confirmación, Personendaten, verification</td>
  </tr>
  <tr>
    <td>\( \mathcal{S} \)</td>
    <td>31.03</td>
    <td>
      Q: Is Canon Inc. a Kabushiki gaisha? newline-A: Canon Inc. is a Japanese company. Japanese companies are Kabushiki gaisha. Thus, Canon Inc. is
    </td>
    <td>yes, Yes, indeed YES, true, Indeed</td>
  </tr>
  <tr>
    <td>\( \mathcal{A} \)</td>
    <td>25.08</td>
    <td>
      Q: Is Canon Inc. a Kabushiki gaisha? newline-A: Canon Inc. is a Japanese company. Japanese companies are Kabushiki gaisha. Thus, Canon Inc. is a Kabushiki gaisha. So the answer is
    </td>
    <td>confirmation, confirmación, confirmier, verification</td>
  </tr>
</table>

Table 10: Attention score of the key attention heads (on StrategyQA in Gemma2-9B) on different tokens and top-\( k \) tokens after projecting the output of heads into the vocabulary space. The attention heads are obtained according to the activation patching result in Figure 10. The term Head 25.02 denotes the 2nd head in the attention layer of the 25th layer of the model.

<table>
  <tr>
    <th>Pos.</th>
    <th>Head</th>
    <th>Attention score</th>
    <th>Projection</th>
  </tr>
  <tr>
    <td>\( \mathcal{O} \)</td>
    <td>34.14</td>
    <td>
      newline-Question: A crane uses many a steel cable when working a what?newline-(A) about (B) strip (C) which (D) construction site (E) building-newline-Answer: A crane is a machine that is used to lift and move heavy objects. It is usually used in construction.
    </td>
    <td>construction, Konstruktion, autorytatywna, Construction</td>
  </tr>
  <tr>
    <td>\( \mathcal{A} \)</td>
    <td>31.15</td>
    <td>
      newline-Question: How many a steel cable when working a what?newline-(A) about (B) strip (C) which (D) construction site (E) building-newline-Answer: A crane is a machine that is used to lift and move heavy objects. It is usually used in construction sites. So the answer is (D).
    </td>
    <td>construction, constructions, struction, traction, construcción</td>
  </tr>
</table>

Table 11: Attention score of the key attention heads (on CommonsenseQA in Gemma2-9B) on different tokens and top-\( k \) tokens after projecting the output of heads into the vocabulary space. The attention heads are obtained according to the activation patching result in Figure 19. The term Head 34.14 denotes the 14nd head in the attention layer of the 34th layer of the model.

Table 12: Examples of Reasoning Cases from StrategyQA, CommonsenseQA, and SocialIQA Datasets. Three datasets evaluate distinct aspects of reasoning capabilities: CommonsenseQA focuses on basic conceptual understanding, SocialIQA assesses social and emotional intelligence, and StrategyQA tests multi-hop reasoning abilities that require the integration of multiple pieces of evidence through complex inference chains. The answer is generated by Gemma2-9B. In CommonsenseQA and SocialIQA, the entities are often abstract names or professions with no specific meaning. Therefore, we treat the options in the context as attributes, the final predicted option as the predicted attribute, and the remaining options as candidate objects.

<table>
  <tr>
    <th>Dataset</th>
    <th>StrategyQA</th>
    <th>CommonsenseQA</th>
    <th>SocialIQA</th>
  </tr>
  <tr>
    <th>Question</th>
    <td>Can Harry Potter book a flight on Asiana Airlines?</td>
    <td>The artist was sitting quietly pondering, then suddenly he began to paint when what struck him? (A) sadness (B) anxiety (C) inspiration (D) discomfort (E) insights</td>
    <td>remy had a good talk with aubrey so aubrey understood remy better now. How would Remy feel as a result? (1) unsatisfied (2) calm (3) anxious</td>
  </tr>
  <tr>
    <th>Answer</th>
    <td>Harry Potter is a fictional character. Fictional characters cannot book flights. Thus, Harry Potter cannot book a flight on Asiana Airlines. So the answer is no.</td>
    <td>The artist was sitting quietly pondering, then suddenly he began to paint when <b>inspiration</b> struck him. So the answer is: (C) inspiration.</td>
    <td><b>Remy</b> had a good talk with Aubrey. Thus, Aubrey understands Remy better. Remy will feel <b>calm</b> as a result. So the answer is: (2) calm.</td>
  </tr>
  <tr>
    <th>Answer Type</th>
    <td>Yes / No</td>
    <td>Multiple Choice (C) inspiration</td>
    <td>Multiple Choice (2) clam</td>
  </tr>
  <tr>
    <th>Answer Token</th>
    <td>no</td>
    <td>artist</td>
    <td>Remy</td>
  </tr>
  <tr>
    <th>Subject</th>
    <td>Harry Potter</td>
    <td>inspiration</td>
    <td>calm</td>
  </tr>
  <tr>
    <th>Predicted Object</th>
    <td>fictional character</td>
    <td>sadness, anxiety, discomfort</td>
    <td>unsatisfied, anxious</td>
  </tr>
  <tr>
    <th>Candidate Object</th>
    <td>wizard, British, magic</td>
    <td></td>
    <td></td>
  </tr>
</table>
Figure 11: Scaling analysis of model behavior patterns across different sample sizes during answer generation on StrategyQA.

Figure 12: Logit attribution results on StrategyQA of Llama2-7B. (a) Probability of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at \( \mathcal{S} \). (b) Probability of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at the end of question (\( Q_e \)). (c) Probability of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at \( \mathcal{O} \) prediction. (d) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at \( \mathcal{R} \). (e) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at \( \mathcal{C} \) prediction. (d) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at \( \mathcal{A} \) prediction.

<table>
  <tr>
    <th>ID</th>
    <th>Feature Explanation</th>
  </tr>
  <tr>
    <td>115620</td>
    <td>Phrases related to confrontation and dynamics involving identity.</td>
  </tr>
  <tr>
    <td>99851</td>
    <td>References to characters and elements from the Harry Potter series.</td>
  </tr>
  <tr>
    <td>82918</td>
    <td>Concepts related to creation and storytelling in various media.</td>
  </tr>
  <tr>
    <td>114490</td>
    <td>Elements related to character dynamics and development in storytelling.</td>
  </tr>
</table>

Table 13: Top-scoring features decoded by SAE in the output of MLP at layer 37 when predicting \( \mathcal{O} \).

<table>
  <tr>
    <th>Layer</th>
    <th>ID</th>
    <th>Feature Explanation</th>
  </tr>
  <tr>
    <td>7</td>
    <td>106518</td>
    <td>References to specific characters and items from a fictional universe.</td>
  </tr>
  <tr>
    <td>7</td>
    <td>113897</td>
    <td>References to characters and locations from the Harry Potter series.</td>
  </tr>
  <tr>
    <td>32</td>
    <td>5548</td>
    <td>References to specific characters and events from the Harry Potter series.</td>
  </tr>
  <tr>
    <td>32</td>
    <td>94534</td>
    <td>References to the concept of "world" or "global" themes</td>
  </tr>
</table>

Table 14: Top-scoring features decoded by SAE in the output of MLP at layer 7 and 37 at \( \mathcal{S} \).
Figure 13: Comparative analysis of model behavior during reference errors. The plots show logit attributions for correct (orange) and wrong (blue) objects across different layers in Residual blocks, Attention mechanisms, and MLPs.

Figure 14: Logit attribution results on StrategyQA of Gemma2-9B. (a) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) when predicting \( \mathcal{C} \). (b) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) when at \( \mathcal{R} \).

Figure 15: Logit attribution results on SocialIQA of Llama2-7B. (a) Probability of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at the position of predicting \( \mathcal{O} \). (b) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at the position of predicting \( \mathcal{A} \).

(a) Trace back at \( \mathcal{O} \)    (b) Trace back at \( \mathcal{A} \) (to \( \mathcal{O} \))

Figure 16: Distribution of key heads (Llama2-7B) during tracing back at different token positions on SocialIQA (averaged on 100 samples). The red squares indicate heads that have a significant positive impact on predicting the output token.

<table>
  <tr>
    <th>Head</th>
    <th>Top tokens in projection</th>
  </tr>
  <tr>
    <td>25.01</td>
    <td>Hogwarts, wizard, wizards, children,</td>
  </tr>
  <tr>
    <td>25.02</td>
    <td>Brito, British, London, Westminster-</td>
  </tr>
  <tr>
    <td>29.06</td>
    <td>book, chapters, books, Book, bookId</td>
  </tr>
  <tr>
    <td>29.14</td>
    <td>wizards, wizard, Hogwarts, Harry</td>
  </tr>
</table>

Table 15: Top-scoring tokens in the key attention heads output when predicting \( \mathcal{O} \). (i.e., “fictional character” for “Harry Potter”. )
Table 16: SSFT results using CommonsenseQA as the training dataset.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Tuned Params.</th>
    <th colspan="2">ID Task</th>
    <th colspan="4">OOD Task</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>CSQA<br>Acc. Δ</th>
    <th>Winogrande<br>Acc. Δ</th>
    <th>StrategyQA<br>Acc. Δ</th>
    <th>SocialIQA<br>Acc. Δ</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>-</td>
    <td>75.7 -</td>
    <td>61.2 -</td>
    <td>70.7 -</td>
    <td>73.0 -</td>
    <td>68.3 -</td>
  </tr>
  <tr>
    <td>+ SFT</td>
    <td>9B</td>
    <td>81.3 <b>+5.6</b></td>
    <td>59.8 -1.4</td>
    <td>71.0 <b>+0.3</b></td>
    <td>77.4 <b>-5.6</b></td>
    <td>66.1 <b>-2.2</b></td>
  </tr>
  <tr>
    <td>+ SSFT</td>
    <td>0.2B</td>
    <td>82.1 <b>+6.4</b></td>
    <td>65.1 <b>+3.9</b></td>
    <td>70.7 -</td>
    <td>74.3 <b>+1.3</b></td>
    <td>70.0 <b>+1.7</b></td>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>-</td>
    <td>61.1 -</td>
    <td>62.5 -</td>
    <td>53.4 -</td>
    <td>60.2 -</td>
    <td>58.7 -</td>
  </tr>
  <tr>
    <td>+ SFT</td>
    <td>6.7B</td>
    <td>72.3 <b>+11.2</b></td>
    <td>57.8 -4.7</td>
    <td>53.5 <b>+0.1</b></td>
    <td>55.7 <b>-3.0</b></td>
    <td>56.2 <b>-2.5</b></td>
  </tr>
  <tr>
    <td>+ SSFT</td>
    <td>0.2B</td>
    <td>73.5 <b>+12.4</b></td>
    <td>63.1 <b>+0.6</b></td>
    <td>56.2 <b>+2.8</b></td>
    <td>63.2 <b>+3.0</b></td>
    <td>61.8 <b>+3.1</b></td>
  </tr>
</table>

Table 17: SSFT results using SocialIQA as the training dataset.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Tuned Params.</th>
    <th colspan="2">ID Task</th>
    <th colspan="4">OOD Task</th>
    <th rowspan="2">Average</th>
  </tr>
  <tr>
    <th>SocialIQA<br>Acc. Δ</th>
    <th>Winogrande<br>Acc. Δ</th>
    <th>StrategyQA<br>Acc. Δ</th>
    <th>CSQA<br>Acc. Δ</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>-</td>
    <td>73.0 -</td>
    <td>61.2 -</td>
    <td>70.7 -</td>
    <td>75.7 -</td>
    <td>69.2 -</td>
  </tr>
  <tr>
    <td>+ SFT</td>
    <td>9B</td>
    <td>80.2 <b>+7.2</b></td>
    <td>59.0 <b>-2.2</b></td>
    <td>72.0 <b>+1.3</b></td>
    <td>72.1 <b>-3.6</b></td>
    <td>67.7 <b>-1.5</b></td>
  </tr>
  <tr>
    <td>+ SSFT</td>
    <td>0.2B</td>
    <td>81.1 <b>+8.1</b></td>
    <td>64.2 <b>+3.0</b></td>
    <td>70.9 <b>+0.2</b></td>
    <td>77.0 <b>+1.3</b></td>
    <td>70.7 <b>+1.5</b></td>
  </tr>
  <tr>
    <td>Llama2-7B</td>
    <td>-</td>
    <td>61.1 -</td>
    <td>62.5 -</td>
    <td>53.4 -</td>
    <td>60.2 -</td>
    <td>58.7 -</td>
  </tr>
  <tr>
    <td>+ SFT</td>
    <td>6.7B</td>
    <td>72.3 <b>+11.2</b></td>
    <td>57.8 -4.7</td>
    <td>53.5 <b>+0.1</b></td>
    <td>55.7 <b>-3.0</b></td>
    <td>56.2 <b>-2.5</b></td>
  </tr>
  <tr>
    <td>+ SSFT</td>
    <td>0.2B</td>
    <td>73.5 <b>+12.4</b></td>
    <td>63.1 <b>+0.6</b></td>
    <td>56.2 <b>+2.8</b></td>
    <td>63.2 <b>+3.0</b></td>
    <td>61.8 <b>+3.1</b></td>
  </tr>
</table>

Figure 17: Logit attribution results on CommonsenseQA of Llama2-7B. (a) Probability of \( O_p \) and \( O_c \) at the position of predicting \( O \). (b) Probability of \( A_t \) and \( A_f \) at the position of predicting \( A \).

![Logit attribution results on CommonsenseQA of Llama2-7B. (a) Probability of O_p and O_c at the position of predicting O. (b) Probability of A_t and A_f at the position of predicting A.](page_104_1042_627_312.png)

(a) Trace back at \( O \)   (b) Trace back at \( A \) (to \( O \))

Figure 18: Distribution of key heads (Llama2-7B) during tracing back at different token positions on CommonsenseQA (averaged on 100 samples). The red squares indicate heads that have a significant positive impact on predicting the output token.

![Distribution of key heads (Llama2-7B) during tracing back at different token positions on CommonsenseQA (averaged on 100 samples). The red squares indicate heads that have a significant positive impact on predicting the output token.](page_104_1372_627_312.png)
(a) Trace back at \( \mathcal{A} \) (to \( \mathcal{O} \))    (b) Trace back at \( \mathcal{O} \) (to choices)

Figure 19: Distribution of key heads (Gemma2-9B) during tracing back at different token positions on CommonsenseQA (averaged on 100 samples). The red squares indicate heads that have a significant positive impact on predicting the output token.

(a) Trace back at \( \mathcal{A} \) (to \( \mathcal{O} \))    (b) Trace back at \( \mathcal{O} \) (to choices)

Figure 21: Distribution of key heads (Gemma2-9B) during tracing back at different token positions on SocialIQA (averaged on 100 samples). The red squares indicate heads that have a significant positive impact on predicting the output token.

(a)    (b)

Figure 20: Logit attribution results on CommonsenseQA of Gemma2-9B. (a) Probability of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at the position of predicting \( \mathcal{O} \). (b) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at the position of predicting \( \mathcal{A} \).

(a)    (b)

Figure 22: Logit attribution results on SocialIQA of Gemma2-9B. (a) Probability of \( \mathcal{O}_p \) and \( \mathcal{O}_c \) at the position of predicting \( \mathcal{O} \). (b) Probability of \( \mathcal{A}_t \) and \( \mathcal{A}_f \) at the position of predicting \( \mathcal{A} \).
<table>
  <tr>
    <th>Input</th>
    <td>Question: John cannot run the entire length of the track, he had been used to the field. The _ is short.<br>(1) track (2) field<br>Answer: A person who cannot run the entire length of a track likely feels uncomfortable or out of practice on a surface that is different from what they are used to. If John had been used to the field, it suggests that he is more accustomed to that environment. Therefore, the track must be</td>
  </tr>
  <tr>
    <th>SSFT model output</th>
    <td><b>longer</b> than the field, making it difficult for him to run its entire length. ✔️</td>
  </tr>
  <tr>
    <th>Base model output</th>
    <td><b>shorter</b> than the field, as he struggles to run its entire length. ✗</td>
  </tr>
</table>

(a) Case study: output of SSFT and Base model

(b) Probing attention layer output for “shorter” and “longer” on SSFT model

(c) Probing attention layer output for “shorter” and “longer” on Base model

(d) Probing MLP layer output for “shorter” and “longer” on SSFT model

(e) Probing MLP layer output for “shorter” and “longer” on Base model

Figure 23: Comparison between the SSFT and Base models: (a) Case study highlights that the SSFT model correctly predicts the answer, while the Base model fails. (b, c) Probing results for attention layers show enhanced knowledge retrieval in the SSFT model compared to the Base model. (d, e) Probing results for MLP layers demonstrate improved reranking capability in the SSFT model. These findings confirm that the identified modules—attention heads for knowledge retrieval and MLP layers for reranking—are critical for accurate reasoning and were effectively strengthened through SSFT.

Reviewer paper summaries: This paper examines the internal reasoning processes of large language models (LLMs) in commonsense reasoning tasks, which remain challenging due to the complex interdependencies among generated tokens.
Summary of strengths: The paper addresses an important and underexplored aspect of LLMs: understanding their internal reasoning processes during commonsense reasoning tasks.

The study’s methodology and findings have the potential to inform future research on improving LLM interpretability and reasoning performance.
Summary of weaknesses: The study relies on three existing interpreting tools to analyze the internal reasoning processes of LLMs but does not provide sufficient justification or explanation for their selection. The reliability and limitations of these tools are not critically discussed, which undermines the credibility of the analysis. Interpreting tools often make simplifying assumptions that may not fully capture the complexities of LLM reasoning, and the paper does not address these potential shortcomings.

The final conclusions regarding the LLM reasoning process are summarized by humans and are based on a relatively small number of "bad cases." This approach introduces subjectivity and potential bias into the analysis, as it relies on human judgment and is limited by the scale of the cases the human reviewers examined.
Comments, suggestions and typos: The paper is thought-provoking and enjoyable to read. However, it would benefit from larger-scale empirical evidence with more cases to strengthen the robustness and generalizability of the interpreting results.
Overall assessment scores: 2.5