Deciphering and Enhancing Commonsense Reasoning in LLMs from the Perspective of Intrinsic Factual Knowledge Retrieval

Authors: Yiqun Wang, Sile Hu, Chaoqun Wan, Yonggang Zhang, Xiang Tian, Yaowu Chen, Xu Shen, Jieping Ye
Primary area: interpretability and explainable AI
Keywords: Model interpretability, Chain-of-Thought, Large Language Models

Abstract:
Commonsense reasoning in large language models (LLMs) bridges the gap to physical world, thus allowing them to think and behave more like humans. Previous research has shown that LLMs acquire the underlying factual knowledge from extensive training corpora and store it within their parameters. However, how LLMs apply this knowledge during the inference phase remains unclear. This lack of transparency makes it difficult to determine whether shortcomings in LLMs are due to a lack of factual knowledge or insufficient reasoning capabilities.
In this work, we aim to decipher the commonsense reasoning process into human-understandable steps. By interpreting the hidden states in different transformer layers and token positions, we uncover a specific mechanism by which LLMs execute reasoning.
Our extensive experiments indicate: 1) both attention head and multi-layer perceptron (MLP) contribute to the generation of factual knowledge from different perspective. 2) The process of commonsense reasoning in LLMs involves a clear sequence of knowledge augmentation, knowledge retrieval and answer generation, akin to retrieval-augmented generation.
Building on these findings, we have discovered that LLMs often contain relevant facutal knowledge but fail to retrieve the correct knowledge at top. To address this issure, we selectively fine-tuned the key heads and MLPs, resulting in notably improvements in reasoning performance in both in-domain and out-of-domain settings.

Decision: reject
Reviewer scores: 5, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730251233693

thread_type:
official_review

<thread_object>
creator: reviewer_4kuu

<content>
summary:
The paper employs techniques such as Path Patching, Logit Lens, and Sparse AutoEncoders (SAE) to interpret the internal mechanisms of large language models (LLMs) in commonsense reasoning. Specifically, the authors find that: (1) certain positions in the LLM's internal states augment attribute information; (2) later tokens also encode significant attribute information across multiple layers; (3) deeper MLP layers are responsible for retrieving and reranking relevant knowledge; and (4) the final answer is largely a result of transferring and consolidating this accumulated previous information.  
Building on these, they analyze the error distribution and discover that most errors stem from the model’s failure to retrieve relevant knowledge. As a result, they selectively fine-tune specific attention heads and MLP layers to enhance the model's ability to retrieve and rerank knowledge. Through experiments they show that the proposed SSFT (Selective Supervised Fine-Tuning) technique outperforms standard SFT while using less than 10% of the model’s parameters.

strengths:
1)	The paper provides a detailed analysis of how the internal states of LLMs function during commonsense reasoning, making the mechanisms of LLMs more transparent. The reviewer believes this will interest a broad audience within the community and serve as a foundation for future research.

2)	The paper introduces a novel fine-tuning technique (SSFT) that selectively fine-tunes the most critical attention heads and MLP layers, achieving improved performance with significantly reduced computational resources.

weaknesses:
1.	The methodology section could benefit from additional details, particularly in Section 3.2, which covers the interpreting module. Providing more explanation on how each of the modules works would be helpful for readers who are not familiar with these techniques, as currently, they may need to consult other papers. The reviewer suggests that adding these details would enhance the coherence and readability of the paper.
2.	The writing in the Experiment Results section could be further refined, especially in sections 4.2, 4.3, and 4.4. There are two issues that the reviewer may think to consider for improvement:   
   a.	When introducing specific terms like “concept” and “attributes”, it would be useful to refer back to the Preliminary section or include footnotes to avoid confusion. Readers might be unclear about which specific concepts or attributes you are referring to, given that multiple are mentioned in the preliminary section. Similarly, the distinction between “general attribute” and “predicted attribute”, which appears frequently in these sections, could be clarified to avoid ambiguity.   
        b.	Another improvement would be to more explicitly connect each section. Upon first reading, it can be difficult to understand how sections like 4.3 build on 4.2, and I had to spend some time figuring out the relationships. Explaining how each section progresses and relates to the previous one would help guide the reader through the flow of the results.

3.	More explanation could be added to the figures to assist with interpretation. While the figures look visually informative, they take some time to understand. For example, Figure 6a lacks guidance on how it should be interpreted, and adding this clarification would make it easier for readers to grasp the key points.
4.	The logic error example in Section 4.6 is a bit confusing. In the correct answer, it states that Sony sold more units than Sega but concludes that “Sony did not win the war”, which seems contradictory. Given that Sony outsold Sega, shouldn’t the conclusion be “Sony win the war”? This contradiction between the conclusion and the green-underlined statements needs further clarification to resolve the confusion.
5.	There are way too many citation format problems: e.g., line 080, “Wang et al. (2023a)” -> “(Wang et al. 2023a)”. The reviewer hopes that the authors thoroughly check all related issues and fully polish the paper presentation.
6.	In the main experimental results presented in Table 2, the authors only compare the results using the Llama2 as the backbone LLM. Yet they do not analyze the performance across other important relevant LLMs, both open-sourced and proprietary. This omission likely limits the generalizability of the conclusions drawn from the proposed method. The reviewer thus suggests adding more MLLMs for experimental comparison.
7.	More importantly, the reviewer may have identified a problem with the baselines used for comparison by the authors. For instance, they have ignored the performance comparison with all successful RAG-related methods, which should have been included in Table 2. Thus, the reviewer is a little bit skeptical about the consistency of the conclusions. Please provide further detailed explanations.

Overall, the reviewer is open-minded. If the authors can actively and effectively address these concerns, the reviewer would consider raising the rating further.

questions:
1.	Are the probing experiments conducted based on the average results across both datasets, StrategyQA and CommonsenseQA? Have you observed any differences in model behavior between these two datasets?
2.	Could you clarify the distinction between “General Attribute” and “Predicted Attribute”?
3.	What are the exact values for k and l in the SSFT experiment setup? Just to confirm, do these parameters refer to the attention heads and MLP layers you found in section 4.4 as responsible for retrieving and reranking knowledge in the LLM?
4.	Do the authors plan to release the code? the reviewer could not find any provided code or metadata that would allow for a technical review of the details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697850301

thread_type:
official_review

<thread_object>
creator: reviewer_dwrj

<content>
summary:
The paper investigates the commonsense reasoning processes within large language models (LLMs) by analyzing hidden states across layers and token positions using interpretability techniques such as Path Patching, Logit Lens, and Sparse Autoencoder. The authors propose a five-stage reasoning model within LLMs and identify specific failures in knowledge retrieval. To enhance reasoning performance, they introduce a selective supervised fine-tuning (SSFT) approach that targets specific layers of the model. Experiments demonstrate slight improvements over full-model fine-tuning, particularly in out-of-domain tasks, suggesting potential benefits of selective tuning for commonsense reasoning tasks.

strengths:
•	Application of Interpretability Techniques: The use of Path Patching, Logit Lens, and Sparse Autoencoder offers a diverse set of tools to interpret commonsense reasoning within LLMs.

•	Focused Analysis on Model Components: The identification of key components, such as attention heads and MLP layers, that influence commonsense reasoning could inform future efforts to improve model architectures.

•	Resource-Efficient Fine-Tuning: Introducing SSFT presents a potentially more efficient alternative to full-model fine-tuning, showing moderate improvements in certain scenarios.

weaknesses:
•	Limited Novelty: The proposed reasoning process and interpretability insights do not substantially differ from existing concepts, limiting the originality of the work.

•	Marginal Improvements from SSFT: The performance gains from selective fine-tuning are minimal, making it unclear whether the approach offers significant advantages over existing methods.

•	Surface-Level Interpretability Analysis: The interpretability techniques are applied without providing deep insights into the specific reasons behind the model’s failures in commonsense reasoning tasks.

•	Narrow Experimental Scope: The experiments focus on a limited set of datasets, lacking diversity that could provide a more comprehensive evaluation of the model’s reasoning abilities across different contexts

questions:
1. Have the authors tested the SSFT approach on larger models or more complex tasks to assess its broader applicability?

2. How does this work differentiate itself from prior studies on retrieval-augmented generation and knowledge editing, and what unique insights does it provide on commonsense reasoning?

3. Are there plans to extend the interpretability analysis to explore deeper relationships between reasoning components, such as hierarchical or causal connections?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703276215

thread_type:
official_review

<thread_object>
creator: reviewer_keqz

<content>
summary:
This paper investigates the commonsense reasoning capabilities of large language models (LLMs). The authors aim to make the reasoning process of LLMs more transparent and understandable by dissecting it into human-comprehensible steps. Through the analysis of hidden states in different transformer layers and token positions, the paper uncovers a mechanism by which LLMs execute reasoning. The commonsense reasoning process in LLMs is found to involve a sequence of knowledge augmentation, retrieval, and answer generation, similar to retrieval-augmented generation. The paper also identifies that LLMs often contain relevant factual knowledge but fail to retrieve it correctly. To address this, the authors selectively fine-tune key heads and MLPs, leading to improvements in reasoning performance.

strengths:
-	The proposed approach provides a understanding of the inner workings of LLMs during commonsense reasoning.
-	The paper introduces a selective fine-tuning strategy that targets less than 10% of the model's parameters, leading to notable performance enhancements. It is particularly effective for out-of-domain settings.

weaknesses:
-	The experiments are mainly conducted on specific models like Llama2-7B and Gemma2-9B. It would be better to expand the coverage of the experiments.
-	Although GPT-4 is used to assist in the analysis, there may still be some uncertainty in the interpretation of the results. The "Interpreting Module" may not be completely accurate in all cases.

questions:
1.	How do the authors ensure that the selective fine-tuning approach does not lead to overfitting, particularly when focusing on specific components of the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730739940909

thread_type:
official_review

<thread_object>
creator: reviewer_schf

<content>
summary:
This paper investigates how Large Language Models (LLMs) perform commonsense reasoning by deciphering their internal mechanisms. The researchers discovered that LLMs execute reasoning through a five-stage process similar to Retrieval-Augmented Generation (RAG): knowledge augmentation, recall, re-ranking, rationale conclusion, and answer generation. The study found that both attention heads and multi-layer perceptrons (MLPs) contribute to factual knowledge generation from different perspectives. When LLMs fail at commonsense reasoning tasks, it's often due to incorrect knowledge retrieval rather than a lack of knowledge. To address this, the researchers developed a selective fine-tuning approach targeting specific attention heads and MLPs crucial for knowledge retrieval, which improved reasoning performance while modifying less than 10% of the model's parameters. The study was conducted using Llama2 and Gemma models, focusing on a standardized template for commonsense reasoning questions to ensure controlled experimentation.

strengths:
1. This paper studies the internal mechanism for LLMs to conduct commonsense reasoning. The authors split the reasoning processes into 5 stages and find that the retrieval stage is the main source of reasoning errors. This paper provides insightful discovery of the underlying mechanism of LLMs.
2. Based on the new discovery, this paper also proposed a fine-tuning method that only focuses on top-K attention heads.
3. Extensively experimental results show the effectiveness of the methods.

weaknesses:
1. The evaluation is limited to a small scope. The authors conduct their experiments solely on yes or no questions in the commonsense domain. More datasets with diverse formats should be included, like WinoGrande and SocialIQA. This eliminated scope impairs the conclusion of this paper and raises the doubt of generalization in more formats and more domains.
2. The evaluation of the paper heavily relies on GPT-4 for both data synthesis and analysis verification. The accuracy of GPT-4 on those tasks remains unclear, and the authors need to provide more experiments to show the agreement between GPT-4 and human experts.
3. The authors just used others' methods for efficient fine-tuning without any modification. In Section 3.3, the authors only use a single paragraph to finish the description of their methods for efficient tuning. Meanwhile, this methods is just a copy of previous method from the paper "Interpreting and improving large language models in arithmetic calculation."

questions:
No
</content>
</thread_object>

</thread>
<thread>
creation date:
1732631300740

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response About Revision

comment:
First of all, we would like to express our gratitude to all the reviewers for their time and efforts in reviewing our paper. The valuable comments from the reviewers have greatly contributed to strengthening our manuscript.

We have revised our manuscript based on the reviewers' comments and have uploaded it. Here, we summarize our modifications, following the original order presented on this webpage:


1. We have added a comparison with related work on factual knowledge interpretability (Thanks to Reviewer DWRJ).

2. We have included formal definitions for the key concepts used in the paper in Section 3.1 (Preliminary) and adjusted the terminology used in the experimental sections based on these definitions (Thanks to Reviewer 4kUU).

3. We have reorganized the explanation of the interpreting module in Section 3.2, refining its framework and process (Thanks to Reviewer 4kUU).

4. We have added Section 3.3 (Instantiation of Interpreting Module) to provide specific details on how the interpreting module is applied. Additionally, we have included a diagram in the appendix to illustrate the input and output of each interpreting module (Thanks to Reviewer 4kUU).

5. We have reviewed the parts of the paper involving GPT and conducted a systematic evaluation of the performance differences between GPT and humans, providing the details of the prompts used (Thanks to Reviewers keqz, scHF).

6. We have added Section 4.1 (Experiment Overview) at the beginning of the experimental section to clarify the structure of the experiments and the relationships between the sections (Thanks to Reviewer 4kUU).

7. We have included probing experiment results for Gemma2-9B on two new datasets (Winogrande and SocialIQA) and validated the effectiveness of SSFT on these new datasets (Thanks to Reviewers scHF, DWRJ).

8. We have added probing and SSFT experimental results for Qwen2.5-72B on StrategyQA (Thanks to Reviewers keqz, DWRJ, 4kUU).

9. We have included an explanation of the interpretability analysis before and after SSFT training, demonstrating how SSFT enhances the model's capabilities (Thanks to Reviewer scHF).

10. We have reorganized the experimental tables for improved clarity and coherence (Thanks to Reviewer DWRJ).

11. We have updated the captions for all figures in the experimental sections, providing more detailed explanations to help readers understand the key ideas presented in each figure (Thanks to Reviewer 4kUU).

12. We have corrected errors in the figure from Section 4.6, issues and citation format errors (Thanks to Reviewer 4kUU).

Finally, we would like to thank all the reviewers once again for taking the time to review our paper and providing us with your invaluable feedback. We sincerely appreciate your constructive comments on our work. 

Please feel free to ask any remaining questions as we would be glad to provide further clarification. We look forward to any further discussions with all the reviewers.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the internal commonsense reasoning mechanisms in large language models (LLMs) by interpreting hidden states across transformer layers and token positions. The authors identify a five-stage reasoning process involving knowledge augmentation, retrieval, re-ranking, rationale conclusion, and answer generation, akin to retrieval-augmented generation. To address common retrieval errors, they introduce a (parameter-efficient) Selective Supervised Fine-Tuning (SSFT) approach, targeting specific attention heads and MLP layers, which improves reasoning performance.

Strength: The paper provides a detailed analysis of LLMs’ internal mechanisms during commonsense reasoning, offering insights into key components like attention heads and MLP layers. The proposed resource-efficient SSFT method demonstrates performance improvements.

Weakness: Reviewers pointed out that the evaluation scope is narrow, with experiments focusing on limited datasets and specific models. This has been partially addressed during the rebuttal. Also the novelty of the work is limited to applying existing techniques to analyzing a new problem.