Title: Who Reasons in the Large Language Models?
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

reasoning

interpretability

Transformer
Abstract: Despite the impressive performance of large language models (LLMs), the process of endowing them with new capabilities---such as mathematical reasoning---remains largely empirical and opaque. A critical open question is whether reasoning abilities stem from the entire model, specific modules, or are merely artifacts of overfitting. In this work, we hypothesize that the reasoning capabilities in well-trained LLMs are primarily attributed to the output projection module (o_proj) in the Transformer’s multi-head self-attention (MHSA) module. To support this hypothesis, we introduce Stethoscope for Networks (SfN), a suite of diagnostic tools designed to probe and analyze the internal behaviors of LLMs. Using SfN, we provide both circumstantial and empirical evidence suggesting that o_proj plays a central role in enabling reasoning, whereas other modules contribute more to fluent dialogue. These findings offer a new perspective on LLM interpretability and open avenues for more targeted training strategies, potentially enabling more efficient and specialized LLMs.


Peer reviews:
comments: SUMMARY:
This paper investigates the reasoning of LLMs, namely, which part of the LLM is responsible for the reasoning abilities.
They do so by using 4 investigation tools given a base model A and a reasoning model B: comparing the weights of A and B, merge the weights of A and B, finetuning A with frozen components, destroying components in A.

They pinpoint the reasoning module to the output_projection of multi-head self attention.

STRENGTHS AND WEAKNESSES:
* Strength *

1. This is an interesting research question
2. It has large potential of influence on training of LLMs
3. Nice methodology

* Weaknesses *

My main remark is that the one of the roles of the output_projection in the MHSA layer is to map from the attention outputs to the mlp inputs. I'm not convinced that the lack of effect of the "q, k, v" in the merge setup is not due to their lack of importance but a mismatch between their outputs and mlp inputs.

QUESTIONS:
Did you try measuring the impact of the "output_proj" layer on tasks that do not involve reasoning?

LIMITATIONS:
The only benchmarks used are reasoning ones, curious what is the effect on non-reasoning benchmarks
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper empirically studies the hypothesis that the model reasoning capabilities attribute to output projection’s parameters in the model multi-head self-attention. The proposed method, Stethoscope for Networks (SfN), compare the weight differences between fine-tuned and base models under controlled modules of merging, tuning, or destruction, and provides interpretable insights into the origin of capabilities like reasoning.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. Interesting idea under \"Stethoscope for Networks (SfN) metaphor. Delta/Merge/Freeze/Destruction works well with the ideation with parameter isolation.\n2. Instead of existing work with factual recall, syntax, token, prediction, or attention layers, the work tries to narrow down their scope of exploration even further down into components of attention modules (o_proj, q/k/v_proj).\n3. The question is valuable to ask where the reasoning is localized/attributed primarily.\n\n### **Weaknesses**\n1. Writing needs more work/ esp. introduction (motivation) is a bit chaotic and not compelling.\n- LLMs still struggle with trial-and-error experiments for developing new capabilities -> can you provide more examples? what challenges are we aiming to tackle? \n- Black-box style success -> can we be explicit how it is problematic to recent successes in reasoning capabilities of PPO, DPO, SFT, etc?\n2. The experiments are not riguorous; not compelling if the research question is addressed enough. Table 2: F2 is based on attention layers updated on pre-trained model activations (by default) but without being updated with the final layer. This is naturally expected that F2 and F4 shows small changes.  \n3. The originality of the work is not quite strong. Recent works not limited to classic works [a, b] show that MLP often act like \"fact retriever\" or local updaters while attention moves information around, enabling reasoning chains and variable binding. But that doesn't mean the reasoning only happens in attention outputs as the attention layer updates with model activations that are frozen yet pre-trained knowledge. Not sure how this contribute novel findings beyond already existing works.\n\n\nref\n[a] Transformer Feed-Forward Layers Are Key-Value Memories, EMNLP 2021\n[b] Locating and Editing Factual Associations in GPT, NeurIPS 2022\n\nQUESTIONS:\n1.  How did you unfreeze o_proj, without unfreezing q/k/v_proj? Did you just not update the gradients? If you are not changing what the model pays attention to but how the attention module can map the attended results as o_proj? \n2. Table 2: F2 is based on attention layers updated on pre-trained model activations (by default), no? Did you mean you didn't update the proj_o with the final layer after updating with activations with premature layers? If so, isn't it naturally expected that F2 and F4 shows small changes; F4 achieves a bit of performance gains as it is shown?\n3. What do authors think their novel contributions are from this work, that are beyond the previous observations (MLP=factual storage, attention=reasoning logic, layer updates=all parts contribute, even if frozen at inference)?\n\nLIMITATIONS:\nLayer updates mean \"reasoning\" isn't isolated. Their conclusion risks a localization fallacy that tuning o_proj improves reasoning. But the experimental setups/results are a bit unclear/weak to conclude that o_proj is the place the reasoning happens. It still seems that they use frozen model activations (which model stores their factual associations) and observing weight changes or successful performance after tuning o_proj with those factual associations do not seem sufficient evidence that reasoning is localized here but rather convincing to say it is a whole computation of model circuits.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the underlying mechanisms of reasoning abilities in LLMs. The paper hypothesizes that the output projection module within the transformer multi-head self-attention is primarily responsible for reasoning capabilities. To support this hypothesis, authors introduce Stethoscope for Networks (SfN), a suite of diagnostic tools, including the Delta, Merge, Freeze, and Destruction Stethoscopes. Through empirical evidence from these tools, they suggest that the output projection module plays a central role in reasoning, while other modules contribute more to fluent dialogue. The findings offer insights into LLM interpretability and propose avenues for more targeted and efficient reasoning training strategies.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a novel hypothesis regarding the localization of reasoning ability in LLMs, specifically pointing to the o_proj module. The introduction of the Stethoscope for Networks (SfN) framework with its various diagnostic tools provides a systematic and empirical approach to probing LLM internals. They provided extensive experiments, and the consistency of observations across different model sizes adds to their findings. The paper is well-structured and generally easy to follow. The introduction clearly lays out the problem and the paper's hypothesis. Each Stethoscope method is explained with its underlying assumption and how it contributes to the overall hypothesis. The core hypothesis, attributing reasoning capabilities primarily to the o_proj module, is novel and deviates from common assumptions that reasoning stems from the entire model or more complex interactions. However, some recent and relevant studies seem to be missed in related work and discussion, for example:\n\n```\nRuis, Laura, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. \"Procedural knowledge in pretraining drives reasoning in large language models.\" arXiv preprint arXiv:2411.12580 (2024).\n```\n\nQUESTIONS:\n- While the paper empirically demonstrates the importance of o_proj in reasoning, a deeper theoretical understanding of why o_proj plays this role is missing. Could the authors elaborate on potential mechanisms or computations within o_proj that might facilitate reasoning? For instance, does it act as a bottleneck for abstract representation, or does it aggregate information in a specific way that enables logical steps?\n- The experiments are primarily conducted on Qwen2.5 models. While the results are consistent within this family, it would be beneficial to discuss the potential generalizability of the findings to other LLM architectures. Are there architectural differences that might influence the role of o_proj in reasoning, or do the authors believe this finding is broadly applicable across modern LLMs?\n- The Freeze Stethoscope section notes that for larger models (e.g., 7B), simply merging o_proj is insufficient, and an o_proj + layernorm merge is considered. Could the authors expand on the specific interplay between o_proj and layernorm in enabling reasoning, especially for larger models? Understanding this interaction might reveal a more nuanced picture of reasoning localization.\n- The paper mentions that fine-tuning with reasoning examples leads to strong reasoning capabilities in model B. Is there any hypothesis or preliminary observation on how the type of training data might influence the specialization of modules, particularly o_proj?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: This paper investigates which components within large language models (LLMs) are primarily responsible for the emergence of reasoning abilities, particularly in mathematical and logical tasks. The authors propose a systematic diagnostic toolkit, \"Stethoscope for Networks (SfN),\" which includes several probing techniques to analyze the internal behavior of LLMs. Through comprehensive empirical studies, the paper shows that the output projection module (o_proj) in the Transformer’s multi-head self-attention mechanism plays a central role in reasoning, while other modules contribute mainly to conversational fluency. The findings present new insights into modularity and targeted training in modern LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1、The study introduces a simple yet effective methodology for attributing functional capabilities to specific modules inside LLMs.\n2、The empirical analysis is thorough and well-supported by clear ablation and merge experiments.\n3、Identifying o_proj as critical for reasoning could lead to more efficient and targeted fine-tuning approaches.\n4、The proposed SfN toolkit may be broadly useful for interpretability research beyond reasoning tasks.\nWeakness\n1、Lack of theoretical explanation: The paper mainly relies on empirical analysis and does not provide a deeper theoretical understanding of why the o_proj module dominates reasoning in Transformers.\n2、Limited generalization: Experiments are only conducted on a few open-source models such as Qwen and DeepSeek, and mainly on specific reasoning tasks (e.g., math problems), so it is unclear whether the findings generalize to other architectures and tasks.\n3、Some conclusions rely on qualitative judgments: The division of modules for \"conversational ability\" is mainly based on output examples and observation, lacking large-scale statistical validation and being somewhat subjective.\n4、Generality of the SfN tool is not fully validated: Although the authors propose SfN as a universal diagnostic toolkit, experiments are focused on reasoning modules and tasks, lacking evidence of its effectiveness on other types of models and tasks.\n\nQUESTIONS:\n1、The analysis shows o_proj is key for reasoning in the tested models, but have you tried the same methods on models with different architectures (e.g., LLaMA, GPT-style, or encoder-decoder models) to see if this conclusion still holds?\n2、In Section 3.2 (Destruction Stethoscope), the conversational ability analysis is mostly based on selected examples. Could you provide more systematic statistics or human evaluations to support the proposed \"division of labor\" between modules?\n3、The paper mentions potential applicability of SfN to other networks (Section 4), but there are no experiments beyond reasoning tasks. Do you have preliminary results or plans to test SfN on abilities such as knowledge retention or code generation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-07"}