Who Reasons in the Large Language Models?
Authors: Jie Shao
Jianxin Wu
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
reasoning
interpretability
Transformer
Primary area: deep_learning
TL;DR: We identify the output projection module as the key component enabling reasoning in LLMs, and propose Stethoscope for Networks (SfN) to diagnose and support this claim.

Abstract: Despite the impressive performance of large language models (LLMs), the process of endowing them with new capabilities---such as mathematical reasoning---remains largely empirical and opaque. A critical open question is whether reasoning abilities stem from the entire model, specific modules, or are merely artifacts of overfitting. In this work, we hypothesize that the reasoning capabilities in well-trained LLMs are primarily attributed to the output projection module (o_proj) in the Transformer’s multi-head self-attention (MHSA) module. To support this hypothesis, we introduce Stethoscope for Networks (SfN), a suite of diagnostic tools designed to probe and analyze the internal behaviors of LLMs. Using SfN, we provide both circumstantial and empirical evidence suggesting that o_proj plays a central role in enabling reasoning, whereas other modules contribute more to fluent dialogue. These findings offer a new perspective on LLM interpretability and open avenues for more targeted training strategies, potentially enabling more efficient and specialized LLMs.

Who Reasons in the Large Language Models?
Jie Shao Jianxin Wu'
National Key Laboratory for Novel Software Technology, Nanjing University, China School of Artificial Intelligence, Nanjing University, China shaoj ammmma.....mmmmm jju.edu..
edu. en, wujx2001@njj e.r.0001nnuue.........
Abstract
Despite the impressive performance of large language models (LLMs), the pro- cess of endowing them with new capabilities- such as mathematical reasoning- remains largely empirical and opaque. A critical open question is whether reasoning abilities stem from the entire model, specific modules, or are merely artifacts of overfitting. In this work, we hypothesize that the reasoning capabilities in well- trained L.LMs are primarily attributed to the output projection module (o_proj) in the Transformer's s multi-head self-attention (MHSA) module. To support this hypothesis, we introduce Stethoscope for Networks (SIN), a suite of diagnostic tools designed to probe and analyze the internal behaviors of LLMs. Using SIN, we provide both circumstantial and empirical evidence suggesting that o_proj plays central role in enabling reasoning, whereas other modules contribute more to fluent dialogue. These findings offer new perspective on LLM interpretability and open avenues for more targeted training strategies, potentially enabling more efficient and specialized L.LMs.
Introduction
Although large language models (LLMs) 29 6 42
have exhibited great success and potential in various aspects. developing new capabilities for LLMs [54] [7 38 [4] is still trial and error experimentation process most cases. For example, one of the most exciting milestones is LLMs
that can reason 18 13 40 e.g., solving complicated mathematical problems using a reasoning
sequence that is agreeable by human experts.
This success, however, is still in the black-box style. Currently, there are two primary approaches inspiring reasoning capabilities in LLMs. For the most advanced models [3] 52 reinforcement learning method (for example, PPO (37. DPO (30], or GRPO (88) is commonly adopted to enhance the model's ability to solve complex mathematical or programming problems in step-by-step manner 44 A more efficient alternative involves supervised fine-tuning (SFT): by providing the backbone LLM with well-prepared, diverse, and step-by-step reasoning traces-ofter generated through handcrafted examples or existing reasoning models (55) 25) 52 tte model surprisingly acquires reasoning abilities after training. However, despite the practical success of this method, the underlying mechanism remains largely unexplained. It is still unclear why or how this ability emerges. Several potential explanations may account for this phenomenon:
Case Is it the LLM in its entirety (i.e., the union of all its weights) that leads this capability, such that this miracle is not explainable?
Case 2 Or, is there certain module(s) in it that should be praised for this success, such that advance our understanding of L.LMs??
*Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Stethoscope for Networks W
1. The Delta Stethoseope
The Merge Stethoseope
Conjecture:
Reasnsing
Model
2
radeus
Sone Madd
60
Mak
enrg:
Bave Noald
60
Marped Mode
Chat
LLM
3. The Freeze Stethoscupe
4. The Destruction Stethoscope
Box
Medd
chat
k, proj MLP Renson: o_proj
Figure I: Stethoscope for Networks. SfN is a framework designed to identify which components of an LLM give rise specific abilities. By comparing weight changes and observing behaviors under controlled module merging, tuning, or destruction, SfN provides interpretable insights into the origin of capabilities like reasoning.
Case 3 Or in the worst scenario, is reasoning an illusion (e.g., by overfitting to certain types of data), such that we have overestimated the potentials of LLMs?
definitive answer to any of the above questions will be extremely valuable to guiding the future direction of L.M research. Even hypothesis or conjecture supported by circumstantial evidences will be highly enlightening, too, let alone when convincing empirical evidences are available.
To this end, our hypothesis is that Case 2 holds in LLMs that reason well. To be more precise, we hypothesize that it is the output projection's parameters (o_proj) in the Transformer [44 multi-head self-attention (MHSA) module that is in charge of reasoning in an LLM.
To support hypothesis, propose a few techniques for diagnosing LLM's behaviors, in particular, the potential functionalities and impacts of various modules it. We call these techniques Stethoscope for Networks, SfN (summarized and illustrated in Figure [T. Starting from reasoning-enhancedd models, we argue that the weight differences hetween a base I.I.M and its fine-tuned counterpart (e.g., for reasoning tasks) provide firsthand and crucial evidence for understanding internal changes. We refer to this approach as the Delta Stethoscope.
In addition, we introduce two novel and previously unexplored methods within the SIN framework: the Merge Stethoscope and the Destruction Stethoscope. The Merge Stethoscope replaces specific modules in base model with those from reasoning model. Surprisingly, the resulting variant can maintain fluent dialogue and demonstrate improved reasoning ability in some cases. This phenomenon offers strong clues about the origin and localization of reasoning capability in LLMs. The Destruction Stethoscope, in contrast, systematically disables individual modules and observes the resulting behavior to infer the functional roles of each component. We also propose the Freeze Stethoscope, which selectively freezes parts the model during fine-tuning. By controlling which modules are updated, provide convincing empirical support for earlier insights and clues into the localization of reasoning within LLMs. With different gadgets we propose in SIN, we provide not only sanity check level tests for our hypothesis, but also more convincing circumstantial supports and even direct empirical evidences. In short, the contributions in this paper two-fold:
With various diagnosis evidence (SfN), projection o_proj mainly responsible for the reasoning in LLMs. The impact of this finding include not only potential ways to improve LLM that reasons (e.g., training much faster), but may generalize to produce better L.MMs for other tasks (e.g., for a vertical LLM designed specifically for domain). Our further conjecture is that other modules combined together lead to lucid conversations, but o j_proj is less important in conversational ability.
confident in hypothesizing that the output
The proposed Stethoscope for Networks (SfN) gadgets are a set of tools that are useful in understanding modern L.MMs and even other networks, which have the potential to enhance our understanding of L.MM or deep neural network and may lead to alternative routes for further deep learning research.

---

2
Key Hypothesis: Output Projection is the Key for Reasoning
To present our findings, we start by introducing necessary background information and notations, while discussions on related work are deferred to Section 5
Modern LLMs 42 29 mostly consist of many Transformer blocks. A Transformer [44 block
is composed of a multi-head self-altention (MHSA) module and a multi-layer perceptron (MLP) module. Components in MHSA include various projections, such those for computing Q K and V. denoted as q-proj. k_proj, and v_proj, respectively. The output projection (o_proj) produces MHSA's output. Components in the MLP are mainly linear projections: up, down, and gate [16] 42 S projections, denoted as up-proj, down_proj, and gate_proj. respectively. The computation process is defined as:
(wqz) (wqx))()x)) (m,,))
Softmax
T Fatta = Wo
Vd
Fmlp
Waos
(Mggatt
O (Wapz)
For simplicity, we omit residual connections and present the computation at the token level, without using matrix vectorized notation. Other essential components not explicitly included in equa- tion include rotary positional embeddings (RoPE) [3]] input embeddings (embed_tokens) layer normalization (layernorm). and the language modeling head (Im_head),
Let A be an LLM with weak or no reasoning ability. By carefully procuring a dataset of reasoning examples
B 23 [2], one can cleanse and improve the quality of the dataset into the training data D, and then finetune the existing model A by using techniques such as SFT. The resulting LLM, model B, exhibits strong reasoning capabilities. For example, in commonly adopted practices, the base LLM A is typically a widely used open-source model such as Qwen2.5-Mann 1,55, 7B or Qwen2.5-14B, variant, such
32B (53). The reasoning model B denotes publicly available massoning-enhancee as DeepSeek-Re R----iiiilll--eeeeeeee 1.5B, 7B. 14B, 32B (13), which comes with a clearly specified base model and well-documented training procedure. Models that are either not open-sourced 13 40)
or neen-sourced without sufficient training details 41 or access to the base model [2] are not discussed in this paper.
2.1 The Delta Stethoscope
In the above scenario, it is obvious that A and B share exactly the same network architecture and structure, with their sole difference being the weights (parameters) inside various components. Suppose w((A) (B)) denotes the set of weights for all modules in A (B). Then, it is natural to conclude that to understand the difference between A and B (i.e., reasoning or not), we should focus on the difference between w(A w(A) and w B). Hence, we propose our first Stethoscope for Network.
Assumption (The Delta Stethoscope) Suppose A and B are two LLMS with weak and strong reasoning ability, respectively, and obtained by finetuning from A. Then w w(B) - w(A) contains essential information if we want to pinpoint the source of the reasoning ability in B.
For each component X (e.g. X= q-proj), compute the l norm of the weight difference,, [| - w (A)lle, and visualize the results across all the blocks in Figure 2 For simplicity and due to space constraints, we present three representative comparisons: A is Owen2.5-Math- or Qwen2.5-14B. 32B (53) and is DeepSeek- RR RiI-Dittill-Qwen--.55 14B, 32B [33. Additional results for other model sizes (7B and 8B) are provided in the appendix and exhibit similar patterns.
(54)
For the 1.5B models, the signal less clear, but o o_proj still exhibits a distinct pattern compared to q, Sts,v_proj-shvve
the largest change within the attention module and the second-largest across the entire model. As model size increases to 14B and 32B, this trend becomes more pronounced. In both cases, the most notable observation is that when X = o_proj, the l norm is at least two times larger than any other component, indicating the substantial changes in this module during reasoning enhancement.
In Figure we further analyze the distribution of relative weight changes wx (B) for each aix (A)
linear module. To improve clarity and visual appeal, we plot the distribution every 5 layers and clip values in the range [-1.0, 1.0] to mitigate the influence of outliers. The vertical axis represents the
3

---

Qwen2.5-Math-1.55 DeepScee-R1 -ereaa--RR--iiill--e-----..
3 p00-
-10]
0_3m]
14:21
cown gen]
0.00
Qwen2.5-14B vs, DoepSeek-R PrrSscck-I--Diil---- Rt--Deill-Qwan-145
4_2e0--
x_500--
0.pro
pro]
prq-
Qwen2.5-32B DeepSock-R Distill-Qwee-222
S_peij-
x_peq-
glt pmq
[20]
32
(layer)
Figure 2: Per-module L2 distance of linear weights between models A and B. Notably, the o_proj module shows the second-late largest change in 1.5B models, and the largest in 14B and 32B models, highlighting its potential importance for reasoning. Similar trends are observed in 7B and 8B models (see appendix).
Qwea-255Matb-11 1.5 DeepSoek-RI Ri-.ii---l---eeee-el-ee
DeqpSeekRR Ditillll---oottaaal
DeepSeckRR Di
self attn--_pooj
solf atto proj
self www.tttnnnnno proj
self, AIEOLO pro)
MLPup_proj
MLP MLP.gate proj
MLP. ML..dww, proj
Figure 3: Layer-wise distribution of relative weight changes between models A and B. While most modules display a unimodal distribution, the o_proj module uniquely exhibits a bimodal distribution, highlighting its distinctive behavior. Consistent patterns observed across models of other sizes, with detailed results provided in the appendix.
frequency. A striking and consistent finding is that all linear modules- except e_proj-exhibe a unimodal distribution centered around zero, whereas o_ o-proj uniquely displays clear bimodal pattern, highlighting its distinct role.
Both observations hold consistently across model sizes and base models: o_proj exhibits the largest second-largen weight shift, and the overall weight difference patterns remain strikingly similar. Therefore, is reasonable to guess that the output projection o_proj plays a pivotal role in curating B's reasoning ability. We however, not aware of o o_proj's specific role: is solely responsible for reasoning? Or, it collaborating with another module(s)? Or, in the worst scenario, this difference in | (B) wx (A)]le, and (x(B) (A) coincidental?
(x)

---

Level I: Nonsense
Level II: Irrelevant
Output facks coherence or meaning.
Grammaticaly correct, but contextually off aff-topic.
Can you short paragraph about importance reading books?
Q: Can write short peragraph about the importance of reading books?
A1:
A2: the importance of the importance of course...
A: Can you provide me with some data?
course, course, the importance
Level Context-Aware
Level IV: Correct and Logical Reasoning
Understandss the question, but may apply flawed reasoning.
Fully understands the context and provides valid solution.
Q: Can you write a short paragraph about the importance of reading books?
Q: Every morning, Aya This morning, If she walks s+1/2 kilometers par hour, how many minutes will the walk take?
Az Absolutelyl Reading a crucial skill that can help students improve... and understand the implications of these akills.
A: First, the problem says that Subtract:
Every morning, Aya ... morning. she walks at s+1/2 kilometers per hour, how many minutes First, is set up the equations... ... Simplify and
9(
= 1.6 5(s 2) 11.25 -==
5
take?
Then: 3.6
-4-12 24
60
37
Today; speed kmrh, walk = 180 min, total 180 minutes.
144y af = 252;
60
Final snswer: 12
Figure 4: Four levels of responses generated by the LLM. From level to level IV, the model exhibits stronger language organization and logical reasoning skills. Each example includes question (e.g., math problem from AIME or typical user-issued request) and the corresponding response generated by the LLM.
2.2 The Merge Stethoscope
We design another gadget, the Merge Stethoscope, to answer this question. Suppose an LLM M is formed by merging models A and B, that is, M has the same structure A and B, while subset of its modules' parameters come from A and the rest from B. In a conversational or reasoning task, what will the output of M look like? We can imagine 4 levels of different output, as
Level I A sequence of random or nonsense tokens.
Level II A sequence that looks like normal sentences, but does not fit into the context of the task.
Level III A sequence that is meaningful sentences that match the task's context well but will fail to reason in difficult problems.
Level IV A sequence that reasons and reasons correctly in most cases.
Figure 4 shows examples of level to IV outputs. It is worth highlighting that M is rudely merged from A and B without any further tuming. Hence, the intuitive conjecture will be that M will produce level I output (i.e., ushering meaningless tokens). However, if model M, when merged in a specific configuration, is capable of producing level IV outputs for questions that model A fails to solve, then the specially merged components are likely critical for reasoning.
Assumption (The Merge Stethoscope) Suppose M is created by merging the output projection (o_proj) weights of B (which has strong reasoning ability) and all other components of A (which is weak in reasoning), and further suppose that M has stronger reasoning ability compared to A. Then, we assume o_proj is crucial in achieving reasoning in LLMs.
We attempt a minimal or atomic merge by replacing only the o_proj modules in model A = Qwen2.5-Math-L13. [54] with that of model B DeepSSet-RII-Distil--ooeee
[3] keeping
all other components unchanged. Although we initially expected the resulting model to produce level I or level II outputs, the results turn out to be surprising. On the AIME 2024 benchmark [19] the merged model M2 achieves level IV performance on several questions that model A cannot solve. As shown in Table [ the merged model not only yields correct reasoning and answers, but also tends to generate longer and more detailed responses compared to A. In contrast, replacing other modules such as vg,kvor_rpr and mlp leads to performance degradation. For example, model M2, which replaces vq vo.k,,,,projj produces level III outputs, while M3. which replaces mlp, deteriorates to level I. Only replacing o_proj results in a correct reasoning process and a correct answer, as illustrated in Figure 5 This striking difference motivates further investigation in Section
5

---

Replaced Module
AIME 2024
Average Tokens
Q: Every moming, Ays does a kloometer walk ... the walks at
Model
S4 1/2 klometees per hour, how many minutes will take?
A (Q-1.5B)
0.067
2421
Mj To solve this problem, we need to determine... So, the walk
will take 204 minutes, including the 24 minutes the coffee shop. The final answer 204.
Mi
o_proj
0.200
5418
M2
{q j
0.000
2058
My: To solve this problem ... culput 12. 12.0000000000000 The
Ms
mlp
0.000
15532
output indicates that the time taken for the walk is 12 minutes. So, the final answer is 12.
B (D-1.BB)
0.233
11892
Mj walking speeds increase speeds faster walking speeds
Table 1: AIME 2024 accuracy of the base model, the reasoning model, and their merged variants. Each merged model is constructed by replacing specific modules in model A with the corresponding module from model B.
Increase walking speeds faster walking speeds faster walking
Figure 5: Examples of outputs generated by merged models. Only Mi produces both valid reasoning process and the correct answer.
These results clearly show that the merged model M has a stronger reasoning capacity than A, despite that M is sutured from two completely different models and has never being finetuned. Now we feel confident in our assumption that o o_proj is the key component responsible for reasoning in LLMss.
2.3 The Freeze Stethoscope
As models A and B scale up (e-g., to 7B parameters), merging components such as q,k, v_proj or mlp still results in significant performance degradation. However, unfortunately, merging o_proj no longer brings notable improvements solving complex mathematical problems-althoughy it does not harm accuracy, and still increases the generated output length.
Our analysis of ||x (B) wx (A)lle, suggests that this is due substantial mismatch in normal- ization parameters (that is, layernorm modules) between A and B at larger scales, compared to smaller models (e.g. 1.5B). Even when we merge both o o_proj and layernorm parameters from B, the resulting model M still fails to reason effectively, probably because the remaining parameters of A are incompatible with the normalization parameters of B. To investigate this hypothesis in larger LLMs, we introduce the Freeze Stethoscope.
Assumption (The Freeze Stethoscope) Suppose that an LLM F is obtained by supervised fine- tuming using the datasel 'D. F is initialized from A, and both o o_proj and normalization components are tuned while other components are frozen. If exhibits strong reasoning ability, then we assume that o_ o_proj is crucial in achieving reasoning in LLMS even in large-scale models.
worth noting that embed_ tokens and 1m_ head are also tuned Normalization module pa- rameters are unfrozen by default. We adopt the pipeline of sl (25 as our baseline, which uses the base model A Qwen2.5-32B-IIns
s1K containing 1,000 high-quality reasoning traces. The results are shown in Table 2 where our model F, corresponds to model B in Assumption]
and the dataset
3 We do not strictly follow the training or testing setup of sl, primarily due to limited computational resources and the lack of an exact testing recipe to reproduce the reported results. However, our objective is not to optimize accuracy via testing tricks or prompt tuning, but to highlight the effectiveness of proj tuning compared to full- parameter tuning. For fair comparison, we adopt the "Budget Forcing Wait 2x" setting from sl and retain all configurations without hyperparameter tuning.
Using this simplest possible experimental setup. Table clearly shows that simply tuning o o_proj and layernorm (model F2)) leads to strong reasoning ability, while at the same time only tuning layernorm (model Fi) harms the reasoning of the L..M. Further unfreezing the parameters of k, kq,k,v>_pree v}_proj (model F3)) yields little additional gain even negative impact.
The training loss curves are shown in Figure O When all parameters including MLP are unfrozen, the model exhibits clear signs of overfitting, likely using the large MLP capacity to memorize the training set. contrast, tuning only o_proj yields a smoother and more stable curve. Combined
Without tuning these components, finetuning failed to converge.

---

Loss
3213
LLUSS
141
0.4
0.4
0.2
0.2
Steps
Wm
Steps
0
50
100
150
200
250
300
0
50
100
150
200
250
300
Figure 6: Training loss curves for fine-tuning O
on reasoning tasks.
Different models unfreeze different sets of parameters, as detailed in Table 2
Model
Fintuned Modules
#Param (B)
Steps/s
AIME 2024
Math 500
GPQA Diamond
A (Q-32B)
0.167
0.836
0.485
F
Emb Head
1.5
0.055
0.200
0.756
0,444
P2
Emb Head o_proo
3.2
0,052
0.367
0.890
0.520
F4
Emb Head iq..kvv.)
5,6
0.044
0,300
0.886
0.525
P (B)
32.8
0.015
0.367
0.906
0.591
(Q-14B)
0.133
0.810
0.449
Fi
Emb Head
1.5
0.106
0.133
0.722
0.414
F
Fmhh Head o.proj
2.8
0.099
0.266
0.848
0.485
Pa
Emb Head (q. All
3.7
0.081
0.233
0.854
0.490
(L)
14.7
0.053
0.266
0.872
0.530
Table 2: Reasoning performance of different fine-tuning strategies on Qwen2.5-(14B, 32B)-
Instruct. Emb denotes embed_ #Param refers to the number of trainable parameters, Steps/s indicates training speed, and the last three columns report commonly used metrics for evaluating reasoning models.
embed_tokens, tokens, Head denotes lm_head and Attn denotes the entire MHSA.
with its competitive performance, this suggests that the model learns to reason rather than simply memorize. Hence, we are now prepared and feel supported to propose our key hypothesis:
Hypothesis (Outstanding Output Projection) In an LLM that reasons well, we hypothesize that the output projection (o_ (o_proj) component is the single at least the most important module that dominates its reasoning ability.
With carefully chosen tuning strategy and hyperparameters, there is reason believe that tuning only o_proj (+LN) can reach the level of model B in terms of reasoning performance. And, beyond exhibiting reasoning abilities, Table 2 also shows that tuning only o o_proj (+LN) has other significant advantages; e.g., significantly faster finetuning (3 times faster) and smaller GPU memory consumption. These advantages will become more established when larger LLMs are tuned.
Conjecture: Conversation Hinges on Other Modules but Not Output
We are mainly concerned with two abilities of LLMs: conversation and reasoning, which map level III and IV our categorization LLM's outputs, respectively. Our Hypothesis/ is on reasoning, but there module several modules accounting for lucid conversations? In this section, further propose a new stethoscope to diagnose this question and raise our conjectures accordingly.
3.1 The Destruction Stethoscope
Our previous stethoscopes follow "constructive proof" style, while now resort to the "proof by contradiction" style. If one module in an LLM "destructed", and the LLM can still produce level III conversation outputs, then we have good reasons guess that this module is not important in conversational ability; while is important the LLM ceases to dialogue regularly.
Assumption (The Destruction Stethoscope) Suppose module X destructed (i.e., its normal functionality is disabled by some destruction method) in LLM A. We denote the resulting LLM as
7

---

Destruction Method
Output Level
Destruction Method
Output Level
Module
Module
q-proj k_proj
q-proj k_proj v _rroj o-proj
v. v-proj
III
II III
Zero
o-proj
III
ReInit
up-proj
up up-proj gate_proj down_proj
gate_pro down_proj I
Remove
Table 3: Output levels of different modules under the three destruction methods: Zero, ReInit, t, and Remove. All experiments are based on Qwen2.5-32B with destruction applied to specific layers.
D. Then, the fact that D continues (or ceases to) produce level III output (meaningful sentences in the conversation's 's context) indicates whether X is important for conversational abilities or not.
We propose 3 destructors to destroy module:
Zero Set all parameters within X to O.
ReInit Re-initialize all parameters inside using Gaussian random numbers (mean=0, std=0.02).
Remove Remove the entire layer.
The Zero destructor is often equivalent setting the output activation of X to (e.g.. in linear module like o_proj). We want to emphasize that ReInit incurs more serious damages to an LLM than Zero does. Zero may change activations zero, but ReInit exerts random effects (i.e., noise) to LLM activations. What is more important, these random effects will act as input to the next Transformer block and the noise is quickly amplified. Hence, level or output is expected when X is destroyed (especially when reinitialized) in large number of Transformer blocks.
3.2 Conjectures Concerning the Conversation Capability
For model Qwen2.5-32B with 64 layers, we observe that destroying modules in early or late layers- where input and output representations more sensitive consistently yields level I outputs. To avoid this, we restrict destruction blocks 5-30. This range is empirically chosen, as affecting more layers often causes all outputs degrade to level I, making distinctions between modules impossible.
The experimental results are presented Table Specifically, we destroy selected modules and analyze the corresponding output. The Remove destructor removes the transformer layers as a whole. Note that the results are not statistics computed in many different experiments it only reflects the conversation illustrated in Figure 4. but we observed similar patterns for other conversations.
Table reveals reveals distinct roles of modules in conversation. Notably, h.proj--crucal for reasoning- appears unimportant for conversation. In contrast, all MLP components (up_proj, down_pro) wwn_prjj. gate_proj) are essential. Within MHSA, q-proj and k_proj are important, while v-proj plays minor role. Based on these (admittedly weaker) observations, we propose the following conjecture.
Conjecture (Division Labor) Based on current observations, an LLM can be roughly divided as two sets of modules: output projection o_proj) and all others, where o_proj is mainly responsi- ble for reasoning and other modules for conversation.
Then, output projection plays unique role if this conjecture holds. Hence, we further propose another conjecture for it.
Conjecture 2 (Output Projection Plugin) With conversational capabilities provided by other (frozen) modules, output projections may act as a plugin. For example, one set of o_proj for reasoning, and another set of o_pro prrjjfa for migrating LLM to a vertical domain.

---

4 Potential Implications and Applications
This paper mainly diagnoses LLMs from a theoretical, highly abstract perspective. However, our hypothesis and conjectures can also have highly practical implications and applications as long as they are correct at least partially hold.
Fast and better reasoning LLMs. By finetuning only o_proj, we can potentially find better reasoning LLM with much faster training and much smaller GPU memory footprint.
Integrating non-reasoning and reasoning LLMs. There is a recent trend to integrate chatting and reasoning LLMs into one model [52]. When we finetune a base LLM into a reasoning one using the previous procedure, they only differ in o_proj. layernorm. uubed_tokent and Imhead, which occupy only 10% of model size. Hence, the two LLMs are easily loaded as one LLM with two sets of these module for different purposes.
Vertical LLMs. Similarly, when equipped with different output projection plugins, one may adeptly obtain vertical LLMSs for different domains.
Understanding deep neural networks. The proposed Stethoscopes for Networks might useful gadgets to understand other deep models, and new stethoscopes can be further developed. They will be potentially useful in diagnosing existing networks and even in providing alternative directions future deep learning research.
Related Work
Large Language Models. Modern LLMs such as GPT 29 6). LLaMA
42 43 Qwen [5] [3
and other representative models 7 20 adopt an auto-regressive architecture and have demonstrated impressive capabilities across a wide range of natural language processing tasks, including question answering [32] (22,, summarization 226 227. and translation [5I]. These models are typically trained on large-scale corpora using next-token prediction objectives, and their performance has been shown to scale with model size [21]. Further improvements in alignment and usability have been achieved through instruction tuning (28] g 47 and reinforcement learning from human feedback (RLHF) g 30), enabling more controllable and helpful dialogue generation.
Reasoning Models. While LLMs exhibit emergent reasoning abilities 48 35 recent efforts have
further enhanced these capabilities through fine-tuning and architectural modifications [36 [6].
Chain-of-thought prompting [49] encourages intermediate reasoning steps, improving performance in arithmetic tasks, while self-consistency decoding [46] improves robustness by sampling multiple reasoning paths. Inspired by OpenAl's o1 [18.. most advanced models now employ reinforcement lcarning (37 [30] to generate long reasoning traces with sparse rewards. This leads significant
improvements, particularly in complex math, code, and other professional domains [33 [52). Despite
these advances, the origin and location of reasoning ability in LLMs remain underexplored.
Interpretability of LLMs. Understanding the inner workings of LLMs has attracted growing interest. Prior efforts include attention visualization (45), probing [15], and model editing 22 34]
with the aim of interpreting internal representations. Other studies decompose the behavior of the model into attribute functions specific modules (T]] The "Physics of Language Models" series D 2 3 investigates LLMs through controlled setups to reveal empirical and universal laws that dictate LLM behavior. However, these studies often exclude the most advanced models focus on narrow, synthetic settings, offering limited insight into real-world models. Their findings provide little practical guidance for understanding reasoning state-of- of-the-art models.
Conclusions
This work investigates a fundamental question in understanding large language models (LLMs): Is there a component or several components that are responsible for achieving the reasoning ability in LLMs? If the answer is affirmative, which components are responsible for the improvement?
We hypothesize that the output projection (o (o_proj) module plays a central role enabling reasoning capabilities. To support this, propose Stethoscope for Networks (SfN), diagnostic framework

---

that encompasses several probing techniques. Through the proposed Delta, Merge, Freeze, and Destruction stethoscopes, we observe consistent patterns indicating that o_proj is critical for reasoning, while other modules primarily support conversational fluency. These findings open new directions for efficient and modular L.LM training.
Our findings are primarily based on limited set of model families and reasoning benchmarks, and may not generalize to all architectures or tasks. Some diagnostic results rely on qualitative assessments rather than statistical validation. Furthermore, while the role of o_proj is empirically highlighted, theoretical understanding of its function reasoning remains to be established.
Acknowledgments and Disclosure of Funding
This work was partly supported by the National Natural Science Foundation of China under Grant 62276123.
JW proposed the assumptions (Stethoscopes for Networks), hypothesis and conjectures. JS started this line of research in our group, proposed the Zero destructor, and effectively supported our main findings with experimental results. JW and JS wrote the paper.
We thank Ke Zhu for discussions.
References
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309. /4316, 2023.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: P.2t knowledge manipula- tion. arXiv preprint arXiv:2309, 14402, 2023.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Physies of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2-404.05405, 2024.
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 01001660065555 2016.
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Nina Mielke, Alec Radford, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
[8] Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in neural information processing systems, volume 30, 2017.
[9] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Xin Wang, Xingyu Yuan, Adams Yu, Sharan Narang, al. Scaling instruction-finetuned language models. arXiv preprint arXiv: 1416, 2022.
[10] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
[11] Mor Geva, Tal Schuster, and Jonathan Berant. Transformer memories. arXiv preprint arXiv: 2021.
layers are key-value
10

---

[12] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[13] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprini urXiv:2501.12900 2025.
[14] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseck-coder. When the large language model meets programming-the rise of code intelligence. arXiv preprint wrXiv:2401.14199 2024.
[15] John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4129-4138, 2019.
[16] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le. Transformer quality in linear time. In International conference machine learning, pages 9099-9117. PMLR, 2022.
[17] Binyuan Hui, Jian Yang. Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jia- jun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint w4Xiv:2409.12188 2024.
[18] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameyy, et al. Openai ol system card. arXiv preprint arXiv:2412.
2024.
[19] Maxwell Jia. Aime 2024 dataset. http:://wnggiggaaeeccc///tttttts AIME_ 2024 2024.
[20] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[21] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.0838 2020.
[22] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[24] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359 11779,, 2022.
[25] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19399 2025.
[26] Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text sum- marization using sequence-to-sequenee rnns and beyond. arXiv preprint arXiv: 1602.06023, 2016.
[27] Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint 18X1:188000087455 2018.
11

---

[28] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedhack. arXiv preprint arXiv:2203.02/55, 2022.
[29] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
[30] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems,
2023.
[31] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing. Networking. Storage and Analysis, pages 1-16. IEEE, 2020.
[32] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint (:Xiv:66060052550 2016.
[33] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & duta mining, pages 3505-3506, 2020.
[34] Gautam Reddy. The mechanistic basis of data dependence and abrupt learning an in-context classification task. arXiv preprint 2rXiv:2312.03002 2023.
[35] Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. Pro- cedural knowledge in pretraining drives reasoning in large language models. arXiv preprint arXiv:2411.12580, 2024.
[36] Timo Schick, Ananya Dwivedi-Yis Yu, Roberta Raileanu, Saghar Hosseini, Murray Chadwick, Gau- rav Mishra, Siddharth Karamcheti, Neil Houlsby, Aravind Elangovan, Mike Lewis, et al. Tool- former: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.v.170..06347, 2017.
[38] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint Xiv:24402.03300 2024.
[39] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
[40] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501. 12599, 2025.
[41] Qwen Team. QwQ-32B: Embracing the Power of Reinforcement Learning, March 2025.
[42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[43] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
12

---

[45] Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. Xixiv preprint (eXiv:1906.42200 04284, 2019.
[46] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 1rXiv:2203.11171,2000 2022.
[47] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruett Aligning language models with self-generated instruc- tions. arXiv preprini arXiv:2212. 1205000 2022.
[48] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint wwXi:22006006682, 2022.
[49] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thouuh prompting elicits reasoning in large language models, arXiv preprint arXiv:2201.11903 2022.
[50]
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020.
Association for Computational Linguistics.
[51] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint 10Xiv:1609.08144, 2016.
[52] An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang. Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025.
[53] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15111 15115 2024.
[54] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvemmnn, arXiv preprint arXiv:2409. 12/22, 2024.
[55] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming reinforcement learning for open base models in the wild, 2025.
[56] Denny Zhou, Dale Schuurmans, Xuezhi Wang, Ed Chi, and Quoe V Le. Least-to-most prompting enables complex reasoning in large language models, arXiv preprint 2rXiv:2205.10623 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately reflect the key contributions presented in Section 2 and while the experimental findings provide thorough support for the validity of our claims.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper clearly acknowledges its main limitations in Section 6, providing a thorough discussion of the associated constraints and challenges.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification: This paper adopts a primarily empirical approach, with conclusions and methods grounded in and supported by experimental results rather than theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present simple and easily reproducible pipeline, fully described in Sections 2 and 3. All essential hyperparameters and experimental settings are specified, with
additional details for replication provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: The code associated with this paper will be released open-source upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ http:/.cccpuullc//uuucece Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All key experimental selection criteria- are thoroughly described in Sections 2 and 3. Additional experimental details can be found in the appendix.
datasets, hyperparameters, and their
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper includes detailed information on the statistical significance of the experiments provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed information the computational resources used used-including GPU type and execution time-is provided in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have carefully considered and complied with the NeurIPS Code of Ethics throughout the course of our research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: A discussion of potential socictal impacts, including both benefits and risks, is provided in appendix.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not involve models that generate high-risk content or rely web-soureed data, thereby minimizing the need for additional safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All data, code, and methods used in this study are open-source, with proper citations and full compliance with their respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssie thcode wwwccom/aaaoo//c datasetss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any assets; therefore, associated documen- tation is provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or human subject research; thus, this question is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects or crowdsourcing, and therefore requires no study participants, risk assessment, IRB approval.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Large language models play central role in our work, as the research focuses on analyzing, modifying, and evaluating LLM behavior. The core methodology involves fine-tuning and interpreting L.L components, making their usage essential and original to the contributions of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (/crpe///eaarggccc//ooooeeeeeees for what should or should not be described.
20

---

Technical Appendices and Supplementary Material
A Experimental Details
We primarily utilize open -soureedd models to conduct experiments in this work. Given that DeepSeek- RI is one of the most widely adopted reasoning models, and its authors have released series of distilled models based on R1 [3], including both the specified base and finetuned reasoning models, we adopt their configurations in our study. Specifically, we use the DeepSeek- - R1-Distill-Quus Bistill--Qwen [1]
models with sizes of 1.5B, 7B, 14B, 32B and 70B as our reasoning models, and select Qwen2.5-
Math-1.5B, 7B [54], LLaMA3. 1-8B [12], Qwen2.5-14BB 32B [53] or 3.1aa...--700I-tttuu []2] as base models. All models are loaded and run using the Transformers library [50]
Our evaluation framework is based on the seeevallaatonnharnesst package [0] To accelerate inference, we use vLLM 23 as the backend, which may slightly affect performance due to backend-specifie optimizations. In the Merge Stethoscope experiments, we observe that the "chat" interface often generates irrelevant or nonsensical responses, while the "generate" interface produces coherent and contextually appropriate outputs. We suspect this discrepancy arises from misinterpreted system prompts. Therefore, we rely on the "generate" interface and implement custom evaluation toolkit.
For the Freeze Stethoscope experiments, rate of le-5, weight decay of le-4, a batch size of 16, and train for 5 epochs. Due to hardware
build the codebase of s [25]. We use learning
limitations (i.e., lack of access 16 H100 GPUs), we leverage DeepSpeed [33] with ZeRO Stage [53] Evaluation following the modified pipeline by the authors of sl,
3(3 to enable efficient training. The base model used here is S.ven..5-32B-Insse again conducted with revaalaationhaareee which disables generation of the and-of-thinkinn token and optionally appends the string "Wait" to the reasoning trace to encourage model reflection. We adopt the Budget Forcing "Wait" x2 as our default testing configuration.
All visualization and inference experiments on 1.5B-14B models are conducted single NVIDIA A100 GPU. For training and evaluating 32B-70B models, we use a cluster of 8 NVIDIA A100 GPUs. Training typically takes around 6 hours, while testing on single dataset usually requires about hours.
B More Experimental Results
In the main paper, we present visualization results for the 1.5B, 14B, and 32B models. Here, we supplement those results by providing additional visualizations for the 7B, 8B, and 70B models. Following the Delta Stethoscope pipeline, we visualize both the absolute weight shift |ax (B) wx (A)le, and the relative weight shift (B) (4) The absolute weight shifts are shown in (x)
Figure 7 and the relative weight shifts are presented in Figure The trends observed in the main paper remain consistent across these additional models. Notably, o o_proj consistently exhibits the largest weight shift, with the effect being especially pronounced in the 70B model. Moreover, o_proj is the only module that displays a bimodal distribution in the relative weight shift.
C Statistical Significance and Broader Impacts
We report appropriate information regarding the statistical significance of our experiments. While we do not primarily focus on classical significance tests such as p-values, we provide multiple forms of empirical evidence-such under controlled manipulations, and loss curves under different tuning strategies- that collectively establish the robustness of our findings. These analyses serve practical alternative to traditional error bars or confidence intervals and help substantiate our key claims.
consistent module-specific weight shifts, response-level comparisons
This research has both promising benefits and important risks to consider. On the positive side, the proposed Stethoscope for Networks (SfN) framework provides a novel set of tools for interpreting LLMSs, especially by localizing specific Sapabilities-Sure as reasoning- individual components like the output projection (o (o_proj). These tools may significantly improve our understanding of L.LMs, enabling more transparent, modular, and efficient model development. For instance, if reasoning
21

---

Qwas2.5 Math-7B DeepSack-R1-Diiiie --iaalll--uen--7
9,amd
21.08
aet png
u2.5re
10.79
dewegan
1hma-1. 1-38 DespSeoicRi Ri -Datili- FomillLLlmm---B
0.20
kyne
prg
gro
Llua3 3-70nt- vs. DeapSeck-R R--Darilllllee lama-708
*,310
(9te gmy
un.
dowa
Figure 7: Per-module L2 distance of linear weights between models A and B. Notably, the o o_proj module shows the largest in 7B, and 70B models, highlighting its potential importance for reasoning.
Qwen2 MMath 70 vs. DeepSeek RI Ristiiiill- Qwen 7B
Llama 3.1-8B DeepSeck Distil Liana
Jamaa 3-700B-rrstuctt ReepSock-I--Dieillk I-Distill-- R-Dik-ll--lilll-aa----g Joma-708
welf self
self
self proj
MLPuup
MLP gale proj
MLPdown, proj
Figure 8: Layer-war wise distribution of relative weight changes between models A and B. While most modules display a unimodal distribution, the o_proj module uniquely exhibits a bimodal distribution, highlighting its distinctive behavior.
abilities can be enhanced by tuming small subset of parameters, it could greatly reduce computational costs and increase accessibility for developing domain domain-speeific specific lightweight models.
However, this line of work also carries potential risks. Precisely identifying and isolating reasoning- related components might lower the barrier for targeted manipulation, such unauthorized transfer or removal of reasoning abilities across models. This could facilitate misuse scenarios, including capability extraction, tampering, model theft. Furthermore, while the diagnostic methods proposed aim to support interpretability, there a risk that they may be overinterpreted, leading to an inflated sense of model transparency that does not generalize across architectures tasks.
22


Reviews:
[{"review_id": "8uQBGyButu", "reviewer": "Reviewer_GJy8", "summary": "Summary: This paper investigates which components within large language models (LLMs) are primarily responsible for the emergence of reasoning abilities, particularly in mathematical and logical tasks. The authors propose a systematic diagnostic toolkit, \"Stethoscope for Networks (SfN),\" which includes several probing techniques to analyze the internal behavior of LLMs. Through comprehensive empirical studies, the paper shows that the output projection module (o_proj) in the Transformer’s multi-head self-attention mechanism plays a central role in reasoning, while other modules contribute mainly to conversational fluency. The findings present new insights into modularity and targeted training in modern LLMs.", "questions": "1、The analysis shows o_proj is key for reasoning in the tested models, but have you tried the same methods on models with different architectures (e.g., LLaMA, GPT-style, or encoder-decoder models) to see if this conclusion still holds?\n2、In Section 3.2 (Destruction Stethoscope), the conversational ability analysis is mostly based on selected examples. Could you provide more systematic statistics or human evaluations to support the proposed \"division of labor\" between modules?\n3、The paper mentions potential applicability of SfN to other networks (Section 4), but there are no experiments beyond reasoning tasks. Do you have preliminary results or plans to test SfN on abilities such as knowledge retention or code generation?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1、The study introduces a simple yet effective methodology for attributing functional capabilities to specific modules inside LLMs.\n2、The empirical analysis is thorough and well-supported by clear ablation and merge experiments.\n3、Identifying o_proj as critical for reasoning could lead to more efficient and targeted fine-tuning approaches.\n4、The proposed SfN toolkit may be broadly useful for interpretability research beyond reasoning tasks.\nWeakness\n1、Lack of theoretical explanation: The paper mainly relies on empirical analysis and does not provide a deeper theoretical understanding of why the o_proj module dominates reasoning in Transformers.\n2、Limited generalization: Experiments are only conducted on a few open-source models such as Qwen and DeepSeek, and mainly on specific reasoning tasks (e.g., math problems), so it is unclear whether the findings generalize to other architectures and tasks.\n3、Some conclusions rely on qualitative judgments: The division of modules for \"conversational ability\" is mainly based on output examples and observation, lacking large-scale statistical validation and being somewhat subjective.\n4、Generality of the SfN tool is not fully validated: Although the authors propose SfN as a universal diagnostic toolkit, experiments are focused on reasoning modules and tasks, lacking evidence of its effectiveness on other types of models and tasks.", "was_revised": true, "final_justification": "I decide to keep my score.", "rebuttal": "Thank you for your valuable comments and suggestions.\n\n---\n### W1\n\n> … theoretical explanation …\n>\n\nWe fully agree with you that theoretical explanation is missing, which is intentional though. As you mentioned in the review, our empirical analysis of the special role of o_proj is “well-supported”, but we really do not have a corresponding theoretical analysis yet. And, to be honest, theory for LLM in a whole is definitely much weaker than empirical results, which is not limited to our work.\n\nFurthermore, we have a guess (not even a hypothesis) on this. [1] suggests that attention is primarily responsible for retrieving information, while the MLP is responsible for memorization: attention operates across tokens and MLP functions within individual tokens. \n\nThe unique importance of the o_proj layer in reasoning stems from its pivotal position within the Transformer block—**immediately following the attention computation and directly preceding the MLP**. Thus, it is possible (based on [1]) that o_proj receives inputs containing integrated cross-token information, which supports analytical capabilities, and its outputs feed into the MLP, potentially activating internal knowledge.\n\nThis guess might be useful for understanding the theoretical side of o_proj, but we did not include it in our paper intentionally—it is difficult to find supportive evidences either mathematically or empirically for this guess. As such, we prefer to stick to empirical studies. a choice that seems to suit the current stage of LLM research.\n\n---\n### W2 & W4 & Q1 & Q3\n\n> … whether the findings generalize to other architectures and tasks.\n> \n\n> … lacking evidence of its effectiveness on other types of models and tasks.\n> \n\n> …  tried the same methods on models with different architectures … to see if this conclusion still holds?\n> \n\n> …  there are no experiments beyond reasoning tasks. …  test SfN on abilities such as knowledge retention or code generation?\n> \n\nThank you for your suggestions. We agree that it is important to demonstrate the generalizability of our findings. In addition to the results presented in the main paper—primarily based on the Qwen and DeepSeek model series—we also include further evidence using the delta stethoscope method on both **LLaMA-3.1-8B vs. DeepSeek-R1-Distill-LLaMA-8B** and **LLaMA-3.3-70B-Instruct vs. DeepSeek-R1-Distill-LLaMA-70B**. These results, shown in **Figure 1** and **Figure 2** of the **supplementary material** along with our paper, consistently support the same conclusion as in the main text: the o_proj layer exhibits the most significant change. Notably, for LLaMA-3.3-70B-Instruct, this phenomenon is particularly clear and prominent compared to all other models.\n\nFurthermore, we are excited to present delta stethoscope results on **Qwen3-235B-A22B-Instruct-2507 vs. Qwen3-235B-A22B-Thinking-2507**, one of the most advanced open-source models, released just a few days ago. These models adopt a MoE architecture, which differs substantially from standard dense models. Each model consists of 94 layers, with each layer containing 388 linear modules. To compare the \"instruct\" and \"thinking\" variants, we compute the L2 norm of the weight differences for each type of linear module across all 94 layers. Our analysis reveals that the **o_proj module consistently exhibits the largest or second-largest change in every single layer—most often ranking as the largest.** This consistently significant shift in o_proj further reinforces our conclusion about its central role in supporting reasoning within Transformer models.\n\nIn addition, we conducted more experiments using the **freeze stethoscope** method across different models and tasks to strengthen our findings. The experimental settings follow those described in the main paper. Specifically, we report results for the following configurations: baseline, o_proj fine-tuning and full-parameter fine-tuning. As in the original setup, both the embedding and lm_head layers are kept unfrozen by default.\n\n- **Reasoning on other architecture**: We further conduct experiments on reasoning tasks using an additional model. Specifically, we adopt **LLaMA-3.1-8B** as the baseline and perform SFT on the s1k dataset, following the same setup as described in the main paper. The results are shown in the table below. It is evident that fine-tuning only the o_proj layer achieves competitive performance with the significantly fewer trainable parameters, highlighting its critical role in reasoning tasks.\n    \n    \n    | Fintuned Modules | AIME2024 | MATH 500 | GPQA Diamond |\n    | --- | --- | --- | --- |\n    | - (Base model) | 0.067 | 0.456 | 0.283 |\n    | o_proj | 0.133 | 0.584 | 0.323 |\n    | All  | 0.133 | 0.592 | 0.354 |\n\n- **Chemistry:** We also investigate whether our approach generalizes to domain-specific capabilities (**vertical LLM models**). To this end, we use the SMolInstruct dataset [2] for SFT and evaluate the models on several molecular science tasks. We select three representative benchmarks: I2F (IUPAC name to molecular formula), I2S (IUPAC name to SMILES), and SIDER (side effect classification of drugs). The baseline model is **Mistral-7B** [3]. Although we do not fully reproduce the performance reported in the original work—possibly due to differences in training recipes or the total number of training steps—we ensure that all training settings and comparisons are fair and consistent. Notably, our o_proj fine-tuning approach still achieves competitive results.\n\n  | Fintuned Modules | I2F | I2S | SIDER |\n  | --- | --- | --- | --- |\n  | - (Base model) | 0.000 | 0.000 | 0.381 |\n  | o_proj | 0.616 | 0.518 | 0.598 |\n   All | 0.632 | 0.521 | 0.658 |\n\n- **Climate:** We further evaluate our method in another specialized domain—climate science—using the SFT dataset and benchmark introduced in [4]. In this experiment, we adopt **LLaMA‑3.1‑8B** as the base model and evaluate performance on two aspects: answer generation and tool usage. The results show that fine-tuning only the o_proj layer achieves performance comparable to full-parameter fine-tuning, highlighting the critical role of o_proj in adapting to domain-specific tasks again.\n    \n    \n    | Fintuned Modules | Answer | Tool Usage |\n    | --- | --- | --- |\n    | - (Base model) | 0.391 | 0.483 |\n    | o_proj | 0.787 | 0.563 |\n    | All | 0.792 | 0.598 |\n\nWe are sorry that we do not have the resources (time, GPU, etc.) to finish investigation on code generation. But thank you for your suggestion—we do plan to finish it in the near future.\n\n---\n### W3 & Q2\n\n> The division of modules … lacking large-scale statistical validation\n> \n\n> In Section 3.2 (Destruction Stethoscope) … provide more systematic statistics or human evaluations\n> \n\nThat’s a great suggestion—Thank you! To quantitatively assess the model’s capability in daily conversation, we evaluate its **perplexity (PPL)** on the **DailyDialog** dataset [5], which consists of multi-turn dialogues from everyday scenarios. Specifically, we use the test set to compute the PPL of a LLM.\n\nFollowing the destruction stethoscope experimental setup, we select Qwen2.5-32B as the baseline model and re-initialize specific modules within blocks 5 to 30. The resulting PPLs are reported in the table below, where the first row represents the model without any modifications.\n\nFrom the perspective of perplexity, we can directly observe the functional importance of each module in daily dialogue. Notably, the {q_proj, k_proj} and MLP modules are highly critical for conversational coherence, with MLP exerting the most significant impact. In contrast, v_proj has a relatively minor effect—its re-initialization still allows sentence generation, albeit with reduced logical consistency. Interestingly, **o_proj appears to have the least influence** in this context: **re-initializing it results in only slight degradation**, and the model can still produce largely coherent and natural responses. These experiments provide more reliable statistical evidence and further validate the conclusions drawn from the destruction stethoscope analysis.\n\n| Destructed Module | PPL | Destructed Module | PPL |\n| --- | --- | --- | --- |\n| - (Base model) | 10.8 |  |  |\n| q_proj | 212.9 | up_proj | 411.8 |\n| k_proj | 268.6 | gate_proj  | 232.4 |\n| v_proj | 97.2 | down_proj | 505.1 |\n| o_proj | 34.6 |  |  |\n---\n\n[1] Dong, Yihe, et al. \"Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer.\" *arXiv preprint arXiv:2506.01115* (2025).\n\n[2] Yu, Botao, et al. \"Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset.\" arXiv preprint arXiv:2402.09391 (2024).\n\n[3] Jiang, Albert Qiaochu et al. “Mistral 7B.” arXiv preprint arXiv:2310.06825 (2023).\n\n[4] Lyu, Bohan, et al. \"Adapting While Learning: Grounding LLMs for Scientific Problems with Tool Usage Adaptation.\" *Forty-second International Conference on Machine Learning*.\n\n[5] Li, Yanran, et al. \"Dailydialog: A manually labelled multi-turn dialogue dataset.\" *arXiv preprint arXiv:1710.03957* (2017)."}, {"review_id": "c2rBVYAu4d", "reviewer": "Reviewer_6DXm", "summary": "This paper investigates the underlying mechanisms of reasoning abilities in LLMs. The paper hypothesizes that the output projection module within the transformer multi-head self-attention is primarily responsible for reasoning capabilities. To support this hypothesis, authors introduce Stethoscope for Networks (SfN), a suite of diagnostic tools, including the Delta, Merge, Freeze, and Destruction Stethoscopes. Through empirical evidence from these tools, they suggest that the output projection module plays a central role in reasoning, while other modules contribute more to fluent dialogue. The findings offer insights into LLM interpretability and propose avenues for more targeted and efficient reasoning training strategies.", "questions": "- While the paper empirically demonstrates the importance of o_proj in reasoning, a deeper theoretical understanding of why o_proj plays this role is missing. Could the authors elaborate on potential mechanisms or computations within o_proj that might facilitate reasoning? For instance, does it act as a bottleneck for abstract representation, or does it aggregate information in a specific way that enables logical steps?\n- The experiments are primarily conducted on Qwen2.5 models. While the results are consistent within this family, it would be beneficial to discuss the potential generalizability of the findings to other LLM architectures. Are there architectural differences that might influence the role of o_proj in reasoning, or do the authors believe this finding is broadly applicable across modern LLMs?\n- The Freeze Stethoscope section notes that for larger models (e.g., 7B), simply merging o_proj is insufficient, and an o_proj + layernorm merge is considered. Could the authors expand on the specific interplay between o_proj and layernorm in enabling reasoning, especially for larger models? Understanding this interaction might reveal a more nuanced picture of reasoning localization.\n- The paper mentions that fine-tuning with reasoning examples leads to strong reasoning capabilities in model B. Is there any hypothesis or preliminary observation on how the type of training data might influence the specialization of modules, particularly o_proj?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper presents a novel hypothesis regarding the localization of reasoning ability in LLMs, specifically pointing to the o_proj module. The introduction of the Stethoscope for Networks (SfN) framework with its various diagnostic tools provides a systematic and empirical approach to probing LLM internals. They provided extensive experiments, and the consistency of observations across different model sizes adds to their findings. The paper is well-structured and generally easy to follow. The introduction clearly lays out the problem and the paper's hypothesis. Each Stethoscope method is explained with its underlying assumption and how it contributes to the overall hypothesis. The core hypothesis, attributing reasoning capabilities primarily to the o_proj module, is novel and deviates from common assumptions that reasoning stems from the entire model or more complex interactions. However, some recent and relevant studies seem to be missed in related work and discussion, for example:\n\n```\nRuis, Laura, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. \"Procedural knowledge in pretraining drives reasoning in large language models.\" arXiv preprint arXiv:2411.12580 (2024).\n```", "was_revised": true, "final_justification": "Thanks for the detailed responses, which addressed most of my concerns.", "rebuttal": "Thank you for your comments and suggestions.\n\n---\n\n### W1\n\n> … some recent and relevant studies …\n> \n\nThank you. This paper offers valuable insights about reasoning models, and we will discuss it in the related work part.\n\n---\n\n### Q1\n\n> … potential mechanisms or computations within o_proj that might facilitate reasoning …\n> \n\nAccording to [1], the attention mechanism is chiefly responsible for gathering relevant contextual cues, while the MLP predominantly handles the storage and activation of internalized knowledge. \n\nThus, we **guess** that the mechanism by which o_proj facilitates reasoning stems from its **strategic position** within the Transformer block—immediately following the attention computation and directly preceding the MLP. At this junction, o_proj receives inputs that contain rich, integrated cross-token information. Its outputs are then passed into the MLP, potentially activating internal knowledge representations.\n\nIn this sense, o_proj acts as a **bottleneck within the block**—regulating **how much aggregated information** flows from the attention module into the MLP. \n\nWe, however, did not include this guess in our paper, intentionally—it is difficult to find supportive evidences either mathematically or empirically for this guess. Before presenting it to the community, we want to convince ourselves first that this guess have a good chance to be true. We are not there yet.\n\n---\n\n### Q2\n\n> … generalizability of the findings to other LLM architectures.\n> \n\nWe fully acknowledge the importance of demonstrating the generality and robustness of our findings. Beyond the core experiments based primarily on the Qwen series presented in the main text, we provide additional evidence using the delta stethoscope method on two model pairs: **LLaMA-3.1-8B vs. DeepSeek-R1-Distill-LLaMA-8B** and **LLaMA-3.3-70B-Instruct vs. DeepSeek-R1-Distill-LLaMA-70B**. As shown in Figures 1 and 2 of the **supplementary materials**, both comparisons consistently validate the conclusion that the **o_proj layer undergoes the most substantial changes.** In particular, this effect is most striking in LLaMA-3.3-70B-Instruct, where the distinction is significantly more pronounced than in other models.\n\nWe further investigate this phenomenon in a MoE architecture by comparing two recent SOTA open-source models: **Qwen3-235B-A22B-Instruc-2507** and **Qwen3-235B-A22B-Thinking-2507**. We are excited to share our recent findings based on these two newly released models. These models differ significantly from standard dense architectures. Each consists of 94 layers, with 388 linear modules per layer. To compare the \"instruct\" and \"thinking\" variants, we compute the L2 norm of the weight differences for each type of linear module across all layers. Our analysis shows that the **o_proj module consistently exhibits the largest or second-largest change in every layer, most often ranking as the largest**. This consistently prominent shift in o_proj further supports its central role in enabling reasoning within Transformer models.\n\nWe also conduct extended evaluations using the freeze stethoscope method across diverse model families. The setup aligns with that described in the main paper. We compare performance across three configurations: baseline, fine-tuning only o_proj, and full-parameter fine-tuning. Specifically, we applied this framework to **LLaMA-3.1-8B and Mathstral -7B**, and fine-tuned them on the s1k dataset using the same SFT protocol as in the main experiments. As reported in the table, fine-tuning solely the o_proj layer consistently delivers competitive performance, despite involving substantially fewer trainable parameters. This finding again highlights the central role of o_proj in enabling reasoning capabilities in large language models.\n\nTo enhance the robustness of our findings, we conduct extended evaluations using the freeze stethoscope across diverse model families. The setup follows that described in the main paper. We compare three configurations: baseline, fine-tuning only the o_proj layer, and full-parameter fine-tuning, while keeping the embedding and lm_head layers unfrozen in all cases. Specifically, we apply this framework to **LLaMA-3.1-8B and Mathstral-7B**, fine-tuning them on the SFT s1K dataset using the same protocol as in the main experiments. As shown in the table, fine-tuning only the o_proj layer consistently yields competitive performance, despite involving significantly fewer trainable parameters. This result further underscores the central role of o_proj in supporting reasoning capabilities in LLMs.\n\n| Base Model | Fintuned Modules | AIME2024 | MATH 500 | GPQA Diamond |\n| --- | --- | --- | --- | --- |\n| **LLaMA-3.1-8B** | - | 0.067 | 0.456 | 0.283 |\n|  | o_proj | 0.133 | 0.584 | 0.323 |\n|  | All  | 0.133 | 0.592 | 0.354 |\n| **Mathstral  7B** | - | 0.067 | 0.566 | 0.126 |\n|  | o_proj | 0.100 | 0.614 | 0.166 |\n|  | All  | 0.133 | 0.602 | 0.157 |\n\nIn summary, we have evaluated our findings across a range of models, including **LLaMA-3.1-8B** and **Mathstral -7B** (small models), **LLaMA-3.3-70B-Instruct** (a large dense model), and **Qwen3-235B-A22B-Instruct-2507** (a large-scale MoE model). These results collectively demonstrate that our conclusions are broadly applicable across modern LLM architectures.\n\n---\n\n### Q3\n\n> … specific interplay between o_proj and layernorm in enabling reasoning\n> \n\nThat’s a good question. We consider the case of LayerNorm now. Can LayerNorm alone induce reasoning ability? The answer is **no**. Through both the merge stethoscope and freeze stethoscope analyses, we observe that LayerNorm itself does not directly contribute to reasoning. However, it still plays a vital **supporting role**. In most architecture of modern LLMs, the positions of o_proj and LayerNorm are as follows:\n\n$$\nx_{\\text{attn}} = \\text{LayerNorm} \\left( w_o \\left[ \\text{Softmax} \\left( \\frac{(w_q x)(w_k x)^\\top}{\\sqrt{d}} \\right) (w_v x) \\right] +x\\right) \n$$\n\n$$\nx_{\\text{mlp}} = w_{\\text{down}} \\left[ \\sigma(w_{\\text{gate}} x) \\odot (w_{\\text{up}} x) \\right] + x\n$$\n\nThe LayerNorm sits directly **between o_proj and the MLP**. While it contains only a small number of parameters and therefore has limited capacity, it plays a **crucial role as a translator**—mapping the output of o_proj into a form that is compatible and interpretable by the downstream MLP.\n\nThis translation becomes especially important in both the merge stethoscope and freeze stethoscope settings. In these setups, the MLP is held fixed while o_proj is updated. Such updates inevitably introduce a distribution shift in the output of o_proj, which can misalign with the expectations of the fixed MLP. To bridge this gap, it is essential to also update the LayerNorm. By doing so, we ensure that the transformed output remains aligned with the fixed MLP, allowing it to process the information effectively and **enabling reasoning** to emerge.\n\n---\n### Q4\n\n> … how the type of training data might influence the specialization of modules …\n> \n\nThat’s exactly an aspect we are deeply interested in. First, it is important to highlight that **model size significantly influences module specialization**. As shown in Figure 2 of the main paper and Figure 1 of the supplementary material, we observe a clear pattern: when the model is smaller, the MLP layers exhibit larger parameter changes, while o_proj updates are relatively minor. This suggests that **smaller models, which lack sufficient capacity and internal knowledge, tend to rely more heavily on the MLP to memorize training data directly**. \n\nAssuming the model is **sufficiently large**, we then explore how the type of training data affects the functional specialization of different modules. Using Qwen2.5-32B as our base model, we fine-tune on diverse data types, including complex mathematical problems like **s1K**, domain-specific datasets such as **SMolInstruct** (chemistry) [2] and **climate science** data [3]. Across these experiments, we find that o_proj consistently shows the most significant updates, while MLP layers change only modestly. This suggests that when the model already possesses relevant knowledge, and the fine-tuning task involves **applying that knowledge to solve problems**, the o_proj layer becomes the primary site of adaptation.\n\nIn contrast, when the training data contains entirely **new information** **that the model has never encountered before**, the MLP appears to take on a more dominant role. To investigate this, we follow the setup in [4] and fine-tune the model on the **bioR** dataset (a reproduction version on GitHub), which includes synthetic biographical data for 100,000 fictional individuals. Since this data introduces new knowledge rather than tasks requiring reasoning over existing concepts, we observe that the MLP layers undergo substantial updates, while attention-related modules remain relatively unchanged.\n\nBased on these observations, we have the following guess: when the task involves solving problems using existing knowledge, the o_proj layer is the most critical and undergoes the greatest changes. However, when the training data introduces novel knowledge, the model relies more heavily on the MLP to encode and internalize that information.\n\n---\n[1] Dong, Yihe, et al. \"Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer.\" *arXiv preprint arXiv:2506.01115* (2025).\n\n[2] Yu, Botao, et al. \"Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset.\" arXiv preprint arXiv:2402.09391 (2024).\n\n[3] Lyu, Bohan, et al. \"Adapting While Learning: Grounding LLMs for Scientific Problems with Tool Usage Adaptation.\" *Forty-second International Conference on Machine Learning*.\n\n[4] Allen-Zhu, Zeyuan, and Yuanzhi Li. \"Physics of language models: Part 3.1, knowledge storage and extraction.\" *arXiv preprint arXiv:2309.14316* (2023)."}, {"review_id": "efQCMWfubH", "reviewer": "Reviewer_gHUg", "summary": "The paper empirically studies the hypothesis that the model reasoning capabilities attribute to output projection’s parameters in the model multi-head self-attention. The proposed method, Stethoscope for Networks (SfN), compare the weight differences between fine-tuned and base models under controlled modules of merging, tuning, or destruction, and provides interpretable insights into the origin of capabilities like reasoning.", "questions": "1.  How did you unfreeze o_proj, without unfreezing q/k/v_proj? Did you just not update the gradients? If you are not changing what the model pays attention to but how the attention module can map the attended results as o_proj? \n2. Table 2: F2 is based on attention layers updated on pre-trained model activations (by default), no? Did you mean you didn't update the proj_o with the final layer after updating with activations with premature layers? If so, isn't it naturally expected that F2 and F4 shows small changes; F4 achieves a bit of performance gains as it is shown?\n3. What do authors think their novel contributions are from this work, that are beyond the previous observations (MLP=factual storage, attention=reasoning logic, layer updates=all parts contribute, even if frozen at inference)?", "limitations": "Layer updates mean \"reasoning\" isn't isolated. Their conclusion risks a localization fallacy that tuning o_proj improves reasoning. But the experimental setups/results are a bit unclear/weak to conclude that o_proj is the place the reasoning happens. It still seems that they use frozen model activations (which model stores their factual associations) and observing weight changes or successful performance after tuning o_proj with those factual associations do not seem sufficient evidence that reasoning is localized here but rather convincing to say it is a whole computation of model circuits.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### **Strengths**\n\n1. Interesting idea under \"Stethoscope for Networks (SfN) metaphor. Delta/Merge/Freeze/Destruction works well with the ideation with parameter isolation.\n2. Instead of existing work with factual recall, syntax, token, prediction, or attention layers, the work tries to narrow down their scope of exploration even further down into components of attention modules (o_proj, q/k/v_proj).\n3. The question is valuable to ask where the reasoning is localized/attributed primarily.\n\n### **Weaknesses**\n1. Writing needs more work/ esp. introduction (motivation) is a bit chaotic and not compelling.\n- LLMs still struggle with trial-and-error experiments for developing new capabilities -> can you provide more examples? what challenges are we aiming to tackle? \n- Black-box style success -> can we be explicit how it is problematic to recent successes in reasoning capabilities of PPO, DPO, SFT, etc?\n2. The experiments are not riguorous; not compelling if the research question is addressed enough. Table 2: F2 is based on attention layers updated on pre-trained model activations (by default) but without being updated with the final layer. This is naturally expected that F2 and F4 shows small changes.  \n3. The originality of the work is not quite strong. Recent works not limited to classic works [a, b] show that MLP often act like \"fact retriever\" or local updaters while attention moves information around, enabling reasoning chains and variable binding. But that doesn't mean the reasoning only happens in attention outputs as the attention layer updates with model activations that are frozen yet pre-trained knowledge. Not sure how this contribute novel findings beyond already existing works.\n\n\nref\n[a] Transformer Feed-Forward Layers Are Key-Value Memories, EMNLP 2021\n[b] Locating and Editing Factual Associations in GPT, NeurIPS 2022", "was_revised": true, "final_justification": "The author response was somewhat disappointing. \n\nWhile the tone was mostly professional, it included dismissive remarks (e.g., “I don’t know have you heard PEFT or LoRA”) and an inappropriate appeal to other reviewers’ scores to suggest my rating was a misunderstanding. More importantly, when clarification was provided in good faith, the authors did not acknowledge or engage with it, even though they responded to later follow-ups from other reviewers. \n\nAccordingly, several key concerns were deflected, particularly the misinterpretation of post-hoc weight changes in o_proj as causal evidence for reasoning localization. Instead of addressing the conceptual flaws, the response largely reiterated procedural details. \n\nGiven these concerns and after reviewing all the rebuttals, I lean more towards rejection to acceptance.", "rebuttal": "### W1\n\n> Writing ...\n> \n> - LLMs … trial-and-error experiments … -> … provide more examples …\n> - Black-box … -> … how it is problematic …\n\nThese comments are not intended to dismiss the remarkable achievements of LLMs, but rather to highlight that there remains considerable room for improvement. I have hands-on experience pre-training and fine-tuning LLMs at the 8B scale. My typical training pipeline involves collecting large volumes of high-quality data, followed by empirical validation through training runs. However, in practice, the outcome is often **unpredictable**—some data that appear to be high quality fail to yield meaningful improvements on specific benchmarks. Based on this experience, I believe the development of LLMs still involves substantial trial and error.\n\nIndeed, SFT, RLHF, and reinforcement learning for reasoning have all demonstrated significant success. However, we still lack a deep understanding of **how** these models actually work. As Geoffrey Hinton has repeatedly emphasized, LLMs remain largely opaque—we still know very little about their internal mechanisms [1]. The **debate over reasoning capabilities** further underscores this opacity. For example, works such as [2,3,4] argue that reasoning success can be attributed to data contamination or memorization rather than true reasoning. However, many researchers strongly disagree, and the debate is ongoing.\n\nRegardless of which side is correct, the core issue is clear: **we lack a mechanistic understanding of LLMs, especially for reasoning tasks.** We believe treating them as black boxes is insufficient. Our work aims to shed light on the inner workings of reasoning in LLMs through novel analytical techniques.\n\nAs for the writing, we believe the logic and structure of the paper are clear and coherent. Notably, the other three reviewers rated the paper’s **clarity as “3: good,” “4: excellent,” and “4: excellent.”** In contrast, your rating of “1: poor” suggests there may have been a misunderstanding during your review. **If further clarification is needed, we would be more than happy to engage in discussion.**\n\n---\n\n### W2 & Q1 & Q2\n\n> Table 2: F2 is based on attention layers updated on pre-trained model activations (by default) but without being updated with the final layer. This is naturally expected that F2 and F4 shows small changes.\n> \n\n> Table 2: F2 is based on attention layers updated on pre-trained model activations (by default), no? Did you mean you didn't update the proj_o with the final layer after updating with activations with premature layers? If so, isn't it naturally expected that F2 and F4 shows small changes; F4 achieves a bit of performance gains as it is shown?\n> \n\nWe apologize, but **we are genuinely unclear about the exact meaning of your questions.** Nevertheless, we have attempted to interpret them to the best of our understanding. **If our interpretation does not align with your intention, we would be happy to engage in further clarification.**\n\nFirst, when you mention that “F2 is … updated on pre-trained model activations without being updated with the final layer” could you please clarify what you mean? As far as we understand, this seems to reflect a misunderstanding of the actual computation process. Our SFT is performed based on the model’s **final output**. The training procedure is **identical to** **standard** **SFT** for LLMs or typical end-to-end training in neural networks\n\nIt’s unclear whether you mean that F2 shows only small changes compared to F4, or that both F2 and F4 exhibit small changes compared to the baseline A. If you meant the former, it is not self-evident that F2 and F4 would yield only minor differences. For example, in the 32B model, F2 involves just **3.2B** unfrozen parameters, while F4 has **32.8B**—**ten times more**. If you meant the latter, that also doesn’t seem accurate, as Table 2 clearly shows that both F2 and F4 achieve **significant improvements on reasoning benchmarks** compared to the baseline A.\n\n> How did you unfreeze o_proj, without unfreezing q/k/v_proj? Did you just not update the gradients? If you are not changing what the model pays attention to but how the attention module can map the attended results as o_proj?\n> \n\n**Unfreeze only a part of parameters is very common** in LLMs and diffusion. (I don’t know have you heard PEFT or LoRA). In our setting, we unfreeze only the o_proj parameters while keeping the {q,k,v}_proj frozen. This is done by setting `requires_grad=True` for `o_proj` and `requires_grad=False` for the others. \n\nWhat exactly do you mean by “if you are not changing what the model pays attention to but how the attention module can map the attended results as o_proj”?  To clarify, let’s write out the attention computation: \n\n$$\nx_{\\text{attn}} = \\text{LayerNorm} \\left( w_o \\left[ \\text{Softmax} \\left( \\frac{(w_q x)(w_k x)^\\top}{\\sqrt{d}} \\right) (w_v x) \\right] +x\\right) \n$$\n\n$$\nx_{\\text{mlp}} = w_{\\text{down}} \\left[ \\sigma(w_{\\text{gate}} x) \\odot (w_{\\text{up}} x) \\right] + x\n$$\n\nIn our paper, the o_proj refers to the **weight matrix** $w_o$, which is the **linear layer** following the attention mechanism. Even when only o_proj is unfrozen, **the attention maps can still be indirectly affected.** For example, updating the o_proj in the first layer modifies its output, which in turn alters the input to the next layer—including its attention map. Thus, even if only o_proj is unfrozen, its influence can still propagate **throughout the model.**\n\n---\n\n### W3 & Q3\n\n> originality ...  classic works [a, b] … But that doesn't mean the reasoning only happens in attention outputs …\n> \n\n> … novel contributions … that are beyond the previous observations …\n> \n\nWe would like to emphasize that the roles of various modules in LLMs—including even the division of responsibilities between attention and MLP layers—remain open research questions that are still actively studied in the community [5,6]. Moreover, while the works you referenced are indeed insightful, they were published **prior to the emergence of the first dedicated reasoning models** (such as o1). As such, their research objectives differ fundamentally from ours. These earlier studies were not designed to address the central question we focus on: **who or what is responsible for reasoning** *in reasoning-capable LLMs*.\n\nIn contrast, our work explicitly addresses this question. If you search for it online, you will find that this question **has not been clearly posed or investigated before**. We believe this is a novel and important research direction, especially given the current ongoing debates around whether reasoning in LLMs is real or merely a byproduct of memorization or data leakage.\n\nAdditionally, the hypothesis you mentioned—such as \"attention equals reasoning logic\"—has not been supported by concrete experimental validation, to our knowledge. Our work is the **first** to systematically pose this question and provide a suite of tools to explore it. Our findings may contribute to more efficient training or the integration of thinking and non-thinking models. We believe it is a meaningful and timely contribution to the field.\n\n---\n\n### L1\n\n> … the experimental setups/results are a bit unclear/weak … they use frozen model activations … do not seem sufficient evidence ...\n> \n\nFirst, we would like to clarify that **activations are not frozen** in our experiments—we believe there may have been a fundamental misunderstanding of our methodology throughout your review.\n\nSecond, we have conducted a **comprehensive series of experiments** across a wide range of models. Specifically, we performed **delta stethoscope** analyses on the following pairs:\n\n- Qwen2.5-Math-1.5B vs. DeepSeek-R1-Distill-Qwen-1.5B\n- Qwen2.5-Math-7B vs. DeepSeek-R1-Distill-Qwen-7B\n- LLaMA-3.1-8B vs. DeepSeek-R1-Distill-LLaMA-8B\n- Qwen2.5-14B vs. DeepSeek-R1-Distill-Qwen-14B\n- Qwen2.5-32B vs. DeepSeek-R1-Distill-Qwen-32B\n- LLaMA-3.3-70B-Instruct vs. DeepSeek-R1-Distill-LLaMA-70B\n\nWe also extended our analysis to **Qwen3-235B-A22B-Instruct-2507** and **Qwen3-235B-A22B-Thinking-2507** (see our response to Reviewer 6DXm), and consistently found that **o_proj shows the most significant changes** across all these model comparisons.\n\nIn addition to delta stethoscope, we applied the **merge stethoscope** and **destruction stethoscope** methods. For the **freeze stethoscope**, we conducted experiments on **Qwen2.5-14B** and **Qwen2.5-32B** with reasoning tasks, and further extended our evaluation to **LLaMA-3.1-8B**, **Mathstral-7B**, and **Mistral-7B** across **reasoning, chemistry, and climate** domains.\n\nAcross all experiments, the results consistently support our conclusion that **o_proj is the most critical component for enabling reasoning and other vertical (domain-specific) capabilities**. You mentioned that our evidence is insufficient—we genuinely welcome the opportunity to provide **additional experiments or analyses** to clarify or strengthen our conclusions. Please let us know if there are particular aspects you’d like us to elaborate on.\n\n---\n[1] https://www.newyorker.com/magazine/2023/11/20/geoffrey-hinton-profile-ai\n\n[2] Shojaee, Parshin, et al. \"The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity.\" *arXiv preprint arXiv:2506.06941* (2025).\n\n[3] Shao, Rulin, et al. \"Spurious rewards: Rethinking training signals in rlvr.\" *arXiv preprint arXiv:2506.10947* (2025).\n\n[4] Wu, Mingqi, et al. \"Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination.\" *arXiv preprint arXiv:2507.10532* (2025).\n\n[5] Allen-Zhu, Zeyuan, and Yuanzhi Li. \"Physics of language models: Part 3.1, knowledge storage and extraction.\" *arXiv preprint arXiv:2309.14316* (2023).\n\n[6] Dong, Yihe, et al. \"Attention Retrieves, MLP Memorizes: Disentangling Trainable Components in the Transformer.\" *arXiv preprint arXiv:2506.01115* (2025)."}, {"review_id": "aBEdgLqROB", "reviewer": "Reviewer_bgq1", "summary": "This paper investigates the reasoning of LLMs, namely, which part of the LLM is responsible for the reasoning abilities.\nThey do so by using 4 investigation tools given a base model A and a reasoning model B: comparing the weights of A and B, merge the weights of A and B, finetuning A with frozen components, destroying components in A.\n\nThey pinpoint the reasoning module to the output_projection of multi-head self attention.", "questions": "Did you try measuring the impact of the \"output_proj\" layer on tasks that do not involve reasoning?", "limitations": "The only benchmarks used are reasoning ones, curious what is the effect on non-reasoning benchmarks", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "* Strength *\n\n1. This is an interesting research question\n2. It has large potential of influence on training of LLMs\n3. Nice methodology\n\n* Weaknesses *\n\nMy main remark is that the one of the roles of the output_projection in the MHSA layer is to map from the attention outputs to the mlp inputs. I'm not convinced that the lack of effect of the \"q, k, v\" in the merge setup is not due to their lack of importance but a mismatch between their outputs and mlp inputs.", "was_revised": true, "final_justification": "I leave my score as it is a good paper worthy of being at NeurIPS", "rebuttal": "Thank you for your comments and suggestions.\n\n---\n### W1\n\n> the lack of effect of the \"q, k, v\" in the merge setup is … a mismatch between their outputs and mlp inputs.\n> \n\nThank you for sharing your comments. Indeed, we have considered the possibility you raised. In the merge setup you mentioned, LayerNorm is not merged across all configurations. Therefore, the potential mismatch between the attention output and the MLP input exists not only when merging {q,k,v}_proj, but also when merging o_proj. **In both cases, the input to the MLP deviates from its original distribution, so the comparison remains fair**. Even under this mismatch, merging o_proj still yields substantial improvements, which underscores its unique importance.\n\nMoreover, the role of o_proj—as you pointed out—is to bridge attention outputs and MLP inputs, and this is precisely why we believe it plays a more crucial role in enabling reasoning compared with {q,k,v}_proj.\n\n---\n\n### Q1 & L1\n\n> … on tasks that do not involve reasoning?\n> \n\n> … what is the effect on non-reasoning benchmarks\n> \n\nWe further extended our experiments using the freeze stethoscope method on several non-reasoing tasks to test the generality of our conclusions. The experimental setup is consistent with that outlined in the main paper. We present results under three configurations: the original pretrained baseline, o_proj-only fine-tuning, and full-parameter fine-tuning.\n\nIn the domain of **chemistry**, we examine whether the observed importance of o_proj transfers to specialized scientific tasks. We fine-tune models on the SMolInstruct dataset [1] and evaluate performance on three benchmarks: I2F (converting IUPAC names to molecular formulas), I2S (converting IUPAC names to SMILES strings), and SIDER (predicting drug side effects). Mistral-7B is used as the base model. While we do not fully match the results reported in the original SMolInstruct paper, we take care to maintain consistency and fairness across all experimental settings. Fine-tuning only the o_proj layer yields strong performance while modifying far fewer parameters, highlighting its practical value in domain-specific applications.\n\n| Fintuned Modules | I2F | I2S | SIDER |\n| --- | --- | --- | --- |\n| - (Base model) | 0.000 | 0.000 | 0.381 |\n| o_proj | 0.616 | 0.518 | 0.598 |\n| All | 0.632 | 0.521 | 0.658 |\n\nIn the field of **climate science**, we further assess the generalizability of our method using the SFT dataset and benchmark proposed in [2]. For this experiment, we use LLaMA-3.1-8B as the backbone model and evaluate its performance on two task dimensions: generating answers and interacting with tools. The findings indicate that fine-tuning solely the o_proj layer yields results on par with full-parameter tuning, once again underscoring the pivotal function of o_proj in adapting large models to specialized domains. \n\n| Fintuned Modules | Answer | Tool Usage |\n| --- | --- | --- |\n| - (Base model) | 0.391 | 0.483 |\n| o_proj | 0.787 | 0.563 |\n| All | 0.792 | 0.598 |\n\nI hope these experiments demonstrate the effectiveness of our method on non-reasoning tasks. **If further experiments or discussion are needed, we would be happy to continue the conversation!**\n\n---\n\n[1] Yu, Botao, et al. \"Llasmol: Advancing large language models for chemistry with a large-scale, comprehensive, high-quality instruction tuning dataset.\" arXiv preprint arXiv:2402.09391 (2024).\n\n[2] Lyu, Bohan, et al. \"Adapting While Learning: Grounding LLMs for Scientific Problems with Tool Usage Adaptation.\" *Forty-second International Conference on Machine Learning*."}]

Author rebuttal:

Meta-review: This paper presents a novel investigation into the localization of reasoning abilities within LLMs. The authors hypothesize that mathematical reasoning capabilities are not diffusely distributed throughout the model but are primarily concentrated in the output projection module (o_proj) of the Transformer's multi-head self-attention mechanism. To probe this hypothesis, they introduce a new suite of diagnostic tools named Stethoscope for Networks (SfN). Using SfN, the authors provide empirical evidence suggesting that the o_proj module is indeed central to reasoning tasks, while other components may be more involved in maintaining fluent and coherent dialogue.

The paper received a mixed set of reviews. After a careful reading of the paper and the reviewers' comments, I find myself agreeing with the more positive assessments. The work's primary contribution is its fresh perspective on LLM interpretability. The hypothesis that a complex capability like reasoning could be localized to a specific submodule is a bold and interesting one. While the evidence is indeed empirical, this is characteristic of much of the current research in deep learning, and framing the central claim as a "hypothesis" or "conjecture" is appropriate. 

The findings, even though not theoretically proven, are likely to be of interest to the community and could inspire new lines of inquiry into the functional specialization of LLM components. Furthermore, the introduction of the SfN is a potentially valuable contribution in its own right. Diagnostic tools that allow for a more granular analysis of LLM internals are crucial for advancing our understanding. I believe for a conference like NeurIPS, there is value in accepting work that is thought-provoking and opens up new avenues for exploration, even if it is primarily empirical. This paper has the potential to spark important discussions and shift how we think about training and interpreting these complex models. Therefore, I lean towards acceptance.
Decision: Accept (poster)