Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs
Authors: Honglin Zhang
Qianyue Hao
Fengli Xu
Yong Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models; Reinforcement Learning Fine-Tuning; Edge Attribution Patching
Primary area: interpretability and explainable AI
TL;DR: This work utilizes edge attribution patching (EAP) to investigate the internal differences of LLMs before and after RL fine-tuning, and uncovers that RL enhances activation intensity and diversity in the internal circuitry of LLMs.

Abstract: Large language models (LLMs) acquire extensive prior knowledge through large-scale pretraining and can be further enhanced via supervised fine-tuning (SFT) or reinforcement learning (RL)-based post-training.
A growing body of evidence has shown that RL fine-tuning improves the capability of LLMs beyond what SFT alone achieves.
However, the underlying mechanisms why RL fine-tuning is able to enhance the capability of various LLMs with distinct intrinsic characteristics remain underexplored.
In this study, we draw inspiration from prior work on edge attribution patching (EAP) to investigate the internal differences of LLMs before and after RL fine-tuning.
Our analysis across multiple model families and mathematical datasets shows two robust effects of online RL post-training: (i) an overall increase in average activation intensity, indicating that more internal pathways are engaged and their signals become stronger, and (ii) greater diversity in activation patterns, reflected by higher entropy and less concentrated edge distributions. These changes suggest that RL reshapes information flow to be both more redundant and more flexible, which may explain its advantage in mathematical generalization. Notably, models fine-tuned with Direct Preference Optimization (DPO) deviate from these trends, exhibiting substantially weaker or inconsistent internal changes compared to PPO- and GRPO-based training. Together, our findings provide a unified view of how RL fine-tuning systematically alters the internal circuitry of LLMs and highlight the methodological distinctions between online RL and preference-based approaches.
Our code is open source at https://github.com/tsinghua-fib-lab/llm_rl_probing_analysis.

Published as a conference paper at ICLR 2026
REINFORCEMENT LEARNING FINE-TUNING EN- HANCES ACTIVATION INTENSITY AND DIVERSITY IN THE INTERNAL CIRCUITRY OF LLMS
Honglin Zhang' Qianyue Hao; Fengli Xu, Yong Li Department of Electronic Engineering, BNRist, Tsinghua University Beijing, China
ABSTRACT
Large language models (LLMs) acquire extensive prior knowledge through large- scale pretraining and can be further enhanced via supervised fine-tuning (SFT) or reinforcement learning (RL)-based post-training. A growing body of evidence has shown that RL fine-tuning improves the capability of LLMs beyond what SFT alone achieves. However, the underlying mechanisms why RI. fine-tuning is able to enhance the capability of various LLMS with distinct intrinsic characteristics remain underexplored. In this study, we draw inspiration from prior work edge attribution patching (EAP) to investigate the internal differences LLMs before and after RL fine-tuning, Our analysis across multiple model families and mathe- matical datasets shows two robust effects online RL post-traut tostntrainin:: (i) an overall increase in average activation intensity, indicating that more internal pathways are engaged and their signals become stronger, and (ii) greater diversity in acti- vation patterns, reflected by higher entropy and less concentrated edge distribu- tions. These changes suggest that RL reshapes information flow to be both more redundant and more flexible, which may explain its advantage in mathematical generalization. Notably, models fine-tuned with Direct Preference Optimization (DPO) deviate from these trends, exhibiting substantially weaker or inconsistent internal changes compared to PPO- and GRPO based training. Together, our find- ings provide a unified view of how RL fine-tuning systematically alters the in- ternal circuitry of LLMs and highlight the methodological distinctions between online RL and preference- based approaches. Our code is open source at hhtpp:: 7/github. con/t wwwoo//tsinguuua.aaoooo fib lab/ habbllmm rl probing. analysis
INTRODUCTION
Recent strides in large language models (LLMs) have shifted the developmental focus from pre- training to post-training (Kumar al., 2025). A wide array of post-training strategies, ranging from supervised fine-tuning (SFT) (Dong et al. 2023) to reinforcement learning (RL) (Zhang al. 2025b Hao et al., 2025), has been developed to enhance model performance. Particularly, RL-based fine-tuning has witnessed rapid advancements, encompassing the development of reward models from Outcome Reward Models (ORM) (Lyu et al. 2025) to Process Reward Models (PRM) (Light-) man et al. 2023 Yuan et al. 2024). alongside tramiin algorithms like Proximal Policy Optimiza- tion (PPO) (Schulman et al. 2017) and Group Relative Policy Optimization (GRPO) (Shao et al.) 2024 With such advancements, emerging empirical evidence indicates that RL-based fine-tuning can enhance the capability of LLMs beyond what is achieved by SFT alone (Chu et al. improving performance across range of downstream tasks, including writing (Liao et al.
2025
2025
reasoning (Guo et al. 2025 Xu et al. 2025), and coding (Guo et al. 2024).
Seeking to understand the role of different components within Large Language Models (LLMs) and the origins of their powerful capabilities, growing body of research has focused on probing their internal structures. Initial studies revealed the working mechanisms of L.LMs when solving
Honglln Zhang and Qianyue Hao contribute equally to this work. Yong Li is the corresponding author, email: 1 1iyong78tss ngghua edu.cn

---

Published as a conference paper at ICLR 2026
mathematical problems by analyzing and statistically examining their internal weights (Shao et al. 2023). Subsequently, some research has analyzed patterns in LLM weights by training external neural probes, which are lightweight auxiliary models (Kim et al., 2025 Zheng et al., 2025). Re- cently, researchers have investigated the internal residual pathways of LLMs from a graph-theoretic perspective. They have developed methods such as Automated Circuit Discovery (ACDC) (Conmy it al. 2023) and Edge Attribution Patching (EAP) (Syed et al., [2023] Hanna et al. 2024), which as- sign importance scores edges or sub-modules and reveal internal functional circuits that determine the capabilities of LLMs.
Despite these advances, existing studies on RL-based post-training have predominantly focused on the external behavioral changes of LLMs, while the underlying internal mechanisms remain under- explored (Ren & Sutherland 2024). Conversely, works that do investigate the internal mechanisms concentrate on given LLMs, but do not correlate the internal mechanisms to the RL.-based post- training methodology with which the L.MSs are commonly obtained (Hanna et al. 2024 Kim al.) ).25). As result, the two lines of research, external evaluation of RL effects and internal mech- anistic analysis, have largely progressed in parallel. This gap is partly due to the primary goal of RL wost-training, namely enhancing the ability of LLMs to solve complex reasoning tasks, which makes nontrivial to directly transfer analytical strategies developed on toy problems to the study of RL-induced improvements in real-world problem-solving capabilittes.
To address this, we construct a framework for systematically analyzing the mechanisms through which RL fine-tuning affects LLMs. Specifically, we adopt efficient Edge Attribution Patch- ing (EAP) framework (Nanda 2023). leveraging the cross-entropy computed from partially trun- cated generations on mathematical problem-solving tasks estimate the contribution weights of internal edges. Based on these estimated importance weights, we analyze their distributions be- fore and after RL fine-tuning to interpret changes in internal neuron activations and derive general conclusions regarding the structural effects of RL in the context of mathematical problem solving. Experiments across multiple L.LM pairs on diverse mathematical datasets demonstrate that RI post- training strengthens the activation intensity of internal edge connections and diversifies activation patterns during groblem-solving. Notably, these effects are not consistently observed under DPO training. highlighting differences between DPO and other RL paradigms, which aligns with prior observations in the literature (Xu et al. 2024).
Overall, the uncovered patterns hold across diverse LLM families, each with distinct characteristics such as architecture and training corpus, suggesting a set common internal effects induced by RL fine-tuning reasoning-heavy tasks. These findings provide new insights into how RL post- training reshapes the internal circuitry of L.LMs, thereby bridging empirical performance gains with interpretable shifts in internal information pathways. In doing so, they offer guidance for the future development of both LLMS and post-training methodologies.
PRELIMINARIES
2.1 LARGE LANGUAGE MODELS
Large language models (LLMs) are typically built upon the Transformer architecture, comprising a stack of L identical layers (Vaswani et al.] 2017: Liu et al.) [0224] Bai et al. 2023 Achiam et al. 2023). Each layer consists of two primary sub-structures: a multi- head self-attention mechanism and position-wise feed-forward network (FFN), each surrounded by residual connection. The mathematical formulation described below represents the most common architecture found con- temporary LLMs. Let (2f)) E R Bx dosad denote the input hidden state to the (E + 1)-th layer, where B is the batch size, P is the sequence length, and dmodel is the hidden dimension. Specifically, the raw input embeddings are denoted by Xispor = (0)
The output of the l-th layer, (29,, is computed via the sequential processing of the attention and FFN sub-structures. For the attention sub-structure, the input is first normalized as X LayerNorm (H The attention mechanism is then applied:
(X W 9)
(X wE)")
Attention (X = O atn softmax
(X) W
(1)
ddk
2

---

Published as a conference paper at ICLR 2026
where W W IR dlmoee dapary W R ddmm W Rdhm x X output projection matrices, respectively. Here, dquery is the dimensionality of the query and key vec- tors, and dam represents the dimensionality of the value vectors within the attention computation. Positional embeddings are omitted for simplicity. The residual connection yields the intermediate state: H (2 = H 2) + O attn The FFN sub-structure then processes H(2(-1) after normaliza-
the query, key, value and
tion: X = Lare (H (2)).
The FFN employs a gated mechanism with parallel pathways:
FF in) O in
(Activation (Xin W galee
(X W W ddwn'
gate E R. Aoadd xds W E Rd x and W down E dR d are learned weight matrices, denotes element-wise multiplication, and der is the expanded inner dimension of the FFN. The final output of the layer is obtained via another residual connection: H H O After processing by all L layers, the final hidden states H2 are projected to vocabulary logits via:
where W
up
( LayerNorm H (2())) WT
L ==
emb.
(3)
where W emb E RR x dantel the output embedding matrix and the vocabulary size. The resulting tensor L E IR BxPXV contains the unnormalized logits for each token position.
2.2 UNIFIED VIEW OF LLM POST- TAST-TRINNNG
Previous studies have shown that various post-training methods can be expressed within a unified framework (Shao et al. 2024), encompassing both supervised fine-tuning (SFT) and reinforcement learning (RL)-based approaches. Let Ty denote the current policy parameterized by 9, and let (q. o) represent response pair. The update rule of generic post-training algorithm A can then be written in gradient form as
lo
VoJA( = E (L
GCAlg, o, Ted, Tett Vo log (o q, 0202))
(4)
where specifies the sampling distribution that generates the training pairs (g. o), Tya denotes the reward model or evaluation rule that produces the learning signal, Tref is the reference policy used to anchor relative preference or advantage computations, and GC represents the token-level weight- ing factor derived from these signals in algorithm A. This abstraction places different post-training approaches within a unified mathematical representation, enabling direct comparison between su- pervised and reinforcementddriven update mechanisms.
3 METHOD
Our methodology based on the Edge Attribution Patching (EAP) framework (Syed et al.) 2023 Hanna et al. 2024 Nanda 2023), which adopts graph-theoretic view of LLMs their residual pathways, reflecting perspective that has long been present in prior research. While the original work focuses on automated circuit discovery, we adapt its core principle of deriving gradient- attribution scores for edges to analyze internal information flow differences between models before and after reinforcement learning (RL) fine-tuning.
3.1 GRAPH VIEW OF TRANSFORMER RESIDUAL COMPUTATION
Owing to the residual connections in Transformer layers, the input any sub-module, whether an attention branch or an FFN branch, corresponds to the sum of all preceding sub-module outputs, including the original embedding input. For simplicity, let the attention branch transformation be denoted as Of atto A (H))) and the FFN transformation attention block, the input is projected to compute queries, keys, and values. These projections can be viewed as three parallel computational paths. For clarity of exposition and ease of presentation, the attention block is treated as a single composite transformation and these internal branches are not explicitly expanded. Then the hidden states satisfy:
Of F (H())+1)). Within each
E+1
H (2)
H (0)
+
O'
+
Σ
0
fin
H (2(+1)
H
Σ
O
Σ
02
(5)
1=1
3

---

Published as a conference paper at ICLR 2026
(4)
(0)
Target
0%
Imput
1077
Torget
A
Impot
191)
Ons
NP)
Target
Input
(Rim Fenrrouren)
((y:
Figure 1: Schematic of two-layer simplified LLM. (a) Residual perspective, (b) graph perspective, and (c) edge importance estimation: above the dashed line, ACDC-style methods measure the loss change after edge ablation (2) (), and below, EAP-style methods approximate this via backprop- agated gradients
3 ~ D).
Consequently, each sub-module, namely any attention block A feed-forward block F can be interpreted as node in a directed graph. Let us define the set of nodes as
v= (A' A2, A',F),
(6)
where HD corresponds to the original embedding input. The directed edges, representing the flow of information from sub-module outputs to subsequent inputs, can be formalized as
& { H(O) ( u
{( H(( (o H(T) |ssisesl] (7)
Thus, the LLM can be represented as directed acyclic graph (DAG) g (V, in which nodes correspond to individual sub-modules and edges encode the residual information pathways. This graph-theoretic abstraction facilitates analysis of the model both from network flow perspective and a circuit-based interpretability standpoint, and for more intuitive comparison of the residual stream view and the graph view, see Fig. I[a and (b).
3.2 EDGE-LEVEL ATTRIBUTION
To quantify the importance of individual residual edges, prior work like the Automated Circuit Dis- covery (ACDC) evaluates the change in loss when a given edge is removed (Conmy et al. 2023). Concretely, let (O, H) & denote a directed edge from output O of some sub-module to hidden representation H at a subsequent stage. ACDC defines the edge importance by the loss perturbation:
JACDe(O, (0, H) L (y; (()
L (y: f(x)),
where f(x) is the model output under input x, L(y:.) denotes the supervised loss relative to target y, and fV.. represents the model with edge (O, H) ablated (i.e., setting the corresponding con- tribution zero). While conceptually straightforward, this procedure requires forward passes per edge, rendering it computationally infeasible for large-scale attribution.
By contrast, the EAP framework proposes gradient-based linearization that estimates the same loss perturbation more efficiently. Specifically, for given edge (O, H), consider the ablation H + H H--0 O, which corresponds removing O's contribution. A first-order Taylor expansion around H yields the following compact expression:
AL(O, - (Vn O) (OA O),
where CHC(y; f(x)) E R Bx is the loss gradient with respect to the hidden state H, and (()) denotes the Euclidean inner product.
Considering the computational cost of analyzing large-scale LLMs, adopt IEAP to estimate edge- level importance. Importantly, JAAP can be computed for all edges simultancously with a single forward and backward pass under the zeroing perturbation, as both the forward activations O and
4

---

Published as a conference paper at ICLR 2026
the backward gradients VHC available. This approach enables scalable, fine-grained circuit analysis without the need for separate per-edge ablations, making it tractable even for very large models. For more intuitive comparison of ACDC-style ablation and EAP-style gradient-based attribution, see Fig. Cc).
3.3 SAMPLE SELECTION AND TOKEN-LEVEL TRUNCATION
To ensure fair and tractable edge attribution analysis, we implement systematic filtering and trun- cation procedure on model-generenn token sequences. Let each model in paired set generate token sequence shuse (s) Trme and s (sf1. ...... Tia for given question, where
Tase and TRL are the respective sequence lengths.
Question Filtering. We first select only questions that are correctly answered by both models, and denote the resulting set as Q. To mitigate biases caused by extremely short or long answers, we compute the mean token length across all selected questions for given model pair and dataset:
7 base
RL.
T
(10)
2
TEQ
We then define minimum and maximum allowable lengths, Tiin 3T, Timax TT, and retain only questions satisfying Tuin S Tuss and TR. S Tmax-
Finally, to control for comparable sequence lengths between the base and RL models, we require
T base RL 6,
(11)
(T hase + 1/2
where 8 E (0, 1) is balance coefficient. This ensures that the selected questions are comparable in length across both models, minimizing biases in edge importance estimates.
Token Truncation and Self-Entropy Computation. For the filtered set of questions, define truncation length Tem = a T, where & 0 is a scaling coefficient. Only the first Teu tokens each sequence used. Let L IRV denote the model's logit output at token position and let $1:2.. be the sequence of generated tokens truncated to Teun- We compute the self-entropy ecross-entrope of the model with respect to its own output)
T
exp(LI(st]
Liune
log
(12)
Teut
20 cxp(L,(u]))
where 8, denotes the token actually generated position by the model itself.
This ensures that edge importance is computed based on each model's 's truncated output, maintaining comparability across sequences while avoiding excessive memory usage for overlong generations.
4
EXPERIMENT
4.1
EXPERIMENTAL SETTINGS
In our experiments, to ensure both reproducibility and the generality of the conclusions, we em- ployed four pairs of open-source large language models (LLMs) of approximately 7B parameters, each consisting of a base model and its counterpart after post-training:
Deepseck-Matt (hao et al. 2024): are official DeepSeek models based on the LLaMA-style Transformer. deepseek-math-7b-inng is instruction- -uuned on mathematical datasets such GSM8K, MATH. and MathInstruct, while deepseek-matie r/ is further trained from it with reinforeement learning on GSM8K and MATH using the Group Relative Policy Optimization (GRPO) algorithm.
Both deepseek-math-22 -aate------mt----------cc and deepseek-matt--7b-l
Mistral (Chaplot 2023 Wang et al. 2023): mistral-7b-sth is supervised fine-tuned version of the Mistral-7B model on the MetaMATH dataset, while maahh-shephrrd-mistree mistral---tttt--- fur- ther optimized from it using step-by-step Proximal Policy Optimization (PPO) guided by the
5

---

Published as a conference paper at ICLR 2026
Table 1:
Comparison of four model pairs (SFT vs. RL) across three datasets, three evaluation
metrics, and four hyperparameter settings. Missing values result from GPU memory overflow.
Dataset
Metric
Scale
Deepseek-Math
Mistral
Distilled-Qwer
Qwen2.5
SFT
+GRPO
SFT
+PPO
SFT
+GRPO
SFT
+DPO
Act.
0.03
2.29c-3
2.64e-3
9.47e-7
3.61e-6
6.18c-4
6,87e-4
1.11c-3
1.13e-3
0.1
1. 1.10c-3
1.31e-3
6.76c-4
7.71e-4
4.51e-4
5.59e-4
6.95e-4
6.90c-4
Intens.
0.3
7.47e-4
7.77e-4
4.49c-4
4.92e-4
4.39e-4
4.21e-4
0.5
5.64e-4
6.02e-4
3.58e-4
4.05e-4
MATH
Complex.
Info.
0.03
0.1
1.72e-1
1.96e-1
2.47e-1
2.01c-1
3,39c-2
1.41e-1
2.09c-1
1,58e-2
1.11e-1
1.8le11
2.30c-1
1.96c-1
2.11c-1
1.60c-1
1.34e-1
1.74e-1
0.3
2.64e-1
4.11e-1
4.13e-2
2.86e-1
1.10e-1
1.34e-1
0.5
2.71e-1
2.93e-1
4.52e-2
3.22c-1
Dist.
0.03
3.93e+2
2.53e+2
4.22e+2
5.28c42
6.78c+2
5.03e+2
3.96c+2
3.62e+2
Kurt.
0.1
3,57c+2
2.23e+2
4.51e42
3.07e42
1.27c43
9.20e+2
5,44c+2
4.83e+2
0.3
3.11e+2
1.89++2
3,35c+2
2.65e+2
8,49c+2
7.6le+2
+
0.5
3.03e+2
1.89++2
2.85e+2
2.20e+2
Act.
0.03
2.36e-3
2.22e-3
1.77e-7
1.17e-6
7.08e-4
7.51c-4
1.20c-3
1.19e-3
0.1
1.24e-3
1.21e-3
8.23e-4
9.06e-4
5.15e-4
5.76e-4
8.11e-4
8.10e-4
Intens.
0.3
7.61e-4
7.57e-4
4.92c-4
5.32e-4
4.76e-4
4.69c-4
0.5
5.87c-4
5.99e-4
3.87c-4
4.47e-4
3.71e-4
3.53e-4
College Math
Info.
0.03
1.45e-1
1.96e-1
2,51e-2
1.14e-2
2.13e-1
2.35e-1
8.01e-2
2.17e-1
Complex.
0.1
2.08e-1
2.09e-1
1.65e-1
1.6le-1
1.32e-1
1.64e-1
1.34e-1
1.25e-1
0.3
2. 2.20e-1
2.89c-1
3.29c-1
2.88e-1
1.23e-1
9,95e-2
0.5
2.53e-1
2.83e-1
2.68e-
3.43e-1
1.1le-1
1.05e-1
Dist.
0.03
4.71c+2
2.75e+2
4.81e+2
8,60c+2
5.86c+2
5.08e+2
4.57c+2
3.89++2
0.1
3.48e+2
2.88e+2
3.80e+2
2.64c42
1.15e+3
8.88c+2
5.31e+2
4.60c+2
Kurt.
0.3
3.31e+2
2.19e+2
2.77c+2
2.08e+2
7.51c+2
6.51e+2
+
0.5
3.31c+2
2.12e+2
2.54c+2
2.22e+2
9.15c+2
7.48e+2
Act.
0.03
3.08e-3
2.76e-3
4.83e-7
1.17c-6
1.06e-3
1.15c-3
2.13e-3
2.19c-3
0.1
1.43e-3
1.50e-3
5.90e-4
6.59e-4
6.71e-4
7.72e-4
1.13e-3
1.13e-3
Intens.
0.3
7.80e-4
8.52c-4
3.86e-4
4.44e-4
6.46e-4
6.49c-4
0.5
5.76e-4
6.52e-4
3.01e-4
3.60e-4
4.94c-4
4.90e-4
GSM8K
Info.
0.03
1.56c-1
1.56e-1
6.30e-2
4.00e-2
2.22c-1
3.33e-1
2.19c-1
2.53e-1
Complex.
0.1
1,50c-1
2.30e-1
8.43e-2
1.49e-1
1.60c-1
2.64e-1
1.64c-1
1.80e-1
0.3
1.71e-1
2.27e-1
1.48e-1
2.09e-1
1.09e-1
1.57e-1
0.5
1.37e-1
3.23e-1
1.69e-1
2.66e-1
1.14e-1
1.28e-1
Dist.
0.03
4.73e42
3.05e42
2.05e+2
2.18e42
3.81e+2
3.44c+2
4.68e+2
3.95c+2
Kurt.
0.1
4.57e+2
2.79e42
4.21e+2
3.07e42
7.66ic42
5.60c+2
5.22c+2
4.53e+2
0.3
3.85e+2
2.48e+2
3.99e+2
2.48e+2
7.17e+2
5,88e+2
0.5
4.02e+2
2.49c+2
3.16e+2
2.18c+2
7.8le+2
6.73c+2
MATH- SHEHH-D process reward model on GSMSK and MATH, leading to notable gains in mathematical reasoning accuracy.
Ritieek-----iittll---- Qwen2.5- model distilled from DeepSeek-Re RI, trained via supervised distillation inherit strong reasoning ability.
Distilled-Qwen (Guo et al. 2025 Chen et al. 2025):
is
Qwen2.5 (Qwen et al.) (2025 Zhang et al. 2025a): Qwen2.5-7B-SF7i is fine-tuned with supervised learning on the MATH and Numina-Math datasets, while Qwen2.5-7B-DPO is derived from that SFT model via iterative Direct Preference Optimization (DPO).
We conducted extensive analyses on three public mathematical benchmarks: GSM8K. MATH, and College Math. More detailed characteristics of the analyzed LLMs and implementation details are provided in the Appendix A and Thorough extensive evaluations on multiple benchmarks shown in Appendix D the post-training generally improves the capability of different LLMs.
4.2 METRICS
In our experiments, we quantify differences in LLM behavior before and after reinforcement learn- ing (RL) fine-tuning by analyzing the internal edge-weight matrices obtained from the graph-based attribution procedure. Let ((k) E R denote the edge-weight matrix for sample kt with

---

Published as a conference paper at ICLR 2026
k = 1........ n. Here, no denotes the number of output nodes in the graph structure, and n; de- notes the number of input nodes in the graph structure. The collection of all samples forms a tensor W E X=xx, Based on this imput, we define three complementary metrics:
Activation Intensity (Act.Intens.). This metric quantifies the average magnitude of all edge weights across every sample, output, and input, capturing both how many pathways in the LLM are activated and the strength of their activation:
R0
w
Ant.In
(13)
TU To
Information Complexity (Info. Complex)))... To capture the heterogeneity and unpredictability of edge activations across the entire dataset, we compute Shannon entropy over the absolute values of all edges from all samples, flattened into a single vector. Let pa denote the normalized probability of bin b in a histogram of all [ values, with B bins and small constant to prevent log 0:
Comole. Σ Pb log(ps + e).
(14)
Higher entropy values indicate more complex and less predictable distributions of edge activations, whereas lower values suggest concentrated or more regular patterns. This metric reflects the di- versity of active information pathways within the LLM during inference and highlights how RL fine-tuning may alter the overall internal information structure.
Distribution Kurtosis (Dist.Kurt.). To quantify the overall shape and stability of edge-weight distributions, first compute the kurtosis of each sample's edge-weight matrix and then average across all samples:
Co.s (Wilh)
- -n((t)))
n 1
Dist. Kurt.
(15)
W:(k) - -1((k)) )²)
2 2
Eas i
where (A) is the mean edge weight of sample k. Values approaching zero indicate that individ- ual edge-weight distributions approximate normal distribution. Conversely, significant positive or negative values reflect leptokurtic (heavy-tailed) or platykurtic (light-tailed) distributions, respec- tively. This metric serves to evaluate the impact RL fine-tuning on the tail behavior and outlier characteristics of the overall activation distribution.
4.3 RESULTS AND ANALYSIS
Our main experimental results are presented in Table I We observe that the three model families, Deepseek-Maie Mistral, and Distilled- Qwen, exhibit largely consistent changes in the metrics be- fore and after RL fine-tuning, Specifically, Activation Intensity and Information Complexity tend to increase, while Distribution Kurtosis tends to decrease. Individual exceptions can be seen some cases for Deepseek- and Mistral. However, as the scaling factor a controlling truncation length gradually increases, these exceptions diminish, and the observed patterns become largely consistent, indicating that the phenomenon is relatively robust. However, beyond these observations, also find several differences in the experimental results of the Qwen2.5 series models trained with the DPO method. For instance, their activation strengths do not exhibit clear increasing trend, and the College Math dataset, as a grows to larger values, the Information Complexity metric of their internal pathways remains lower than that of the initial SFT model from which training began.
Taken together, the above observations suggest two key conclusions: (i) Online RL fine-tuning for mathematical reasoning increases the extent and intensity of active information edges in the model. (ii) Online RL fine-tuning diversifies the activation patterns across these information pathways. We next provide further analyses to substantiate these conclusions.
7

---

Published as a conference paper at ICLR 2026
Pathway Engagement Induced by RL Fine-tuning. As shown in our main results (Table D. RL fine- tuning consistently increases Act.Intens. and decreases Kurt, meaning that a substantial number of low- activation edges become more active, effectively engag- ing a larger set of pathways. This trend is observed across different models, datasets, and hyperparameter settings. Figure 2 illustrates this effect with a representative case: the Mistral model on the MATH dataset at a 0.5. The relative change analysis highlights that many connections strengthen after PPO-based RL fine-tuning, confirming that reinforcement learning systematically enhances the propagation of internal signals.
0.6
0.4
10 Outsr Noweaa
30
ungid Seures
5o
50
100
62
122
Figure 2: Relative change in edge acti- vation strength after RL fine-tuning for the Mistral model on the MATH dataset with 0.5.
Diversity of Activation Patterns in Internal Represen- tations. In parallel, we find that Info. info.Complex. gen- erally increases and Dist. Kurt. decreases after RI. fine- tuning as shown in Table indicating that activation pat- terns become more diverse. We provide further visual- ization results relevant to this conclusion, as illustrated in Figure [] panel (a) shows that across inference samples, the internal activation structures exhibit greater variability after RL, as quantified by an increase in one minus the mean correlation of edge- weight matrices between sample pairs, and panel (b) further demonstrates that output-edge entropy combinations. Together, these results indicate that RL enriches the connectivity structure the internal circuitry, leading more robust and flexible information flow essential for logical deduction. Furthermore, shown in Figure 3 panel (a), the Qwen2.5 series models trained with the DPO algorithm also exhibit certain degree of improve- ment in diversity. However, the magnitude of this improvement is relatively lower than that of models trained with other online RL methods.
rises across most mooel--dataset-hyyeppaamaa
0.8
633%)
Base Model
0.9
RL Improvement Region
ang
Oudgutt Entroon Divirstty Scone
RL Model lnter Sample Diverstt
0.6
Toral Experimenae
0.7
Instovament tata: 25.2%
8.6
0.4
0.5
Medels
Despienk-Math
0.2
Distillec-Qeen
Qwen2.5
Equal
0.0
Model
Duepseck
Datilled
(a)
(b)
Figure 3: Comparison before and after RL fine-tuning: (a) diversity of activation patterns across inference samples, including data from all datasets and values; (b) entropy of output edge patterns per node. In (b), data points are arranged sequentially by dataset (College Math, GSM8K, MATH), iterating over {0.03, 0.1, 0.3, 0.5} } for each.
Based on Equation differences the support of the sampling distribution D and the properties of the gradient coefficient GCA across SFT, Online RL, and DPO.
interpret the observed phenomena by analyzing the fundamental
For SFT, the data source static, drawn from a fixed human auman-annotated distribution DEFT = (( Paaa), with constant gradient coefficient GOsFT 1. Consequently, the model op- timizes its internal representations to minimize cross-entropy on a narrow, predefined manifold of "correct solutions." This drives the model to converge towards low-entropy mode that mimics the
8

---

Published as a conference paper at ICLR 2026
training data, resulting in activations concentrated on a small number of outlier edges (high Distri- bution Kurtosis) and limited engagement of redundant pathways (low Activation Intensity).
In contrast, Online RL algorithms like PPO and GRPO fundamentally alter the data source by in- troducing on-policy sampling, where outputs are dynamically generated by the evolving policy it- Pastaa 0; TTo-119)_ This mechanism significantly expands the stochastic support set of the training distribution beyond the SFT subspace, providing the LLM with richer set of reasoning path samples for each query q. Mechanistically, to handle the expanded state space encountered during exploration, the network is compelled to activate and reinforce la- tent or "dormant" internal circuits that were underutilized during SFT. Furthermore, the gradient coefficient in online RL varies dynamically based on feedback from the reward model or rule, Tak-
self: Dnz {(g {o:}{=)
ing GRPO as an example, GCGRPO = (g, o, Tra) + 3 [KeerloLa (o1.s)) - -1). To maximize sy(oraloi,<s)
expected reward, the model is driven to mobilize these less active internal eircuits master rela- tively "harder" problems, as correct responses to such instances typically yield significantly higher gradient coefficients. The observed increase in Activation Intensity and the simultaneous decrease in Distribution Kurtosis reflect this broader utilization of residual pathways. Moreover, as multiple distinct reasoning paths for the same question are reinforced, the entropy of the internal edge weight distribution increases.
Furthermore, from this unified perspective, we can elucidate why DPO exhibits distinct behaviors, particularly its failure to consistently enhance activation intensity and information complexity. Al- though DPO is mathematically derived from the RL objective, it operates as an offline (or semi- offline, where datasets are refreshed only periodically) algorithm. Its data source remains closer to relatively more static distribution: DDPO {(q "0,,,==0=)) ) ~ Pa rather than the real-time policy Ty. Since DPO restricts optimization to the fixed support set of an offline dataset and effectively retains only two potentially stale contrasting samples for each query 9, the mechanistic pressure to expand the network's functional capacity through stochastic sampling is significantly weaker. This explains why Activation Intensity and Information Complexity do not show a consistent upward trend compared the SFT baseline. However, DPO does successfully reduce Distribution Kur- tosis. This is because the preference optimization objective is driven by the gradient coefficient Julo" (q) GC = a (olo Frerlo [ 3 log
Ta(o (q) This soft margin mechanism relaxes the strict Hoeflot(q)
matthing constraints of SFT, favoring broader reward maximization landscape and thereby inhibiting the emergence of high-intensity activation edges to some extent, which can be intuitively understood as mitigating rote memorization. Thus, while DPO attenuates the model's reliance few high-intensity edges during inference (low kurtosis), it lacks the on-policy exploration dynam- ies inherent to Online RL, which are essential for driving the systematic enhancement of average internal activation intensity and diversity.
In summary, we posit that the sampling process is the core factor driving the fundamental inter- nal differences between SFT, DPO, and various online-Ro RL paradigms, which consequently leads to disparities in external performance. In Appendix B we manipulated the sampling dynamics dur- ing online-RL. by adjusting the training temperature and observed phenomena consistent with our expectations. These findings provide robust empirical support for our hypothesis.
5 RELATED WORKS
5.1 INTERPRETABILITY REINFORCEMENT LEARNING
The inherent opacity of deep reinforcement learning motivates studies on improving their explain- ability (Qing et al.) 2022). Research in explainable RL can be generally categorized into pre-hoc and post-hoc techniques, where the former seeks to build inherently interpretable agents while the latter focuses on analyzing trained agents. Pre-hoc research direction focuses on creating inher- ently interpretable agents, such as neuro-symbolic systems that represent policies as mathematical expressions (Landajuela et al. 202 Delfosse et al. 2023), ensuring transparency by design. On contrast, among post-hoc approaches, feature attribution methods are widely applied to generate saliency maps to highlight influential input features (Hao et al.) 2022). Besides, another prominent post-hoc paradigm is policy distillation, where the behavior of a complex neural network is distilled into a simpler surrogate model, such as decision tree, to provide global summary of the agent's

---

Published as a conference paper at ICLR 2026
strategy (Li et al. 2021).
Furthermore, counterfactual methods provide alternative explanatory lens by answering "what if" questions, identifying the minimal state alterations that would have led to different action (Puri et al. 2019 Huber et al. 2023).
Collectively, these diverse approaches reflect a field moving from reactive explanation of opaque models towards transparent and trustworthy intelligent agents. However, these research mainly focus on lightweight RL agents for conventional decision-making tasks, while it remains unexplored how RL works in the emerging post-training applications, where LLMs are trained the agent.
5.2 INTERPRETABILITY OF LARGE LANGUAGE MODELS
Research into the interpretabilltyy of LLMs has largely progressed along two complementary paradigms: mechanistic interpretability and representation interpretability (Singh et al., (0244) Mechanistic interpretability aims to reverse-engineer the patterns learned by model by analyz- ing its fundamental components, such as neurons and attention heads, which often employs causal tracing techniques (Gantla 2025). For instance, one study traced numerical hallucinations to a "Ben- ford's Curse", identifying statistical bias learned from training data that was internalized by a small subset of feed-forww forward network (FFN) neurons, and then causally verified this by demonstrating that pruning these specific neurons corrected numerical errors (Shao et al., 2025p. In contrast, repre- sentation interpretability mainly investigates what information is encoded in the model's internal activation states via external probing models, prominent line of work in this area uses lightweight probes varying from linear models (Kim et al. 2025) to graph models (Zheng et al. 2025), ) decoding concepts within the activation space of the model's 's middle layers. These discovered representations not merely correlational, but the learned probe weights can be repurposed as "steering vectors" to causally intervene on the activations during generation, thereby controlling the model's 's output (Kim et al. 2025). While the former paradigm focuses on how model computes, the latter reveals what knowledge it represents, together offering a more holistic understanding of these complex systems.
While such studies offer valuable perspectives LLM interpretabilityy they predominantly focus on analyzing given LLMs without integrating the training methodology with which the LLMs are obtained into the investigation. In particular, remains unclear how RL, the widely adopted tech- nique in post-training, is able to broadly enhance the capabilities of diverse LLMs with distinct architectural and functional characteristics.
6 CONCLUSIONS
We presented a systematic analysis of how reinforcement learning (RL) fine-tuning reshapes the internal circuitry of large language models (LLMs). Using edge attribution patching, we identified two robust effects across multiple model families: stronger average activation intensity and greater diversity in activation patterns. These findings suggest that online RL enhances both the redun- dancy and flexibility information flow, which may underlie its superior generalization ability in mathematical domains. In contrast, DPO fine-tuning produced weaker or inconsistent changes, em- phasizing the methodological gap between static preference optimization and dynamic online RL. Our results provide unified mechanistic perspective on RL post-training and offer guidance for the design of future post-training algorithms.
LIMITATIONS
Our study acknowledges certain limitations that outline important directions for future research. While the consistency our results suggests potential broader applicability, our empirical valida- tion is currently confined to mathematical reasoning tasks. Verifying whether these internal cir- cuit dynamics hold in domains with open-ended outputs, such as code generation, creative writing, open-ended dialogue and
remains critical subject for future investigation. We also note the limitation regarding model scale, as the significant memory overhead required for granular internal state analysis prevented us from extending our experiments to models larger than 7B parameters.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We fully use open-source models and datasets in the paper, which involve no problem regarding privacy and copyright. We cite the resources in Section 4.1 This work does not involve human subjects, discrimination, bias, or fairness concern,,
REPRODUCIBILITY STATEMENT
For Reproducibility, we describe the general experimental settings in Section 4.1 list the imple- and provide our anonymously open-sourced code at https: 7/github. com/ t iinhhinga--a fib Lab 11m rl probing analysis
mentation details Appendix C
ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China under 2024YFC3307605, the National Natural Science Foundation of China under 62472241, and National Research Center for Information Science and Technology.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Devendra Singh Chaplot. Albert jiang, alexandre sablayrolles, arthur mensch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven le scao, thibaut lavril, thomas wang, timothée lacroix, william sayed. arXiv preprint arXiv:2310.06822, 2023.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Acereason-nemmotron:: Advancing math and code reasoning through rein- forcement learning. arXiv preprint 2i0.ii::505.1640000 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.171611 2025.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 36:16618-16652 2023.
Quentin Delfosse, Hikaru Shindo, Devendra Dhami, and Kristian Kersting. Interpretable and ex- plainable logical policies via neurally guided symbolic abstraction. Advances in Neural Informa- fion Processing Systems, 36:50888-508555 2023.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint arxiv:2300.05492 2023.
Sandeep Reddy Gantla. Exploring mechanistic interpretability in large language models: Chal- lenges, approaches, and insights. In 2025 International Conference Data Science, Agents & Artificial Intelligence (ICDSAAI), pp. 1-8. IEEE, 2025.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong. Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming- the rise of code intelligence. arXiv preprint 240. 2024.

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403.17806 2024.
Qianyue Hao, Wenzhen Huang, Fengli Xu, Kun Tang, and Yong Li. Reinforcement learning en- hances the experts: Large-scale covid-19 vaccine allocation with multi- factor contact network. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp- 4684 4694, 2022.
Qianyue Hao, Lin Chen, Xiaoqian Qi, Yuan Yuan, Zefang Zong, Hongyi Chen, Keyu Zhao, Shengyuan Wang, Yunke Zhang, Jian Yuan, and Yong Li. Reinforcement learning in the era of large language models: Challenges and opportunities. researchgate, 2025.
Tobias Huber, Maximilian Demmler, Silvan Mertes, Matthew L Olson, and Elisabeth André. Ganterfactual-rl: Understanding reinforcement learning agents' strategies through visual coun- terfactual explanations. arXiv preprint arXiv:2302. /2689 2023.
Junsol Kim, James Evans, and Aaron Schein. Linear representations of political perspective emerge in large language models. arXiv preprint arXiv:2503.020800 2025.
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Lim post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321, 2025.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference on Machine Learning, pp. 5979-5989. PMLR, 2021.
Zhao-Hua Li, Yang Yu, Yingfeng Chen, Ke Chen, Zhipeng Hu, and Changjie Fan. deural-to-tree policy distillation with policy improvement criterion. arXiv preprint arXiv:2 06898, 2021.
Jianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, and Runzhi Shi. Rlmr: Reinforcement learning with mixed rewards for creative writing. arXiv preprint a50iv:2508. 18642, 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 19X4v 2024.
Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, Yi Wu, Chao Yu, and Yu Wang. What can rl bring to vla generalization? an empirical study. arXiv preprint arXiv:2505. 19789, 2025.
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao. Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.0678/ 2025.
Neel Nanda. Attribution patching: Activation patching industrial scale. URL: https:///ww. neel- 2023.
nanda. ahhhncccaniiccnnlepeeeebllllaaaa
Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishna- murthy, and Sameer Singh. Explain your move: Understanding agent actions using specific and relevant feature attribution. arXiv preprint 1912 2019.
12

---

Published as a conference paper at ICLR 2026
Yunpeng Qing, Shunyu Liu, Jie Song, Huiqiong Wang, and Mingli Song. A survey explain- able reinforcement learning: Concepts, algorithms, challenges. arXiv preprint arXiv:2211.06665, 2022.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL http:: wrxii http://///ww...////eeeeeo www./sss/1.. 2412. 15115
Yi Ren and Danica Sutherland. Learning dynamics of Ilm finetuning. arXiv:2407.10490, 2024.
arXiv preprint
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:1707.06347 2017.
Jiandong Shao, Yao Lu, and Jianfei Yang. Benford's eurse: Tracing digit bias to numerical halluci- nation in llms. arXiv preprint arXiv:2506.01734 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.01761, 2024.
Aaquib Syed, Can Rager, and Arthur Conmy Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310. 10348, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Peiyi Wang. Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math sheh-shepperd: Verify and reinforce Ilms step-by-step without human annotations. arXiv preprint arXiv:23/2.088 08935, 2023.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprini arXiv:2501.0966 2025.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? a comprehensive study. arXiv preprint a40iv:2404. 10719, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint MXXi:25503.14700 2025.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arXiv preprint arXiv:2412.00 01..0/9981. 2024.
Hanning Zhang, Jiarui Yao, Cheniu Ye, Wei Xiong, and Tong Zhang. Online-dpo-rl: Unlocking effective reasoning without the ppo overhead, 2025. Notion Blog, 2025a.
Kaiyan Zhang, Yuxin Zuo Bingxiang He, Youbang Sun, Runze Liu, Che Jiang, Yuchen Fan, Kai Tian, Guoli Jia, Pengfei Li, et al. A survey of reinforcement learning for large reasoning models. arXiv preprint arXiv: 2509:08222..78 2025b.
Yu Zheng, Yuan Yuan, Yong Li, and Paolo Santi. Probing neural topology of large language models. arXiv preprint 2rX1:22506.01042, 2025.
13

---

Published as a conference paper at ICLR 2026
A CHARACTERISTICS OF ANALYZED LLMS
We employed four pairs of large language models (LLMs), each consisting base model (SFT)
and its post-trained RL counterpart. The models and their download links are listed below:
DeepSeek-Math
deepseck-math-7-innttue /
maseek-mathy mockmmmathhh 7b- instruct
http://///////gooooo//////////// ek-math-7b-rl
deepseck-math-77btt
Mistral
mistral-7b-sft:
Ft
d--Sbepherddmmistzaa-------
Distilled-Qwen
//DeepSeek-Rl--iitt..
A
D-Nemotronn-B
Qwen2.5
-Qwen.5-7B-SFFT:
Qwen2.5-7B-DPO:
PQ
As summarized Table
these LLMs are designed with distinctive structural and functional char-
acteristics.
Table 2: Structural and functional characteristies of the analyzed LLMs.
LLM series
Parameter size
# layers
# heads
Max ctx
Dim
Vocab size
DeepSeek-Math
7B
30
32
4096
4096
102400
Mistral
7B
32
32
4096
4096
32000
Distilled-Qwer Qwen
7B
28
28
131072
3584
152064
Qwen-2.5
7B
28
28
8192
3584
151665
14

---

Published as a conference paper at ICLR 2026
B TRAINING DYNAMICS UNDER SAMPLING INTERVENTIONS
Tempu1t Ron
Group Group Reward
Reward
1.5
m
02
140
Training Steps
Training Staps
(b)
Figure 4: Smoothed moving average curves of reward and group reward standard deviation during RL training under different temperatures, with a sliding window length of 8.
Validation Accuracy
Act.Intens.
(a)
(b)
Tempperttioe -4- Tamp-0.
0,0014
Tamp-10
& Act.Ittnns (RL BBso)
0,0012
0,0010
Acouracy
7.6p54
0.0006
0.0004
0,0032
0,0000
20
40
100
120
100
40
60
80
120
160
160
Training Steps
Training Steps
Sist.Stab.
Info.Complex.
(c)
(d)
0.20
-25
Info Complex. (RL Base)
Dist.Sab (RL Base)
-50
0.10
-100
.14..0
0.00
-0.10
-176
.0.15
60
Training Steps
Training Steps
Figure 5: (a) Accuracy of models trained with different numbers of RL steps on the GSMSK test set. (b), (c), and (d) show, respectively, the differences in Activation Intensity, Distribution Stability, and Information Complexity between models trained with different numbers of RI. steps and the initial model. We highlight the training intervals where the performance gains most pronounced for temperature = 0.6 and temperature = 1.0, corresponding to [60, 120] and [40, 100], and mark the extrema of each metric within these intervals in the expected direction.
To investigate whether the observed internal circuit changes causal drivers of performance im- provement rather than mere byproducts, we designed an intervention experiment. These changes specifically refer to the increased activation intensity and the enhancement of pattern diversity. We
15

---

Published as a conference paper at ICLR 2026
employed Swven..5-3B-Instiin as the base model and limited its maximum generation length to 200 tokens to constrain its initial reasoning capabilities. On this basis, we conducted reinforcement learning training on the GSM8K training set using improved variant of the GRPO algorithm (Yu et al. 2025). The experiment was configured with a batch size of 32 and sampled 4 candidate re- sponses per query. We utilized the RLVR reward function where correct answers received 3 points and completely incorrect answers received -0.5 points. Partial correctness was scored proportionally based on the deviation between the prediction and the ground truth.
Our core hypothesis asserts that effective on-policy exploration drives the activation and consoli- dation of beneficial internal circuits and subsequently leads to performance gains. To verify this, we required control variable to modulate exploration quality. We selected sampling temperature as the intervention method because excessively high temperatures theoretically introduce excessive randomness and alter sampling effectiveness. We compared two temperature settings. A temperature of 0.6 represents effective exploration under standard settings while 1.0 represents noisy exploration under high-entropy settings.
Figure illustrates the reward trajectories and the standard deviation of the Group Reward during training. As shown in Figure 4b) the Group Reward standard deviation under the 1.0 temperature setting is significantly higher than that of the 0.6 setting. This confirms that high temperatures induce higher output mode variability (Liu et al. 2025 Ren & Sutherland] 2024). Such excessive variance implies that the signals generated during exploration are noisier. Consequently, it becomes difficult for the model to reliably identify the underlying patterns corresponding correct solutions.
We tracked the evolution of three key internal metrics relative to the initial SFT model as detailed in Figure (d)) We observed significant differences during the critical mid-training phase be- tween steps 40 and 120. At temperature 0.6, activation intensity showed a clear increasing trend and peaked around step 120. contrast, the growth of this metrie was significantly suppressed at temperature 1.0 where the peak was markedly lower than that of the low-temperature group. This suggests that noisy sampling hindered the full activation of latent dormant circuits. Similarly, the information complexity for the 0.6 group rose sharply between steps 100 and 120. Conversely, the 1.0 group failed to sustain growth and even exhibited a decline during mid-training. Regarding distribution kurtosis in Figure 5jc), the 0.6 group showed a significant decrease. This implies that the model no longer relies excessively on a few concentrated high-activation pathways for reason- ing. In comparison, the 1.0 group exhibited smaller decrease. Overall, the 0.6 temperature group achieved higher peaks in activation intensity and information complexity alongside lower valleys in distribution kurtosis compared to the 1.0 group during the mid-training phase.
The differences in internal circuits mapped directly to downstream task performance. Figure 5la) illustrates the changes in GSM8K test accuracy. The accuracy of the 0.6 group elimbed rapidly alongside the expected changes in internal metrics and finally reached 51.2%. Conversely, the performance growth of the 1.0 group was slow due to suppressed internal circuit evolution and eventually stagnated around 36.5%. This is far below the control group. The results indicate that performance gains are substantially weakened when the normal evolution of internal circuits is ar- tificially suppressed via tigh-temperaatre sampling. This intervention-suppe suppression effect provides effective empirical support for our causal hypothesis. This provides empirical support for the view that diverse and high-intensity internal pathway activation is a key mechanism bridging effective sampling and performance improvement, rather than being a simple byproduct.
http:://ttggingffacc.oo///mooo// Instrucct
16

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
In this section, we provide all implementation details for reproducibility Table 3 In the experi- ments, we selected the hyperparameter configuration combination of dataset and a, we randomly drew 100 pairs of contrastive samples from the final pool of filtered samples.
0.5, 7 1.5, and ò 0.5. For each
Table 3: Implementation details
Module
Element
Detail
OS CUDA
Ubuntu 22.04.3 LTS
12.2
System
Python
3.11
Pytorch Device
2.7.0+cu26
2*NVIDIA A100 80G
17

---

Published as a conference paper at ICLR 2026
D PERFORMANCE OF LLMS
Here we compare the performance of LLMs before and after post-training on multiple benchmarks. As shown in Table 4 post-training generally improves the capability of different LLMs.
Table 4: Performance comparisons of LLMs before and after post-training. Bold numbers indicate better performance.
Minerva math
Olympiad bench
College math
LLM series
Post-training
MATH
GSM8K
AIME24
AMC23
Before
46.2
82.1
22.1
14.5
30.8
3.3
17.5
DeepSeek-Math
After
52.6
87.9
27.2
18.2
33.5
6.7
25.0
Before
29.1
78.2
12.1
5.5
17.5
0.0
12.5
Mistral
After
32.6
84.2
11.8
9.2
19.9
0.0
12.5
Before
88.4
90.3
43.0
49.8
40.0
46.7
87.5
DS-Distill-Qwen
After
95.4
93.4
55.9
65.9
44.6
70.0
95.0
Before
75.7
92.2
32.7
37.6
41.9
16.7
62.5
Qwen-2.5
After
82.6
92.0
40.1
46.4
42.5
26.7
67.5
18

---

Published as a conference paper at ICLR 2026
E
USE OF LLMS
The authors used LLMs to aid or polish paper writing, but all content has been carefully reviewed by the author. The authors used LLMs for literature retrieval and discovery, but all related works have been carefully reviewed and organized by the author. The research ideation in this work was entirely completed by the author and does not involve the use of LLMs.
19


Reviews:
[{"review_id": "ojmWUHoJcr", "reviewer": "Reviewer_arJk", "summary": "This paper uses edge attribution patching to analyze how RL fine-tuning changes LLM internals across four 7B model pairs on math tasks. They find online RL methods (PPO, GRPO) consistently increase activation intensity and diversity in internal pathways, suggesting more redundant and flexible information flow. DPO-trained models don't show these patterns, revealing key differences between preference-based and online RL approaches.", "questions": "1. From part 2.1, H^(2ℓ) is used to mean two different things?\nAt the beginning of the paragraph: \"Let H^(2ℓ) ∈ R^(B×P×d_model) denote the input hidden state to the ℓ-th layer\"\nIn the next sentence: \"The output of the ℓ-th layer, H^(2ℓ), is computed...\" Does the same notation represent both the input and output of layer ℓ?\n\n2. Have you analyzed the dynamics of these changes during RL training? At what point in training do activation intensity and diversity increase—early, late, or gradually throughout? Does this correlate with capability improvements?\n\n3. All experiments focus on mathematical reasoning. Have you tested whether these patterns hold for other domains where RL is commonly applied (e.g., coding, creative writing, instruction following)? If not, how confident are you that these are general properties of RL fine-tuning rather than math-specific phenomena?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper makes a valuable contribution by systematically bridging mechanistic interpretability with RL post-training across four model families, three benchmarks, and multiple hyperparameters. The core findings are clear and consistent: online RL methods (PPO, GRPO) reliably increase activation intensity and diversity in internal pathways, while DPO behaves fundamentally differently—providing mechanistic support for existing empirical observations about these algorithms. The experimental design is rigorous with sound methodology, the EAP framework is computationally scalable and theoretically grounded, and the paper excels in reproducibility with open-source code and detailed specifications.", "weaknesses": "1. No Causal Link to Performance:\nThe paper establishes correlation without causation. While the authors demonstrate that RL training correlates with both altered activation patterns and improved performance, they fail to prove that these internal changes actually cause the capability improvements. Critical intervention experiments are missing: for instance, artificially inducing these activation patterns in SFT models to test whether performance improves, or constraining RL training to maintain low activation intensity to assess whether performance degrades. Without such causal validation, the observed patterns may be mere byproducts rather than drivers of RL's effectiveness.\n2. Descriptive Rather Than Explanatory:\nThe paper describes what changes—higher activation intensity and greater diversity—but not why or how RL objectives produce these patterns. It essentially concludes \"RL makes models different internally\" without explaining the underlying mechanism. Critically, the work lacks any connection to RL theory: there is no discussion of how exploration, credit assignment, policy gradients, or reward shaping might drive these activation changes. This leaves the findings as empirical observations without mechanistic grounding.\n3. Proposed Enhancement: \nIncorporating RLVR (RL with Verifiable Rewards) would substantially strengthen the paper. RLVR would complete the methodological space—contrasting online RL with verifiable rewards against learned rewards (PPO/GRPO) and offline preferences (DPO)—thereby testing whether the key distinction is truly online versus offline training. For mathematical reasoning tasks, RLVR provides cleaner binary reward signals that could yield more interpretable activation patterns. Importantly, relevant RLVR-trained models already exist (e.g., Qwen2.5-Math) and would be straightforward to analyze within the existing framework.\nCombined with additional mechanistic experiments—tracking activation changes throughout training trajectories, ablating specific RL components, and formally connecting observed patterns to RL theory—this approach would transform the paper from a descriptive observation into an explanatory contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jLGX0nKR8R", "reviewer": "Reviewer_VkAD", "summary": "This paper investigates the internal effects of reinforcement learning (RL) fine-tuning on large language models (LLMs), using the Edge Attribution Patching (EAP) framework to analyze residual pathways before and after RL fine-tuning. The authors report two consistent effects across several model families: (1) increased activation intensity and (2) greater diversity of activation patterns. They argue that these internal changes help explain the superior generalization performance of RL-fine-tuned models, particularly under PPO, GRPO, and DPO.", "questions": "1. Which dataset is used for the visualizations shown in Figure 3?\n2. In Table 1, how was the hyperparameter (e.g., truncation scale $\\alpha$) chosen?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles how RL fine-tuning affects the internal circuitry of LLMs, which is underexplored.\n2. The findings are consistent across multiple model families and datasets, offering an interesting mechanistic view of RL’s role in enhancing information flow diversity.", "weaknesses": "1. While the paper observes consistent tendency over the three metrices (Activation Intensity, Information Complexity, and Distribution Kurtosis) between SFT and SFT with RL, it does not further verify whether these internal changes causally contribute to downstream performance improvements compared to SFT.\n\n2. The experiments and analyses are limited to mathematical reasoning tasks, raising concerns about the generalizability of the findings to other domains (e.g., natural language understanding, dialogue).\n\n3. All experiments are conducted on 7B-parameter models; including smaller (e.g., 1B) and larger (e.g., 14B) models would strengthen the claim of generality across scales.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3z7R59Gz4", "reviewer": "Reviewer_kREr", "summary": "This paper investigates the underlying mechanisms by which RL fine-tuning enhances the capabilities of LLMs. Leveraging a graph-theoretic perspective and edge attribution patching (EAP), the authors analyze the internal information flow of LLMs by comparing models before (after supervised fine-tuning) and after RL-based post-training. The study finds that online RL algorithms (like PPO and GRPO) systematically alter the internal circuitry of LLMs in two key ways: Increased Activation Intensity (More internal pathways become active, and their signal strengths increase, suggesting a more engaged and robust information flow) and Increased Activation Diversity (Activation patterns become more varied and less concentrated, indicating that the model develops more flexible and diverse pathways for problem-solving). Overall, the work provides a unified, mechanistic explanation for the performance gains seen from RL fine-tuning, bridging the gap between external model behavior and internal circuit-level transformations.", "questions": "Q1: For W1, could you share any thoughts on the possible causal factors or how you plan to investigate them?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper’s detailed comparison between online reinforcement learning methods (such as PPO and GRPO) and preference-based techniques like DPO—especially regarding the distinct internal transformations they trigger—offers a new lens through which to understand current LLM fine-tuning paradigms.\n\nS2: The main contributions and results of the study are presented with clarity. The notions of activation intensity and activation diversity are both intuitive and insightful, effectively summarizing the internal changes observed. The discussion linking these changes to improved robustness and adaptability in information processing is straightforward, helping demystify the complex inner workings of LLMs. This lucidity allows the paper’s substantial technical insights to be easily grasped and appreciated by a wide range of AI researchers.", "weaknesses": "W1: Although the paper effectively highlights what changes occur—namely, increased activation intensity and diversity—it does not adequately explain why these specific changes arise from online RL. The discussion mainly points to correlations between RL fine-tuning and the resulting internal shifts. To advance beyond correlation, the work should develop clear, testable hypotheses detailing the causal mechanisms by which RL signals (such as reward functions or policy gradient updates) drive alterations in edge attribution and activation dynamics.\n\nW2:  The study focuses on 7B parameter models and primarily LLaMA-style Transformers. While this is a common scale, the generalizability of the findings to a broader spectrum of LLMs remains unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)