RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
Authors: Kohsei Matsutani
Shota Takashiro
Gouki Minegishi
Takeshi Kojima
Yusuke Iwasawa
Yutaka Matsuo
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
Reasoning
RLVR
Interpretability
Primary area: interpretability and explainable AI

Abstract: Large language models (LLMs) are typically trained by reinforcement learning (RL) with verifiable rewards (RLVR) and supervised fine-tuning (SFT) on reasoning traces to improve their reasoning abilities. However, how these methods shape reasoning capabilities remains largely elusive. Going beyond an accuracy-based investigation of how these two components sculpt the reasoning process, this paper introduces a novel analysis framework that quantifies reasoning paths and captures their qualitative changes under each training process (with models of 1.5B, 7B, and 14B parameters on mathematical and code domains). Specifically, we investigate the reasoning process at two levels of granularity: the trajectory-level, which examines complete reasoning outputs, and the step-level, which analyzes reasoning graphs whose nodes correspond to individual reasoning steps. Notably, clustering of unique reasoning trajectories shows complementary effects: RL compresses incorrect trajectories, whereas SFT expands correct ones. Step-level analysis reveals that RL steepens (about 2.5 times), while SFT flattens (reduced to about one-third), the decay rates of node visitation frequency, degree, and betweenness centrality distributions in the reasoning graph. This indicates that RL concentrates reasoning functionality into a small subset of steps, while SFT homogenizes it across many steps. 
Furthermore, by evaluating the reasoning graph topologies from multiple perspectives, we delineate the shared and distinct characteristics of RL and SFT. Our work presents a novel reasoning path perspective that explains why the current best practice of two-stage training, with SFT followed by RL, is successful, and offers practical implications for data construction and more efficient learning approaches.

Published as a conference paper ICLR 2026
RL SQUEEZES, SFT EXPANDS: A COMPARATIVE STUDY OF REASONING LLMS
Kohsei Matsutani"
Shota Takashiro
Gouki Minegishi Yutaka Matsuo
Takeshi Kojima
Yusuke Iwasawa
The University of Tokyo
"ohssei mrohsei.matsunn matsutani @weblab.b................ t. t.u-totyo.acccoo jp
ABSTRACT
Large language models (LLMs) are typically trained by reinforcement learning (RL) with verifiable rewards (RLVR) and supervised fine-tuning (SFT) on reasoning traces to im- prove their reasoning abilities. However, how these methods shape reasoning capabilities remains largely elusive. Going beyond accuracy-based investigation of how these two components sculpt the reasoming process, this paper introduces a novel analysis frame- work that quantifies reasoning paths and captures their qualitative changes under each training process (with models of 1.5B, 7B, and 14B parameters on mathematical and code domains). Specifically, we investigate the reasoning process the trajectory-level, which examines complete reasoning outputs, and the step-level, which analyzes reasoning graphs whose nodes correspond to individual reasoning steps. Notably, clustering of unique reasoning trajectories shows complementary effects: RL compresses incorrect trajectories, whereas SFT expands correct ones. Step-level analysis reveals that RL steepens (about 2.5 times), while SFT flattens (reduced to about one- one-third), the decay rates of node visitation frequency, degree, and betweenness centrality distributions in the reasoning graph. This indicates that RL concentrates reasoning functionality into small subset of steps, while SFT homogenizes it across many steps. Furthermore, by evaluating the reasoning graph topologies from multiple perspectives, distinct characteristics of RL and SFT. Our work presents a novel reasoning path perspec- tive that explains why the current best practice of two-stage training, with SFT followed by RI, successful, and offers practical implications for data construction and more efficient learning approaches.
levels of granularity:
delineate the shared and
INTRODUCTION
Following the advent of OpenAI-ol (Jacch al. 2024) and the open-sourcing of DeepSeek-RI (Guo et al. 2025), post-training for enhancing reasoning abilities to solve complicated logical tasks, including math- ematical problems, has seen surge of interest. Two primary learning methods are adopted for reasoning post-training: Supervised Fine- Tann-T (SFT where the policy is trained to imitate teacher policies by max- imizing log-likelihood using supervision signals from human annotations or strong teacher models (Ye et al.) Muennighoff et al. 2025 Guha et al., 2025); and Reinforcement Learning (RL which maximizes expected rewards optimize the probability of producing correct solutions in verifiable tasks (Jaech al.)
2025
2024
Guo et al. 2025).
'In this paper, we denote SFT as supervised fine-tuning on reasoning traces generated by reasoning LLMs such as DeepSeek-R1 (Guo et al.] 2025) and Gemini-thinking (Comanici et al. 2025). II this paper, we denote RL practical policy-gradient methods such GRPO Shao al. 2024

---

Published as a conference paper ICLR 2026
It has been suggested that RL with verifiable rewards (RLVR) in LLMs simply incentivizes pre-existing
capabilities of the base model (Base model) (Liu et al. Gandhi et al., 2025) since it performs Chain-of-Thought (Wei et al. 2022)
2025d Zhao et al.
2025a Shah et al., 2025
vast vocabulary spaces
within the constraints of the Base model's prior. Recently. Yue et al. (2025)
investigated the Pass @ k
metric (Chen et al. 2021] Song et al.) 2025b] Dang et al.) 2025 Wen et al. 2025 Wu et al. 2025).
which measures the probability that at least one correct solution is found when sampling t independent solutions from the model (i.e., Best-of-kk). They showed that, as & increases, Base model's Pass @k l eventually surpasses that of the RL model trained with RLVR. This observation suggests that Base mod- els already possess the capability to solve problems that RL models can solve. However, these stud- ies primarily evaluate answer accuracy without investigating the underlying reasoning process. Addi- tionally, current state-of-the-art models for mathematics and coding, such as ProRL (Liu et al. (025b) and AceReason (Chen et al. 2025d] Liu et al., 2025e), apply RL starting from DeepSeek-RI (Guo et al., (025) distillation model checkpoints, essentially conducting two-stage training with SFT fol- lowed by RL (SFT+RL models). DeepSeek-RI (Guo al. 2025) also features cold-start integration.
Yet, various SFT+RL training approaches are currently devel- oped through trial-and-error without grasping the distinct roles of RI. (reinforcement) and SFT (imitation). An important ques- tion to ask is then, "how do RL and SFT shape the reasoning process beyond accuracy measurements?"
Trajectory-level
Base
Proben
Unique Trijectontes
x
this paper, we systematically dive into reasoning process at two granularities (Figure D: (1) trajectory-level, where entire thinking generations are regarded as single trajectory, and (2) step-level, where each node (vertex) in the latent space graph (hereafter referred to as the reasoning graph) represents a logical expression (i.e., sentence), such problem setup, a calcula- tion, or a verification.
RL
SFT
x
SQUEEZE
EXPAND
Step-level
Reasoning Graph
Frequency: Dagroe, Centrality
Step
For trajectory-level analysis, we sample multiple outputs from Base, RL, SFT, and SFT+RL model then identify unique tra- jectories by applying clustering
RL
group similar ones. We find that RL decreases the number of unique incorrect trajectories, whether starting from Base or SFT models, whereas SFT in- creases the number of unique correct trajectories, suggesting that RL compresses incorrect trajectories while SFT expands correct ones. We also note that SFT alone preserves incorrect trajec- tories. These results justify the two-stage approach of creating correct trajectories with SFT followed by suppressing incorrect paths with RL. Additionally, RL consistently reduces correct tra- jectories, which provides explanation for why Base model's Pass@k & converges with that of the RL model at large h:
SQUEEZE
Functional (Mub,
SFT
EXPAND
Base
Figure 1 Overview of our analysis. (Top) RL compresses incorrect trajecto- ries, and SFT expands correct trajecto- ries. (Bottom) RL concentrates function- ality (e.g.. hubs) in a small number of steps, and SFT distributes functionality more uniformly across many steps.
At the step-level, ing model outputs into sentences, generating their embeddings, and clustering these representations to define nodes in sentence space. We observe that rank plots for node visitation frequency. degree, and betweenness centrality in reasoning graphs follow exponential laws. Remarkably, analysis of their decay rates reveals that RL elevates the decay rate, whereas SFT degrades it, suggesting that RL not only compresses the graph but also consolidates functionality (e-g., hubs) into fewer nodes (steps).
construct reasoning graphs by segment-
Throughout, Base model refers to the model immediately after pretraining, RL model to the Base model after RLVR, SFT model to the Base model after SFT (distillation), and +FT model to the SFT model further trained with RL.
2

---

Published as a conference paper ICLR 2026
We further investigated the reasoning graph structure through global and local topological metrics. We identified both shared and distinct patterns in how RL and SFT modify reasoning graph topologies. Both RL and SFT convert local acyclic reasoning graph structures to cyclic ones, resulting similar subgraph proportions. However, RL transforms the community struuuuu--t reasoning graphs of Base models into hub- centralized graphs, while SFT, using traces from a strong teacher, weakens community boundaries to form globally connected graphs.
Our contributions are summarized as follows:
Trajectory-level analysis confirms that RL compresses incorrect trajectories while SFT expands correct ones, highlighting why the two-stage approach (SFT then RL) is effective.
Step-level analysis uncovers that RL also consolidates reasoning graph functionality into fewer steps, whereas SFT expands across diverse steps. Moreover, through topological metrics, we demonstrate that, while both RL and SFT generate local cyclic structures, they produce distinct global topologies.
At both trajectory and step-level analysis, we provided empirical support that RL squeezes and SFT ex- pands the reasoning process. Our findings interpret why existing post-training recipes work and suggest directions for developing new training methods and for data curation.
The code is available at kohseim/rl sesllssssseeesseeeeeeeeee _ateeeess_sf___eeeeeeeeeeee expands
2 RELATED WORK
RL for LLM. Research on RLVR has explored how introduces novel reasoning abilitics to LLMs. Yue al. (2025); Song et al.] (2025b): Dang al. (2025); Wen et al. (2025) argues that RLVR merely elicits existing base capabilities rather than developing new ones, as evidenced by Pass@ metrics. Wu eet al. (2225) demonstrates theoretically that RLVR cannot exceed the support of Base model. Furthermore, advanced reasoning abilities, such as backtracking and verification (Gandhi et al. 2024 2025), are amplified only when Base models already possess them (Liu al. 2025d Zhao et al. 2025a] Shah et al. (2225) While RLVR underperforms with Llama (Grattafioni et al. 2024) compared to Owen (Qwen et al. 2025 Yang
ct al. 2025), mid- hid--rrining on mathematical domains is crucial (Wang et al. 2025d). Additionally, research explores self-improvement (Huang et al. [2033] Pang et al. [224 Huang et al. 2025 using iterative internal rewards (Shao al.] 2025 [025] /hou et al. [ Zhao al. 2025b] Prabhudesai et al. |025]2225] Cheng et al. [025)
Chandak et al. (2225) such as confidence measures rather than verifiable rewards. It is argued that biased policy gradients can substantially sharpen distributions even with random rewards (Oertell et al. 2025).
RL vs SFT. The two dominant paradigms for post-training reasoning LLMs are SFT and RL. Previous works analyzed from the perspective of transfer ability (Han et al.) 2025 Li al. (025 Chu et al. 2025) and demonstrate that SFT tends to memorize, whereas RL generalizes (Chu et al. (025), with RL exhibiting superior retention due negative samples (Lai al., 2025) and its inherent on-policy nature (Shenfeld
let al.] 2025). Chen el al. (2025a) observed SFT pseudo-reasoning interferes with RL training in VLMs, and Setlur et al. 2025a) showed the superiority of verifier-based RL under onti-concentration and heterogeneous conditions. Furthermore, several integrated approaches improve performance by combining SFT and RL
(aa al. 2023 Chen et
2025b
Liu et al. 2025c Yoshihara et 2025 Chen et al. 2025c).
Analysis Reasoning Behaviors. Bogdan et al. (025) analyzed reasoning steps mathematical do- mains, Qin et al.] (2025) examines plan-execute-vert
paradigms, and Liu al. (2025a) leverages unary "try again" feedback. Liang et al.) (2025); Cheng et al. (2025) investigates thinking tokens, with particular attention overthinking phenomena (Sui al. (025) and aha moments (Guo et al.
2025) such "Wait"
tokens (Wang et al. 2025a] Ding et al. 2025). ). Furthermore, studies explore steering vectors (Venhoff et al.) 2025). and examine the exploration in reasoning LLMss (Lu et al. 2025 (Shoaaee et al. 2025). Others focus
3

---

Published as conference paper ICLR 2026
Table Comparison of Model Variants. We evaluate Base, RL, SFT, and SFT RL models across three sizes, 1.5B, 7B, and 14B. See Appendix B.1 for detailed model specifications.
Base Model
RL Model
SFT Model
SFT RL Model
1.5B
Qwen2.5-Math-
Qwen2.5-Man 1.5B-
(DepSeck-RI-Distie Qwen-1.5B
Nemotron-Research
Oat-Zero
Reasoning- -wenn-----------
1.5B
7B
Qwen2.5-Math-7B
Qwen2.5-Math-75
DeepSeek-RI-Diss
AceReason-
Oat-Zero
Qwen-7B
Nemotron-7B
14B
Qwen2.5-14B
Qwen-2.5-14BB
DeepSeck-RI-Distin
AceReason-
SimpleRL-Zooo
Qwen-14B
Nemotron- 14B
on the locality structure in the vocabulary space (Prystawski et al. 2023 Kim et al. 2025).
2025
Minegishi et al.
Our work extends beyond outcome- based Pass@ metrics to examine how RL and SFT fundamentally shape the reasoning processes of LLMs, offering novel perspective on the formation of reasoning behavior.
3
TRAJECTORY-LEVELL ANALYSIS
We now investigate how RL and SFT fundamentally reshape reasoning trajectories (paths) by analyzing their distinct effects unique paths. See Appendix C.I for our problem formulation.
3.1 CHARACTERIZING UNIQUE REASONING TRAJECTORIES
We comprehensively study Base, SFT, RL, and SFT+RL models in Table [ on AIME24, AIME25, and AMC23. (See Figure 12 for accuracy comparisons). We also studied the 7B models in Table [ on Hu- manEval (Chen al. 202I). For each problem and model, we generate M 256 samples using a cemperature of 0.6, top P of 0.95, and response length of 16000. Meticulous attention must be paid implementation details regarding prompt templates and response length. Please refer to Ap- pendix B.2 for details. We report Pass@ k results in Figure 12 These samples comprise both M+ correct trajectories and M_ incorrect trajectories. To estimate the number of unique trajectories, we compute pair- wise similarities between the sampled outputs and apply hierarchical clustering based thresholds. The similarity between two reasoning trajectories n²' and T. is measured using the chrF (Popović 2015):
CHRP. CHRR
chrF'3 = (1+33)
32. C..RP + CHRR'
where CHRR the proportion of character n- n-grams in the hypothesis that also occur in the reference and CHRR is the proportion of character n-grams in the reference that also occur in the hypothesis. Compared to BLEU (Papineni al. 2002), which is based on word-level n-grams, chrF uses character-level n-grams and better captures semantic similarity under morphological variation (e.g., "add" vs. "adding"), See Appendix for representative examples of trajectories.
Given the verifiable reward, we split set of M trajectories into the correct set and the incorrect set. For each subset, we construct a similarity matrix S+ E Te M and S_ E xM-, where each entry $1, (chrF F T²) yielding D+ and D_. Since chrF is not an embedding
then defined as d,, 1- metric in Euclidean space, we employ UPGMA (Unweighted Pair Group Method with Arithmetic Mean) (Sokal et al., 1958) for hierarchical clustering rather than Ward's Method (Ward] 1963) or Centroid Linkage. We use a similarity threshold of 60 to cut the dendrograms and report the resulting number of clusters for the correct and incorrect sets. Figure
+ chrF (TT), T²)) /2. The corresponding distances
4

---

Published as conference paper ICLR 2026
Number of Unique Trajectories in Incorrect Outputs
AIME24
AIME25
AMC23
120
Qwenz. 5- 3-Ma Mathh-1.58 (Base)
100
(4.4.80.2)
SFT
(22.2,92.2)
(4.4,97.3)
SFT
Nemotron- Nemmtronn-Reearchee www.-eeeett---soonnnggeeee www.RRoonnggwnnn2222 -.w.in---- (SFT-RL)
DeepS R2 Divtilll--Qee--119 (SFT)
Owenn: Maw.m225--tth---555 1380tttt-eee (RL)
80
RL
(14.4,67.2)
(13.3.52.)
RL
SFT
40
(3.3,46.3)
(2,3,43:1)
RL
(264
RL
20
5.22.)
(8.0,21.00
(7.0.16.5)
(16.8.7.6)
30
35
25
Number of Unique Trajectorics in Correct Outputs
Figure 2: Effect of RL and SFT the Number of Unique Trajectories. The x-axis represents the number of correct clusters and the y-axis represents the number of incorrect clusters for trajectories before and after training of 1.5B models in Table T Plot shows the average across samples. See Appendix C.3 for complete results and for additional results Appendix C.4
plots the number of correct clusters on the horizontal axis and incorrect clusters on the vertical axis. The overall similarity distribution is shown in Appendix and results obtained with BLEU and under different
thresholds are provided in Appendix C.5
3.2 RL SQUEEZES AND SET EXPANDS UNIQUE REASONING TRAJECTORIES
HumanEval
As shown in Figure 2 applying RL from either Base model or SFT model dramatically reduces the number of incorrect trajectories. This indicates that RL enhances Pass@1 1 through probability mass redis- tribution. This aligns with theoretical predictions of empirical sup- port shrinkage (Wu al. 2025) and diversity collapse (Dang et al., 2025). Crucially, we also observe a reduced number of correct trajec- tories, explaining why Base models outperform RL models in Pass @ k at large k (Yue et al. 2025). In contrast, applying SFT to the Base model increases the number of correct trajectories, showing that SFT teaches new solution strategies absent in the Base model. Yet, SFT preserves incorrect trajectories with non-negligible probability mass, potentially improving Pass @k k but not guaranteeing Pass @1 gains. Fi- nally, the two-stage SFT+RL procedure demonstrates complementary mechanisms: SFT expands correct trajectories while subsequent RL compresses incorrect trajectories. This combination, which acquires new solution paths through SFT and removes incorrect paths through RL, maximizes Pass@1 performance. These findings substantiate the Mate-of-thheaat training of SFT followed by RL (Liu al. 2025bb Chen et al. 2025d] Liu et al. 2025e) from reasoning trajectory perspective. We obtained consistent results across other models, including the Llama family (Grattafiori et al., 2024), detailed in Appendix C.4 Consistent
Auahaae dB (PPVEC
SFT
RL
Figure 3: Effect of RL and SFT on the Number of Unique Tra- jectories. The x-axis represents the number of correct clusters and the y-axis represents the number of incorrect clusters for trajecto- ries before and after training of 7B models in Table []
results were also obtained on the code domain using HumanEval as shown in Figure 3 See Appendix C.8
for details.
4 STEP-LEVEL ANALYSIS
In step-level analysis, examine how SFT and RL affect reasoning at more detailed granularity than trajectory-level analysis. We investigate the reasoning graphs from two perspectives: profiling the global structure (Section 4.2) and capturing the local structure (Section 4.33
5

---

Published as a conference paper ICLR 2026
891 Unique Edges 4730 Total Edgee
122 Unique Edges 3605 Total Edgges
4812 Unique Edges/ / 28151 Total Edges
4719 Unique Edges 40367 Total Edges
37.58
Qwen2.5-Math-75 (Base)
Qwen2.5-Math-77 Matt--
SeepSeek-R1-Distieee Qwen-7B (SFT)
BecReason-Nemotron
Zero (RL)
7B (SFT+RL)
Figure 4: Reasoning Graph Example. Reasoning graph for AIME24 Problem #1 built from 256 responses Nodes are arranged sequentially on a circle, consistent across examples.
across the 7B models in Table [ Edge thickness encodes transition frequency, and edge color encodes edge distance. For more examples, please refer to Figure 22 and Figure 23
4.1 CONSTRUCTING REASONING GRAPH
Consider an evaluation dataset D = (xn J with N problems. Given any input x E DD we sample M independent responses. Let each model response क. (for response sample m E [M]) be segmented into sentences: T. = m,. mT, ) where E [Base, SFT, RL, SFT+RL) indexes the model variant and T denotes the number of sentences in response </n- Each sentence r'm,: is mapped into a d-dimensional vector space via sentence embedding function, yielding s'm,t E Rd. For each problem
D, we define the set of sentence embeddings S(x) { ( m E {Base, RL, SFT, SFT+RL), m E [M], t E [Th) }, and collect them across all problems S = { (z, x D. S(x) }. We perform unsupervised clustering of S using K K-mean. This yields a partition of S into K clusters with representative centroids Ck R.d.
C (cI, Ckk]
We denote each cluster by a node uk. so that the node set is V {01,-.... JK}- We define the node set of n! Vn, = { V 3t s'm,t + +v,, J., Each embedding s'm.e assigned to unique node uk. The distance between two nodes v, and by is defined the Euclidean distance between their centroids: d(v;, uj) ||c: Gll2- For each response trajectory nr we derive corresponding sequence of node transitions. Consecutive occurrences of the same cluster assignment are merged into a single node to avoid self-loops. This induces a directed edge set & {v; + Uz) Uū Uj are consecutive and distinct cluster assignments in some - J. Each edge (t; + Ui) is associated with d( f2) and the frequency of this transition. Thus, each model LLM generates a response -h that can be represented as a path in the directed graph 9h = (2% En,,, where nodes correspond clustered semantic units and edge weights reflect their inter-cluster distances.
In our implementation, we employ BGE-large-en-vl.5 (Xiao et al. 2024) as the sentence embedding, where d 1024, set M 256 and K 2000. We conduct experiments for models in Table [ on AIME24, AIME25, and AMC23 and for the 7B models in Table [ on HumanEval. For more details on the implemen- tation, see Appendix [.I] We conduct ablations of the reasoning graph construction for the 7B models in Table varying (i) the number of clusters from our default K 2000 to K 1000, 3000, (ii) the distance metric from Euclidean (I.2) distance to cosine distance, and (iii) the sentence encoder from BGE-large-en- v1.5 to GTE-base-en-vL.. (Zhang et al. 2024) with d 768. Details are provided in Appendix D.4] While our approach builds on Wang et al. (2024); Minegishi et al. (2025), who averaged token representations ex- tracted from each Transformer block within chunks, and performed clustering on hidden states for single model, we instead embed sentences into a shared embedding space and cluster their vector representations jointly across four models. Whereas using each model's internal representations would result in graphs that
6

---

Published as conference paper ICLR 2026
2.300
Vieer Man Lak Hert
AL
0:225
Frequence
1222
Slape
X(R) x AR
2.90
1893
2.22
Base
6FT
SFT
nL
Rank [ineer)
Rank (lmear)
Rank (og)
Rank (lgg)
Linear-Linear Plot.
Log-Linear Plot.
Linear-Log Plot
(d) Log-Loog Plot.
Figure 5: Visitation Frequency Rank Plot. Results from the 1.5B model in Table 1 AIME24 Problem
#1, shown with four combinations of linear/log scales on the x- and y-axes. The x-axis represents the node rank, and the y-axis represents the Visitation Frequency each rank. The rank plot approximately follows an exponential law, showing pear-lineat behavior on a log-linear scale. See Figure 24 for for more examples.
Visitation Frequency
Degree
Betweenness Centrality
Nose
SPT+RL
Base
SPT-8L
Base
SPT
Figure 6: Exponential Decay Rate for Visitation Frequency, Degree, and Betweenness Centrality. Box plots show the estimated exponential decay rate 3 for the (Top), computed across all problems in AIME24, AIME25, and AMC23 for the 1.5B models in Table and for the (Bottom), computed across all problems in HumanEval for the 7B models in Table See Figure 26
2 for complete results.
live in different representation spaces, constructing graphs in this shared sentence embedding space enables direct comparison of the graph properties induced by different models.
4.2 GLOBAL REASONING GRAPH STRUCTURE
The graph visualizations are presented in Figure 4
We observe that RL strengthens some edges while pruning others, whereas SFT creates new connections (see Appendix D.2 for node disparities). For each model and each problem F, we consider weakly connected reasoning graph: g U g = EV)
Estimating Exponential Decay Rate. We investigate how RL and SFT modulate the structure and func- tion (Newman 2003) of complex reasoning graphs by examining the distributional properties node visi- tation Trequency (visitation frequency), node degree (degree), and betweenness centrality (Freeman 1977) within the graphs. For each G¹ visitation frequency is given by M(e where nuu) is the number of node visit and degree is given by H{ D (v > u) gl is defined as -|- Eejeze count) from node to node and (v) is the number of those shortest paths that pass through node
n(v)
U) Betweenness centrality an(l) where ast is the number of shortest paths (determined by edge dat
7

---

Published as a conference paper ICLR 2026
* s,t). Betweenness centrality measures the importance of the reasoning step in mediating the shortest connections in the graph.
For each graph g', we present the rank plots of visitation frequency in Figure 5 We observe that the rank plots of visitation frequency, degree, and betweenness centrality approximately exhibit exponential decay (see Figure 24 for additional rank plots), and hence follow an exponential law. This corresponds to approximately linear decay in log-linear rank plots. As can be seen in Figure 5b] the plots (models) exhibit markedly different deeay rates (i.e., slopes). We therefore investigate the magnitude of this exponential decay. Suppose that the associated value
X(R) EAR
where R denotes the rank of a node and À governs the rate of decay. We estimate exponential decay rate 3 log 10 by linear regression, which is given by log10 X ((R) = a - 3R + ER. where a is an intercept and <n denotes deviations. Figure 5b shows an example (additional examples in Figure 25
RL Squeezes and SFT Expands Graph Functionalities. We estimated the exponential decay rate 3 for the reasoning graphs G' across all problems x in AIME24, AIME25, and AMC23. The results are presented as box plots in Figure (full results in Figure 26
The transition from Base through RL reveals a pronounced structural reorganization, characterized by a marked increase in B. This reflects that high-rank nodes exhibit elevated visitation frequency, degree, and betweenness centrality while low-rank nodes exhibit reduced values of these measures. This suggests that RL consolidates key graph functions (frequency, degree, and centrality) into fewer nodes. In stark contrast, SFT reveals an inverse pattern with reduced B. High-rank nodes display decreased visitation frequency, degree, and betweenness centrality, whereas low-rank nodes show increased levels of these measures. This divergent behavior indicates that RL aggregates functional steps (e.g., hub, central nodes) in the reasoning graph into small number of steps (nodes), whereas SFT, conversely, diversifies them across many steps. We also obtained results on HumanEval that align with these results on mathematical domains, as shown in Figure [ See Figure 37 for details. In our construction of reasoning graphs, nodes are defined by clustering sentences, which tends to produce graphs with high edge density. To address this, we apply graph sparsification by retaining, for each node, the top- 10 or top-20 edges with the smallest Euclidean norm and then estimate the exponential decay rate B, detailed in Appendix D.5 We obtain consistent results where RL decreases and increases SFT increases 3, as shown in Figure
Profiling Global Structure. Next, we profile the global struc- ture of reasoning graphs G through eight topology metrics. We present the edge density, clustering coefficient normalized by the random graph (Watts & Strogatz 1998), assortativity (Newman) 2002), modularity (Girvan & Newman 2002), Freeman centraliza- tion (Freeman] 1978), average path length normalized by the ran- dom graph (Watts & Strogatz| [998), global efficiency (Latora & Marchiori (201). and algebraic connectivity (Fiedler) 1973) of each model's reasoning graph in Figure For detailed descriptions of each metric, see Appendix D.3] As shown in Figure 8 the rea- soning graph of Base model exhibits notably high modularity (Fig- ure 7). low global efficiency, and low algebraic connectivity. This indicates that the nodes are organized into distinct communities (clusters) with weak inter-community connections. Consequently, the reduced robustness and poor reachability efficiency limit the model's ability to fully explore the reasoning graph. However, after RL from Base model, we observe high edge density, low clustering coefficient, low assorta- tivity (Figure 7 and high Freeman centralization. This characterizes graph dominated by small number of high-degree hubs densely connected to peripheral nodes. This can be interpreted as the Base model's 's rea- soning graph being squeezed into structure that enables efficient traversal through a small set of hub nodes.
Low-Assortativity Graph
High-Modularity Graph
Figure 7: Illustration of the Low- Assortativity Graph (Left) and High-Modularity Graph (Right).
8

---

Published as conference paper ICLR 2026
Edge Density
Normalized Clustering Confficient,
Assortativity
Mo dulariity
1121
Freeman Centrallzatton
Normalized Average Path Length
Global Efficlency
Algebratc Connectivvtty
RL
RL SFT
SFT
Figure 8: Comparison of eight graph metries across Base, RL, SFT, and SFT+RL models. Values averaged across different model sizes in Table [ and three datasets, AIME24, AIME25, and AMC23. For details on the eight metrics, see Appendix D.3 See Figure 31 for results by model size.
We also obtained results on HumanEval that align with these results mathematical domains, as shown Figure 38 Finally, SFT and SFT+RL models exhibit low modularity, high global efficiency, and high algebraic connectivity. This reveals reasoning graph characterized by high robustness and superior reach- ability efficiency, without distinct community structures. We observe that global efficiency and algebraic connectivity positively correlated with Pass @ 1/Pass whereas modularity is negatively correlated. This suggests that these metrics relate to the model's ability to effectively explore the solution space and reach the correct answer in single attempt. Details are provided in Appendix [D.]] Furthermore, graph sparsification in Appendix D.5 exhibited the same trend in the changes of the graph metric under RL and SFT, shown in Figure 34
4.3 LOCAL REASONING GRAPH STRUCTURE
Capturing Local Structure with Graphlets. We now turn attention to local structural differences in reasoning graphs we employ graphlet analysis (Milo et al. 2004 Pržulj et al., 2004), which examines small, connected, nonisomorphic induced subgraphs (see Ap- pendix D.3 for more details.).
(a) 4-node Graphlets (G3-G8).
50
We count the 4-node graphlet subgraphs shown in Fig- ure ure 9a in each model's graph. Figure 9h shows that with RI. as well as SFT, we observe a decrease in the pro- portion of acyclic subgraphs, such as G3 and G4, while cyclic structures like G7 and G8 increase. This indi- cates that RL introduces local cyclic structures, reflecting backtracking and verification (Gandhi et al.) (025). into the reasoning graph. Moreover, compared Base model, the RL, SFT, and SFT+RL models all exhibit similar 4- node graphlet proportions. However, as shown Fig- ure 12 there significant performance gaps between the RL model and the SFT/SFT+RL models. This sug- gests that local structure alone cannot fully explain soning performance and global structure, as discussed in Section 4.2 seems to play crucial role.
Proportion (Ng
+ 6.1%
SFT-RL
1.7%
3.99 1.9
1..8%
0
64
G5
G6
GB
(b) Proportion of Graphlets.
Figure Graphlets (G3-G8). (a) 4-node graphlet motifs and (b) their proportions aver- aged across all models in Table and datasets (AIME24, AIME25, AMC23). Arrows indicate the change in proportion after RL.
9

---

Published as a conference paper ICLR 2026
5 DISCUSSION
In this work, we explored how RL and SFT influence mathematical reasoning through a novel reasoning path perspective, examining both trajectory and step-level granularities across multiple model sizes and datasets. From Section 3] the practical success of RL from SFT can be explained by expanding correct trajectories and then compressing incorrect ones, which implies improvements in both Pass@k and Pass@ Of particular note is that SFT preserves incorrect trajectories, indicating that SFT alone does not guarantee Pass@1 performance. This finding precisely accounts for the experimental observation of pseudo reasoning paths induced by SFT in (Chen al., 2025a). Additionally, Section 4.2 reveals that this path-squeezing effect by RL concentrates functionalities (e.g., hubs) into fewer nodes (steps). In contrast, SFT homogenizes these functionalities across diverse steps.
Contrastive Mechanism of RL and SFT. Recently, Wang et al. 2025c b) observed the token level that high entropy thinking (forking) tokens drive reasoning, and RL increases their entropy while decreasing the entropy of non-thinking tokens. Similarly, we observed that RL amplifies the difference between steps with high frequency, degree, and centrality, and other steps. Therefore, applying RL only to functional steps (e.g., hub or central steps) could further improve I.I. reasoning performance and enable more efficient learning. The empirical finding that SFT memorizes and RI. generalizes (Chu et al. (025) may also be related to RL's aggregation and SFT's distribution of reasoning functionalities. In addition, several studies have investigated
RL with exploration bonuses (Cheng et al. 2025 Setlur al.) 2025b] Zheng et al. 2025 Chen et al. 2025e Song et al, 2025a). It would be valuable to analyze whether these approaches merely prevent collapse due excessive squeezing of the reasoning graph, whether they truly expand it in manner analogous SFT.
Reasoning Graph Structure. We observed that RL weakens community structure and promotes efficient transitions in reasoning graphs. This structural shift by RL mirrors the promotion of inter-cluster transi- tions in the community structure (Prystawski by pre-training on a large language corpus. In SFT data curation for reasoning, Muennighoff et al. (2025)
al. 2023 Wang et al.) 2024, Kim et al. 2025 induced
heuristically count "wait" tokens, (Gandhi et al. (0255) prime cognitive behaviors, and Ye et al. (2025) assess step-by-step clarity in the reasoning traces. Our finding that both RL and SFT increase local cyclic structures (Section 4.3) indicates that step-level reasoning behavior is applicable to dataset curation for efficient rea- soning. Leveraging the insight that graph structures facilitating traversal without high modularity are critical for reasoning ability, one promising direction is to incorporate graph metrics (e.g., hub and central nodes) as process rewards in RL.
Our experiments mainly focused on verifiable and competitive mathematical and code domains to evaluate strong reasoning LLMs and Because the DeepSeek- R RD-Distill family is performed SFT with multiple re- sponses per problem (Guo et al. 2025), we additionally perform SFT on s slk-1.1 dataset (Muennighoff et al.) 2025) using a single response per problem. For the 1.5B models in Table we find that our results are hold across both settings, at both the trajectory level (Appendix C.7) and the step level (Appendix (oppendixD.6)...... However, we focused on principled algorithmic differences between RL and SFT without controlling for differences in the training datasets. Investigating their effects on reasoning paths under distribution shift presents intriguing directions, with Han al.] (2025): y: al.) (2025); Chu et al. (2025); (hhnneeld et al. (2025) pursuing similar investigations on generalization and forgetting.
ACKNOWLEDGMENTS
This work supported by the Al Symbiotic Future Society Program at The University of Tokyo.
10

---

Published as conference paper ICLR 2026
REFERENCES
Paul C. Bogdan, Uzay Macar, Neel Nanda, and Arthur Conmy. Thought Anchors: Which LLM reasoning steps matter? arXiv preprint 11XXi,25006.1143,, 2025.
Nikhil Chandak, Shashwat Goel, and Ameya Prabhu. Incorrect baseline evaluations call into question recent 9a8
LLM-RL claims, 2025. URL httt::::: wtt.:elipppp.iiiiioo..o wa..notio... htt.:/ionniittteetooorrceoeo Basel
Eva
intoiiioooni -attnnttttttttttt Quest innettioonneeee con-Recents LLM MLMM-RR CL
4al8deea1
Notion Blog.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. SFT or RL? an early investigation into training rl-like reasoning large Laiguel models. arXiv preprint arXiv:2504. 14468, 2025a.
Jack Chen, Fazhong Liu, Naruto Liu, Yuhan Luo, Erqu Qin, Harry Zheng, Tian Dong, Haojin Zhu, Yan Meng, and Xiao Wang. Step-wise adaptive integration of supervised fine-tuning and reinforcement learn- ing for task-specific LLMs.. arXiv preprint arXiv:2505.. 13026, 2025b.
Liang Chen, Xueting Han, Shen, Jing Bai, and Kam-Fai Wong. Beyond two-Stage training: Cooperative sft and rl for Ilm reasoning. arXiv preprint arXiv:2509.0644 06948,, 2025c.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, and Greg Brockman et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. AceReason-Nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprini arXiv:2505. 2025d.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@k training for adaptively balancing exploration and exploitation large reasoning models. arXiv preprint 20Xv:25008.1751. 2025e.
Daixuan Cheng, Shaohan Huang, Xuekai Zhu, Dai, Wayne Xin Zhao, Zhenliang Zhang, and Furu Wei. Reasoning with exploration: An entropy perspective on reinforcement learning for LLMs. arXiv preprint arXiv:2506. 14758, 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: comparative study of foundation model post-training. In second International Conference on Machine Learning, 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, and Evan Rosen et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint WrXiv:2507.06266 2025.
Xingyu Dang, Christina Back, Kaiyue Wen, Zico Kolter, and Aditi Raghunathan. Weight ensembling im- proves reasoning in language models. arXiv preprint 1xXi:25504.147777 2025.
Bowen Ding, Yuhan Chen, Futing Wang. Lingfeng Ming, and Tao Lin. Do thinking tokens help or trap? towards more efficient large reasoning model. arXiv preprint arXiv:2506.23840, 2025.
Miroslav Fiedler. Algebraic connectivity of graphs. Czechoslovak Mathematical Journal, 23(2):298-305, 1973.
11

---

Published as a conference paper ICLR 2026
Linton C Freeman. A set of measures of centrality based on betweenness. Sociometry, 40(1):35-41 1977.
Linton C. Freeman. Centrality in social networks conceptual clarification. Social Networks, 1(3):215-239
1978. ISSN 0378-8733.
Kanishk Gandhi, Denise H J Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah Goodman. Stream of search (SoS): Learning to search in language. In First Conference on Language Modelingg, 2024.
Kanishk Gandhi, Ayush K Chakravarthy, Anikait Singh, Nathan Lile, and Noah Goodman. Cognitive behav- iors that enable self seff-improving reasoners, or, four habits of highly effective STars. In Second Conference on Language Modeling, 2025.
M. Girvan and M. E. J. Newman. Community structure in social and biological networks. Proceedings of the National Academy of Sciences, 99(12):7821-7826, 2002.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, and Alex Vaughan et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhu- rina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, and Eric Frankel et al. OpenTThought, Data recipes for reasoning models. arXiv preprint arXiv:2506.044 1778 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, and Xiao Bi et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforce- ment learning. arXiv preprint arXiv:2501. 12948, 2025.
Seungwook Han, Jyothish Pari, Samuel J. Gershman, and Pulkit Agrawal. General intelligence requires reward-based pretraining, arXiv preprint arXiv:2502.19402, 2025.
Audrey Huang, Adam Block, Dylan J. Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T. Ash, and Akshay Krishnamurthy, Self-improvement in language models: The sharpening mechanism. In International Conference on Learning Representations, 2025.
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 1051-1068, Singapore, December 2023. Association for Computational Linguistics.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, and Alex Camey et al. Openai ol system card. arXiv preprint arXiv:2412.16770, 2024.
Jeannette Janssen, Matt Hurshman, and Nauzer Kalyaniwalla. Model Selection for Social Networks Using Graphlets. Internet Mathematics, 8(4), dec 1 2012.
Juno Kim, Denny Wu, Jason D. Lee, and Taiji Suzuki. Metastable dynamics of chain-of-thought reasoning: Provable benefits of search, RL and distillation. In Forty-second International Conference on Machine Learning. 2025.
Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Min Xie, Qingfu Zhang, Hongbin Liu, Gaofeng Meng, and Fei Zhu. Reinforcement fine-tuning naturally mitigates forgetting in continual post-training. arXiv preprint 2rX.02250. 2025.
12

---

Published as a conference paper ICLR 2026
Vito Latora and Massimo Marchiori. Efficient behavior of small-world networks. Phys. Rev. Lett., 87: 198701, Oct 2001.
Tianle Li, Jihai Zhang, Yongming Rao, and Yu Cheng- Unveiling the compositional ability gap in vision- language reasoning model. arXiv preprint arXiv:2505. 19406, 2025.
Xiao Liang Zhongzhi Li, Yeyun Gong, Yelong Shen, Ying Nian Wu, Zhijiang Guo, and Weizhu Chen. Beyond pass@1. Self-play with variational problem synthesis sustains rlvr. arXiv preprint arXiv:2508.14022 14029, 2025.
Licheng Liu, Zihan Wang, Linjie Li, Chenwei Xu, Yiping Lu, Han Liu, Avirup Sil, and Manling Li. A simple "try again" can elicit multi-turn IIm reasoning. arXiv preprint arXiv:2507.14295 14295, 2025a.
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. ProRL: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint httv:25555.486644 2025b.
Mingyang Liu, Gabriele Farina, and Asuman Ozdaglar. UFT: Unifying supervised and reinforcement fine- tuning, arXiv preprint arXiv:2505. 00..2255556684,,, 2025c.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang. Chao Du, Wee Sun Lee, and Min Lin. Understanding RI-Zero-Like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025d.
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nemotron 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506. 13284, 2025e.
Jiahao Lu, Ziwei Xu, and Mohan Kankanhalli. Reasoning LLMs are wandering solution explorers. arXiv preprint urXiv:2505.20296 2025.
Lu Ma, Hao Liang, Meiyi Qiang, Lexiang Tang, Xiaochen Ma, Zhen Hao Wong, Junbo Niu, Chengyu Shen, Runming He, Bin Cui, and Wentao Zhang. Learning what reinforcement learning can't: Interleaved online fine-tuning for hardest questions. arXiv preprint arXiv:2506.07527, 2025.
Ron Milo, Shalev Itzkovitz, Nadav Kashtan, Reuven Levitt, Shai Shen-Orr, Inbal Ayzenshtat, Michal Shef- fer, and Uni Alon. Superfamilies of evolved and designed networks. Science, 303(5663): 1542, 2004.
Gouki Minegishi, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo. Topology of reasoning: Understanding large reasoning models through reasoning graph properties. arXiv preprint arXiv:2506.007744 2025.
Niklas Muennighoff, Zitong Yang. Weijia Shi. Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettle- moyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19392 2025.
M. E. J. Newman, Assortative mixing in networks. Phys. Rev. Lett., 89:208701, Oct 2002.
M. E. J. Newman. The structure and function of complex networks. SIAM Review, 45(2):167-256, 2003.
Owen Oertell, Wenhao Zhan, Gokul Swamy, Zhiwei Steven Wu, Kiante Brantley, Jason Lec, and Wen Sun. Heuristics considered harmful: RI with random rewards should not make LLMs reason, 2025. URL
not ion tite/et// Heur st Cons dewoonnidddeeeeerrrrre.ll --With-Random-- Beweandddo-Rewwwada Seepwardd-Shoul---oe I--.th--anndoo--eeaaaahh--hh---- -ake- s 2aaaaassseaaaa0l2222222233333223 6ffce303f019233
13

---

Published as a conference paper ICLR 2026
TingChhng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self seff-improvement by reinforcement learning contemplation. In The Twelfth International Conference on Learning Representations, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charmiak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Maja Popović. chrF: character n-gram F-score for automatic MT evaluation. In Ondřej Bojar, Rajan Chat- terjee, Christian Federmann, Barry Haddow, Chris Hokamp. Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392-395. Association for Computational Linguistics, 2015.
Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Maximiz- ing confidence alone improves reasoning. arXiv preprint arXiv:2505. 22660, 2025.
Ben Prystawski, Michael Y. Li, and Noah Goodman. Why think step by step? reasoning emerges from the locality of experience. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
N. Pržulj. D. G. Comeil, and I. Jurisica. Modeling interactome: scale-free or geometric? Bioinformatics, 20
(18):3508-3515, 07 2004. ISSN 1367-4803.
N. Pržulj. D. G. Corneil, and I. Jurisica. Efficient estimation of graphlet frequeney distributions in pro- tein-protein interaction networks. Bioinformatics, 22(8):974-980, 02 2006. ISSN 1367-4803.
Nataša Pržulj. Biological network comparison using graphlet degree distribution. Bioinformatics, 23(2):
c177-e183, 01 2007. ISSN 1367-4803.
Tian Qin, Core Francisco Park, Mujin Kwun, Aaron Walsman, Eran Malach, Nikhil Anand, Hidenori Tanaka, and David Alvarez-Melis. Decomposing elements of problem solving: What "math" does rl teach? arXiv preprint arXiv:2505. 22756, 2025.
Qwen, :, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2025.
Anida Sarajlić, Noël Malod-Dognin, Ömer Nebil Yaveroglu, and Nataša Pržulj. Graphlet- based characteri- zation of directed networks. Scientific Reports, 6(1):35098, 2016. ISSN 2045-2322.
Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-Time Time compute without verification or RL is suboptimal. In Forty-second International Conference on Machine Learning, 2025a.
Amrith Setlur, Matthew Y. R. Yang, Charlie Victor Snell, Jeremiah Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-Time compute for LLMs. In ICML 2025 Workshop on Long- ConteCt Foundation Models, 2025b.
Darsh Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel. Andrew Ma, Anil Thomas, Anthony Polloreno, Ashish Tanwer, Burhan Drak Sibai, Divya S Mansingka, Divya Shivaprasad, Ishaan Shah, Karl Stratos, Khoi Nguyen, Michael Callahan, Michael Pust, Mrinal Iyer, Philip Monk, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, and Tim Romanski. Rethinking reflection in pre- TeaiTrainng. arXiv preprint arXiv:2504.04022, 2025.
14

---

Published as a conference paper ICLR 2026
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506. 10947, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL's razor: Why online reinforcement learning forgets less. arXiv preprint arXiv:2509.04233 2025.
Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506. 06941, 2025.
R.R. Sokal, C.D. Michener, and University of Kansas. A Statistical Method for Evaluating Systematic Relationships. University of Kansas science bulletin. University of Kansas, 1958.
Yuda Song, Julia Kempe, and Remi Munos. Outcome-based exploration for Ilm reasoning. arXiv preprint 1.Xiv:2509.06944 2025a.
Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, and Udaya Ghai. Mind the gap: Examining the self-improvement capabilities of large language models. In The Thirteenth International Conference Learning Representations, 2025b.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, Hanjie Chen, and Xia Hu. Stop overthinking: A survey efficient reasoning for large language models. arXiv preprint arXiv:2503.166 16419, 2025.
Constantin Venhoff, Iván Arcuschin, Philip Torr, Arthur Conmy, and Neel Nanda. Understanding reasoning in thinking language models via steering vectors. arXiv preprint arXiv:2506.1816 2025.
Chenlong Wang, Yuanning Feng. Dongping Chen, Zhaoyang Chu, Ranjay Krishna, and Tianyi Zhou. Wait, don't 't need to "wait"! removing thinking tokens improves reasoning efficiency. arXiv preprint arXiv:2506.0888 08343, 2025a.
Haozhe Wang. Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierarchical reasoning in llms through reinforcement learning. arXiv preprint arXiv:2509.03644 2025b.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhennu Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforce- ment learning for LLM reasoning. arXiv preprint (51:::2506.01939, 2025c.
Xinyi Wang, Alfonso Amayuelas, Kexun Zhang, Liangming Pan, Wenhu Chen, and William Yang Wang. Understanding reasoning ability of language models from the perspective of reasoning paths aggregation. In International Conference on Machine Learning, 2024.
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. OctoThinker: Mid-training incentivizes reinforce- ment learning scaling. arXiv preprint arXiv:2506.25 12, 2025d.
Jr. Ward, Joe H. Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, (8(301):236-11 2244 1963.
15

---

Published as a conference paper ICLR 2026
Duncan J. Watts and Steven H. Strogatz. Collective dynamics of 'small-world' networks. Nature, 393(6684);
440 442, 1998.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang. Yang Wang. Junjie Li. Ziming Miao, Jiang Bian, and Mao Yang. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base 1lms. arXiv preprint arXiv:2506. 14245. 2025.
Fang Wu, Weihao Xuan, Ximing Lu, Zaid Harchaoui, and Yejin Choi. The invisible leash: Why rlvr may not escape its origin. arXiv preprint arXiv:2507. 14843, 2025.
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian- Yun Nie. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th International ACM SIGIR Confer- ence on Research and Development in Information Retrieval, SIGIR '24, pp. 641-649, New York, NY, USA, 2024. Association for Computing Machinery.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Owen2.5-Math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. /2/22, 2024.
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, and Chenxu Lv et al. Qwen3 technical report. arXiv preprint arXiv:2505.c 09388, 2025.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. LIMO: Less is more for reasoning. In Second Conference on Language Modeling, 2025.
Hiroshi Yoshihara, Taiki Yamaguchi, and Yuichi Inoue. A practical two-Stage recipe for mathematical
LLMs: Maximizing accuracy with SFT and efficiency with reinforcement learning. arXiv preprint arXiv:2507.08267 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In 2nd Al for Math Workshop @ ICML. 2025, 2025.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. SimpleRL Investigating and taming reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 18892, 2025.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai. Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. mGTE: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing: Industry Track, pp. 1393-1412. Association for Computational Linguistics. 2024.
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.07912, 2025a.
Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. Learning to reason without external rewards. arXiv preprint arXiv:2505. 19590, 2025b.
16

---

Published as a conference paper ICLR 2026
Tianyu Zheng, Tianshun Xing. Qingshui Gu, Taoran Liang, Xingwei Qu, Xin Zhou, Yizhi Li, Zhoufutu Wen, Chenghua Lin, Wenhao Huang, Qian Liu, Ge Zhang, and Zejun Ma. First return, entropy-eliciting explore. arXiv preprint arXiv:2507.07017, 2025.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang. Min Lin, and Chao Du Reinforcing general reasoning without verifiers. arXiv preprint 2:X1:2250.. 2025.
17

---

Published as conference paper ICLR 2026
A LLM USAGE
We used LLMs for writing, such grammar correction and rephrasing, coding, and debugging. All gener- ated contents are reviewed and validated by the authors.
SAMPLE GENERATION
B.1 MODELS
Table 2: Comparison of Model Variants. Summary of experimental models used in this study,
Base Model
RL Model
SFT Model
SFT RL Model
Owen2.5-Math- Yang et al.
Owen2.5-Math-L.3.
PeepSeck-RI-iisliee
Cemotron-Researre
2024)
Oat-Zero
Liu et al.
Qwen-1.5B (Guo et al.
Reasoning- Qwen-1.55
2025d)
2025)
LLu al. 2025b
Qwen2.5 1-5-5th--77 7B
Twen2.5-Math-
DeepSeek-RI-Disiilt
AceReason CceReason-Nemotron
Nemotron-
Yang et al. 2024
Oat-Zero (Liu et al. 2025d)
Qwen-7B (Guo et al. 2025)
7B (Chnn et al.
2025d
Qwen2.5 14B
Qwen
Qwen-2.5-14P
DeepSeek DeepSeck-Rl-Distile RI-Distill-
AceReason-Nemotten (Chen et al.
ct al. 2025
SimpleRL-Zoo
ZZng
Qwen- 14B (Guo et al.,
14B
et al. 2025)
2025
2025d)
Owen2.5-Math
AceRReson-Nemotron (Liu et al.
Yang et al. 2024)
1.1-7B
2025e)
Qwen2.5-2 et 2025
Qwwn
Owen2.5-7B-
SimpleRL-Zoo
Zeng
et al. (025)
Llama-3.
Llama-3.1-80 1-BB-
DeepSeek-RI-Dee
(Grattafiori
et al.
SimpleRI
(Zeng
Llama-8B (Guo et al.
2024)
ct al. 2025
2025)
We conducted evaluation using the models specified in Table 2 Qwen2.5-7B- --spp--L--- and Qwen- 2.5- 2.5-14B-Simphe 4BB-SimpeeRL-Zoo are trained from Qwen2.5-7B and Qwen2.5-14B, respectively, using GRPO (Yang and Qwenn255-Math77--Oatt
ett al. 2024) without format rewards. RL-trained from Qwen2.5-Math-11.
are
and Qwen2.5-Math-77 respectively, using Dr. GR.O (Liu et al., 2025d), an improved version of GRPO. Dr.GRPO enhances token efficiency by removing the dividing term and regularization term from GRPO. DeepSeek-RV -Distilll -welll--eee-.... 1.5BB, DeepSeek-RY Ri---istill-Qwa 7B, and ShepSeek-R1--istee -wenlll--ee-11BBB are fine-tuned from 1yen2.5-Math-11.0 and Qwen2.5-14B, respectively, using SFT with DeepSeek-Ro -I's distillation data. Da-See--R--Disill----eaaae Reasoning-Qwen- 1.5B undergoes prolonged RL training for 2500 steps from DeepSeek-R 1.5B. AccReason-Nemottro--2 and AceReason-Nemotron-
1rre2.5--Math-27
are trained from DeepSeek-Ri-n -Diittllll-- Qwen-7B and DeepSeck-RI--issee -weiil----e------e 1ieenn11,,,,B respectively, using large-scale RL on mathematics and cod- ing tasks. AccReason-Nemottro 1.1BBB is trained through large-scale curriculated SFT from Qwen2.5- Math-7B, followed by large-scale RL mathematics and coding tasks. The SFT checkpoint is not publicly available. Llama-3. 1.8.1-88-Sinn -SB-iipleeRL-Zoo is derived from Llama-3.1 1-88B via RL, and DeepSeek-ER - Llama-8B 8B is derived via SFT.
18

---

Published as conference paper ICLR 2026
B.2 INFERENCE
Prompts. In the context of LLM reasoning inference, accuracy demonstrates high sensitivity to prompt template design, necessitating careful attention to template selection and construction. The prompt tem- plates employed in our methodology are showd in Figure 10 We applied the Qwen Template to the following models: Qwen2.5-Math. 1.5B, Dween.5-MMat-l.5B-00 Owen2.5-7B-Simpleke
Qwen2.5-7B. Saven..5-Math-. Qwen2.5-Math- Bathh--B--tteeee Qwen2.5-14B, and Qwen-2.5-14B 1.-11---iim Zoo. Although base models have not been fine-tuned with special tokens and are expected to achieve peak performance without templates (Liu al. 2025d), we employed the Qwen Template ensure explicit gen- eration of stop tokens and maintain experimental consistency with the conditions in Zeng et al.) (2025): ;
Yue al. (225)). The R1 Template was utilized for -eapssee-RR----iil--------- Reasoning- Beasooing-Qwen--ii Qwen-
Wemotron-Researce
AccRReason-Nemotrrnn- 7B, AceReason-Nemotrre
1.1-7B, DeepSeek-R1--ee RI--ittll Qwen 14B, AceReason Neeisson-eemtooo---e 14B, and DeepSeek- R
Please replace and "I" in fig. with with U+2581 1 and U+FFSC, respectively. Following Zeng et al. (2025);
Yue et al.] (2025), we used Llama Template for Llama-3.1-8P and
For
employed the Nemotron-Qwen Template (Liu et al. 2025e). Of particu- lar note, for the models Qwen2.5-Math-1..5 Qwen2.5-Math-1 1.5.hhh-.B--tt-Zeem DeepSSek-RT--istill-Nwee 1.5B, and Nemotron- Cescarch-Reasoning Seassooinn--weeee 1.5B, we identified potential concerns regarding Chinese- English language mixing in the generated outputs. To mitigate this issue, we appended the instruction "Always respond in English only." to the end of each user prompt.
Parameters. For both trajectory-level and step-level experiments, we employed sampling with temperature=0.. and top-p=0. 95. Additionally, we set the response length parameter 16000 tokens. While the source code implementation in Yue et al. (025) utilized vLLM with max mwdeo 1en=4096, this configuration constrains the response length. Consequently, leads to performance degradation for models that generate extended outputs, such as the DeepSeek- RI-Distill and AceReason-Nemoorro families. To address this limitation, we increased the max_model_len parameter to 16000 tokens. However, for models with architectural constraints of specifically Qwen2.5-Math-1..B Qwen2.5-Math-I 1.5thh-11B--000 -Bt-tt- Zero, (-we22.5-Math- and Owen2.5-Math-7BB Baath-B--attZeer Oat-ooro,, maintained max_mode -en=440011 Ac- curacy degradation due to max model len is presented in Figure
Pass@k. We conducted evaluations on AIME24, AIME25, and AMC23 using the same implementation as Yue et al. (2025). For each problem I; contained in the evaluation dataset D = {2; H we sampled n responses and computed the @ k metric for the correct samples G. which is given by: pass@k :=
Ez,-D ("=") [+) The results for n 256 are shown in Figure 12
Response Length. The mean response length for each model across datasets is presented in Figure 13 Following SFT via distillation from DeepSeek-RI. R1, notable increase in response length is observed.
19

---

Published as conference paper ICLR 2026
Qwen Template
<linsstart sssystem
You are a helpful assistant, < -itteanttil____dddl_ im_end>>>>> 
usee (input)
Please reason step by step, and put your final within \boxed ).< im end/>
<lim start Passistant
Nemotron-Qwen Template
sttatt <lim_start
|>system
You are a helpful and harmless assistant. You should think step-by-step.<1ing ind_
|tin_start/uuser (input)
Please place your final answer inside \boxed < <lim_edd>>>>>
<lim_start /start/>assissee >>assistant.
<think>
R1 Template
<lbegin_ss sentencceleeeeeecc_ccceee >>< (user (>(inutt)
Please reason step by step, and put your final answer within \boxed < Alsssssttnt |>sthink>
Llama Template
Question:
(input)
Answer:
Let s think step by step.
Figure 10: Prompt Templates
AIME24
AIME25
Pass @ k
K.
Figure 11: Effect of max max_modelllen Pass @ k. Misspecification of max.model.len causes substan-
tial accuracy degradation the DeepSeck-Rl--e Diiicck-R--Disilll-baee models, which performs lengthy reasoning.
20

---

Published as conference paper ICLR 2026
AIME24
AIME25
AMC23
40
60
1.5B
20
20
Pass
7B
120
14B
42
40
LLAD (010
15
32
122 256
25
125 254
2
36
32
54
125
235
Figure 12: Pass@k performance curves for Base, SFT, RL, and SFT models. Models are the Table D and datasets are AIME24, AIME25, and AMC23.
21

---

Published as conference paper ICLR 2026
AIME24
AIME25
AMC23
20000
20022
Incerect
Incomeet
Incernet
1.5B
20300
10000
10002
tweed 3-Hathh-3
2-Btath 1.5 out pare
DeopSeck aosli qqeas opean13 2-3M
Nemotron-tesenee Beaniting 2.38
Qwen3 5 Matth 1.28
DaupSeck Dioad queen-3-5
Nemweron wnweeeeende dostoolig Spewn 2-35
50 Outi
Qeeend 5 5-Hrtt---31
Ovemz S-eath dat-
Deepteek didti 1.8
Hemetroo qjener3333
Response Length (tokens)
20022
Correct Incer rect
Cornect
Incereet
Incerrect
15983
15300
1509
12983
12500
12500
7B
7383
2500
7500
5053
3300
1000
owend Motth-7
Owen7.55 par
Acaltraam Beve Aoopstoo--7
Owen2 Owen2.5-Matt Matt---
Owen2 Meth 78 Ost
Desp Qwee-79
Arelteaton Nomustoon77
Gwead 5-400-78
Gwent 78 Zare
DeepSasket1 Oween-7
Arafeaavm Hemolronn-
Carrect
Incorrect
Incomect
Incorrect
13000
19002
15506
14B
17500
001
7505
3000
3002
5000
Owen 0omm-25--14 Smplett..Zo
Gerybeak pueen-
Atalfaaton Jewotron
Qwetec25- 2.5- Goppeet---o
Oemn2 5-148
DerpSeak-R2 Glaet Qwen-149 140
Areduzzch Homnttron110 140
Owen 2.6-12.5-116- Simplett. Zed
Oprenc 5-148
Diatti Owen-144
Figure 13: Comparison of Response Length Across Models and Datasets. Models are in the Table and datasets are AIME24, AIME25, and AMC23.
22

---

Published as a conference paper ICLR 2026
C TRAJECTORY LEVEL ANALYSIS
C.I PROBLEM FORMULATIONS
We consider the LLM parameterized by 8, which defines a probability distribution on discrete reasoning trajectories (paths). Let X' denote the input space of natural language problems, and Y' denote the output vocabulary space. Given an input x E X' a path π = (y),.... Vr) is generated with probability m (TT
x) -II.-- (B(e Iz Vex), where ~e))(()) ( a)) 1. Each path is assigned binary reward r(z, TT) {0. 13. with the set of correct paths denoted by II, (xr) and incorrect paths by II_ (x) [[x=)) (1). The probability of sampling correct path, corresponding to Pass 1, is p+ (z; 0) = Σ rell()) тo( 2))((.)))
ExEIL, (x) To ( zz). To achieve the ultimate goal of improving Pass @ SFT and RL are utilized in training reasoning LLMss. SFT optimizes the model to maximize the likelihood of demonstrated trajectories n²:
(OF = -2(, >-D [log (a("" [x)]]
While RL aims to maximize the expected reward under the model distribution:
((0) EEND B- T)].
Put differently,
J(0) = En TT( x)) = ENND (p+ (c; 0)]
which corresponds exactly to maximizing the probability of sampling a correct path (i.c.. improving Pass@ l). Note that, in this research, RL refers policy--radient methods such as GRPO (Shao al. 2024).
In our rajectory-leve analysis, experimentally investigate how SFT and RL affect the LLM's paths by counting the number of unique paths. Let D = {xn { be an evaluation dataset consisting of N problems. For each input D, generate M independent samples from the trajectory distribution na(- x).
The set of trajectories observed in these samples is (x) {ir E I(x): Ej S m T(4) x). where (1). (m)
nT( | xx). This set includes both correct trajectories (x E IT (x)) and incorrect ones (a E (x)). Since M (z) is obtained by random sampling, not all trajectories in [I())) necessarily appear, but those with a higher probability mass are more likely occur multiple times within the M samples. We set M = 256 in the experiments.
Pass@k k (Chen et al. 2021 Yue et al. 2025 is the probability that least one correct solution is found when sampling & independent solutions from the model (i.c.. Best-of-k), which is given by Pass@ k(z; 0) Yue al. (025) found that as & increases, the base model catches up to the RL model
1-(1- p+ px )² A Pass@ k. indicating that the reasoning paths of the RL model are contained within the base model's sampling distribution. We evaluate the models listed in Table [ comparing the base model, SFT model, RL model, and SFT + RL model performance. Given their high capabilities, we conduct experiments on challenging mathematical datasets, AIME24, AIME25, and AMC23.
C.2 IMPLEMENTATION DETAILS
The chrF parameter was set to / = 2, and for UPGMA clustering, the similarity threshold was set to 60, meaning the distance similarity was calculated at 0.4.
23

---

Published as conference paper ICLR 2026
C.3 EXPERIMENTAL RESULTS
The changes in correct and incorrect paths for the models specified in Table 1 across AIME24, AIME25, and AMC23 are presented in Figure [4] The results demonstrate that RL reduces the number of incorrect paths, while SFT increases the number of correct paths. For the 14B model on AIME25 and AMC23, the EceReason-Nemotron 14rtroon-11BB (SFT RI.) model shows minimal changes in path count compared to the SFT model. However, Figure shows that BeeReason-Nemotron 14B does not show performance improvement over the pre-RL DeepSeck-RR L-Distill-Qwent 14B on the AIME25 and AMC23 domains, suggesting that the training may not have been successful at this domain.
AIME24
AIME25
AMC23
Overd Marth L.
Despleabe dwes 1.00
24.-9.21
SFT
SFT
32
1.5B
RL
Number of Unique Trajectories in Incorrect Outputs
RL
(1A1
SFT
(3.344331
43
29
0
Qwend 5-Hues-3
SFT
7B
RL
SFT
RL
BAHA
25
25
26
35
46
28
25
25
35
40
0
25
23
244
309
340
Oween 5-d-------geeee ieppeett-boo
Acaleuse Nom.
SFT
SFT
RL
99
14B
SFT
RL
RL (0-2.740
Number of Unique Trajectories in Correct Outputs
Figure 14: Effect of RL and SFT the Number of Unique Trajectories. The x-axis represents the number of correct clusters and the y-axis represents the number of incorrect clusters for trajectories before and after training of 1.5B, 7B, and 14B models in Table
Additionally, in Appendix we present examples where RL compresses unique incorrect trajectories, and examples where SFT preserves incorrect trajectories, resulting in different error patterns.
24

---

Published as conference paper ICLR 2026
C.4 MORE EXPERIMENTAL RESULTS
In addition to the models in Table 1 we conducted experiments on 1ween2.5-7B-SSmmleRl.. -ataa-31-----Smmple-------
and DeepSeek-RI-De -i--stlll-- Llama-8B (Model details appear in Table 2 The Pass@k k results for each model are presented in Fig- while the trajectory-level changes in the number of correct and incorrect paths are detailed in Fig-
DecReassnnNNemotron--..e.
alama-3.1-8B
ure 15
When Qwen2.5-7B undergoes RL training, incorrect paths substantially compressed. Moreover, 1.1-7B. after SFT and RL training, expands correct paths while squeezing incorrect paths AIME24 and AIME25. Conversely, for AMC23, of both correct and incorrect paths compared to the hase model. This phenomenon occurs because, in the AMC23 domain, shown in Figure 15 Owen2.5-Mathh77 achieves Pass@A k performance comparable AceReason- AccReason-Nemootron- 1.1-7B as k increases, leading to saturation. Finally, we confirm that Llama-3.1-8B also demonstrates squeezing of incorrect paths and expansion of correct paths following RL. training.
ure 16
exhibits squeezing
AIME24
AIME25
AMC23
200
100
130
601
Qwen-
40
32
125
Qwen Math- 7B
Pass @
62
60
90
20
20
Uemer 2.18
80
30
Llama-S0 8B
20
20
30
k
Figure 15: Pass@k performance curves for additional models. Models are from Table ttat included in Table and datasets are AIME24, AIME25, and AMC23.
25

---

Published as conference paper ICLR 2026
AIME24
AIME25
AMC23
129
118
100
100
Qwen- 7B
RL
40
20
20
Number of Incorrect Clusters (Paths)
13
25
33
15
Swen-MMat-Po 7B
SPT+RL
SFT+RL
40
40
SPT+RL
20
047AN8
133.4.31
36
15
25
30
15
22
24
15
(0A.337.4)
(2.2247744)
[18-2213.0]
Lioes 3.3-0
DeepSeelo Utti-Uuanedo
Llama- 8B
158
RL
150
150
KHAA
46
24
40
56
Number of Correct Clusters (Paths)
Figure 16: Additional Result the Effect of RL and SFT the Number of Unique Paths. The axis represents the number of correct clusters and the y-axis represents the number of incorrect clusters for trajectories before and after training of models in Table 2 that are not included in Table
26

---

Published as conference paper ICLR 2026
C.5 DIFFERENT SIMILARITY METRIC AND THRESHOLD.
Using BLEU as the similarity matrix, we performed identical experiments at the trajectory-level. The results were consistent with those obtained using chrF and 60 as a threshold for hicrarchical clustering. Furthermore, experiments with varying similarity thresholds also yielded identical results, shown in Figure 17
Number of Incorrect Clusters
(8.2,97.3)
SFT
(85.2.97.6)
Number of Incorrect Clusters
(3.3,48.3)
RL
50
RL
60
(86.7.70.7)
SFT
30
RL
40
(5.1.38.9)
20
20
10
RL * (5.6,6.8)
(2.4,8.6)
23.2.2.1)
40
100
0.0
2.5
5.0
7.5
10.0
12.5 15.0
17.5
20.0
Number Correct Clusters
Number ofCorrect Clusters
(a) BLEU Threshold = 50
(b) chrF Threshold 50
Figure 17: Number of Correct and Incorrect Clusters (Paths), (a) Different similarity metric: BLEU and
(b) Different Threshold of 7B models in Table Ton AIME24.
C.6 SIMILARITY DISTRIBUTION
Figure 18 illustrates the distributional characteristics of the upper triangular matrix elements (diagonal com- ponents excluded) derived from the similarity matrices in chrF-based clustering for unique path construction across models Table The results demonstrate that RL yields notable increase in similarity measure- ments.
27

---

Published as a conference paper at ICLR 2026
AIMEZS
AMCC3
AME24
DeopSenb--L--ircii Sitti
Memotron www.ee--eerrrreeeeesateeeee Qwen-1.5
AJM
APGC2
AIME24
AIMEIS
AMCZA
Swin2.5-Nathh77a
Wwwm2.5-Wath-Ta 2ho
AUME22
APREESS
ANC23
AMC23
ActRoasen Rometren:78 AIME2S
AIME24
APEENN
AMC21
KINE26
AMC21
2-142
AIV92S
AMC2
AJVIZA
AIMESS
AMC
DeepSesk-A1 Dislll--ween11
Acehlaaon Numotron-lte
Similarity
Trajectory Similarity Distribution. Frequency distribution of the upper triangular matrix ob- tained by extracting the diagonal elements from each model's similarity matrix across different models in Table I and three datasets, AIME24, AIME25, and AMC23.
Figure 18:
28

---

Published as conference paper ICLR 2026
C.7 REPRESENTATIVES OF SFT
In this study, we treat the models distilled from DeepSeek-R1 (Guo et al. 2025 as SFT models (Table 2 However, since these models generate multiple responses per problem during distillation, we conduct trajectory-level analysis for the case where SFT distillation uses single response per problem. We per- formed SFT on the Qwen2.5-Mathh 1...5---tth-15BB in Table ussing the s s1k-1.1 dataset from Muennighoff et al. (2025). We used the Adam optimizer with 10 learning rate, applying cosine decay schedule and weight decay 44, and trained the model for 5 epochs with a maximum sequence length (block size) of 20000.
We generated M=256 responses for AIME24, AIME25, and AMC23, computed pairwise similarities with chrF, clustered them using UPGMA with 60 as a threshold, and calculated the number of unique trajectories. The results shown in Figure 19 This figure shows that RL reduces the number of unique trajectories in incorrect outputs, while SFT increases the number of unique trajectories in correct outputs. This confirms that RL continues compress incorrect trajectories, while SFT expands correct trajectories.
Number of Unique Trajectories in Incorrect Outputs
AIME24
AIME25
AMC23
200
(6.9.188,9)
(8.3.910.))
Quenn @wind.S-Haatee Saatiis-Htth-SS 1.58 (Base]
Qwenn @wenz2.5-Wiid Setnnns--ttthhtt---o 2orb-------- (BL)
175
0wen2.55HHtth-..5 <llek11 (SFT)
330
SFT
133.7)
(4.4.88.2)
SFT
RL
5e
25
(.3.48.3)
[5.0.21.0)
Number of Unique Trajectories in Correct Outputs
Figure 19: Effeet RL and SFT on the Number of Unique Trajectories. The x-axis represents the number of correct clusters and the y-axis represents the number of incorrect clusters for trajectories before and after training 1.5 models.
29

---

Published as conference paper ICLR 2026
C.8 CODE DOMAIN
To validate our findings beyond the mathematical domain, we extend our experiments to code genera- tion using HumanEval (Chen et al.j [022]. For each problem, we generate
128 samples with 7B models in Table [ compute pairwise similarities using chrf, cluster them via UPGMA with threshold of 70, and calculate the number of unique reasoning traeectories. We set sampling parame- ters temperature=0, 6, top-p=0. 95 and max.tokens=16006 Qwen2.5-Math- 7B and Qwen2.5-Mann Swen2.5-Math-oe Oat--att-aaae Dar-Zero, and R1 Template for DeepSeek-RI. Di-ek-i----silll--- Rtl-Disill-Qwen-77 and AccReason-Nemotron--- U+2581 and U+FFSC replaced with and --" in Figure 20
We use Qwen Template for
Qwen Template
</im_start >system
You helpful assistant, imeend] </isstaart/uusrr (input)
Please reason step by step, and complete the above Python function. im__nd|>
<lim start Passistant
RI Template
<I of < >>< (input)
Please step by step, and complete the above Python 4 function. AsslAssist..n stant |><thinks
Figure 20: Prompt Templates for HumanEval.
HumanEval
(23.6.54.4)
Number Unique Trajectories
Incorrect Outputs
40
SFT
Qwen2.5-Math-78 (Base)
Maann25-Math-7--OttZZe (RL)
30
RL
DeepSeekkAL--isstill-ooeee (SFT)
MeReesson-Nemotron (SFT+RL)
20
10
(26.6,3.5)
RL
(48.9,8.4)
(5.8,4.6)
10
30
40
Number of Unique Trajectories in Correct Outputs
Figure 21: Effect of RI. and SFT on the Number of Unique Trajectories. The x-axis represents the number of correct clusters and the y-axis represents the number of incorrect clusters for trajectories before and after training of 7B models in Table on HumanEval.
These findings hold the code domain well, where we observe consistent trends: RL reduces the number of unique trajectories in incorrect outputs, while SFT increases the number of unique trajectories in correct outputs. In the code domain, RI. compresses incorrect trajectories, while SFT expands correct trajectories.
30

---

Published as conference paper ICLR 2026
D STEP-LEVEL ANALYSIS
D.1 REASONING GRAPH CONSTRUCTION
Graph Construction. Following Bogdan et al. (2025), we extract the trajectory up to the </think> token, then split it into sentences, using delimiters ., ?, (only when followed by space) |n|n, (rinjrin. If a chunk exceeds 300 tokens, forcibly split it, and if chunk under 10 tokens, merge it with the previous chunk. We used RAPIDS cuML's GPU-accelerated KMeans with scalable k-means++ initialization, running 10 restarts (n-init 10) and capping each run at 300 iterations (max_iter 300), with 2000 clusters un.clusters=200 The example for node is Table 3 and that for graph
is in Figure 22 and Figure 23
Node
Examples
Hmm, maybe this getting too convoluted.
#3 in 14B (Uncertainity Management)
Wait, perhaps this approach is also getting too messy. DDeepScck-R1-Distiii Sallll--wee--495 Wait, perhaps it's getting too tangled.
Wow, this getting messy.
-eaeston- 14B)
Therefore, y = 13650 + (-65/14) = 13650
11.65v/4 (DeepSeck-
RI RI-Distill-Qwee 11
#64 in 7B (Causal Aggregation)
Then, substitute 4.5 into equation 1: 84 _ 11vT4y
0 84 4.5 11 =0 84 4.5 378. (Qwen2.5-
Zath-
So plug x (AceRcason-Nemotronn
y 15vII: into x² 5x Fy
Now, we have equations of tangents at B and C: (AceRcason- Nemotron-7B)
#1384 in 7B (Interim Summary)
So. if let the tangents from the points of tangency on AB and AC and respectively, then a+bb 5+10-9 Dowen2.5-Math-2
So. to recap, tangent B: 28x 13v14y 140 (DeepSeek-B) RI--tstill Qwen-7B)
Table 3: Representative Node Examples. We converted all mathematical expressions to TRX.
Qwen2.5-Math-7B (Base)
Owwen.55MMth-77B-- Zero (RL)
DeepSeek-R1-Disse Qwen-7B (SFT)
AceReason-Nemootro 7B (SFT+RI)
Figure 22: Visialization of Reasoning Graphs. Results of 7B models Table D on AIME24 Problem Node encodes the node occurrence count, and edge length has meaning. Graphs visualized using Kamada- Kawai layout in NetworkX.
31

---

Published as conference paper ICLR 2026
1003 how Roden
pnioue Rodes 46422 Hodes
Unite Hode, C4725
Qwan2-5-Meth-1.18
Qwen2 5- S-Hath-1.55 1B5 Dat-
DeepSesk-1 R1- -i--iil-----ii
Hemotron-Rassare [SFT+
(a) 1.5B AIME24 Problem #1
inapen feee:: 2000 Tatd Madeo
14318 Reld Hodes
211 Usmuee Hoale 28437 bext lendes
14.99
Qwen2. Math-1.38
Owenz. 5: Math-1.5p Out-
DeepSesk-tt #13 Bistill
Nemotron Cenotron-Research
[SPT+
(b) 1.5B AMC23 Problem #1
dnrane todes best Mooes 1140 4004
sodes 2013 Your Rodees
419 ssague 20850 bas Hodes Ongee
46e useuu hodes lext bodes
30277
11.2
Owen2 7een2-5-Math-78 (Base)
Qwee2 5-Mathh78-Oott Zero (RL)
DengSeel-R1-Biitee Qwen-78 (SFT)
AceReason Nematron-78 [SFT+RL)
(c) 7B AIME25 Problem #1
254 tnigus Modes hodee 11841 heat
Inlous Sodee: 3717 Tele Rodee
460 keya Modeyy 30337 Modes Tatd
434 Uiroae Hodes lodes
S0aer Tate
Kdgg distanc
Qwen2.5-146 5-18 (Base)
Qwen-2.1 145 SimplefL. Zoo (RL)
DeepSesk- R1. Diabill Quen- 140 (SFT)
Acallesson Nemmbron-110 148 [SPT+RL]
(d) 14B AIME24 Problem #1
BE3 Unogas Edges 4045 boli Eages
umique Sdges 2002 Total Ddges
4237 Uriqu Ediges 2000 Teta cages
TOOR HERLY Tete Eages
thense Radee
Mdge dicance
Owen2. 148 (Base)
Owen:22 2.5-1488 1148 SimpleRL-
DeepSeck-ft R11 -i--iiii----iiilll- Biasil- Qween144
Aceftesson Nemotron:344
(c) 14B AIME25 Problem #1
Figure 23: Reasoning Graphs Examples. Nodes arranged sequentially on circle, consistent across exam- ples. Edge thickness encodes transition frequency, and edge color encodes edge distance.
32

---

Published as conference paper ICLR 2026
D.2 REASONING GRAPH ANALYSIS
Estimation of the Exponential Decay Rate. We illustrate in Figure 24 the rank plots obtained for Visita- tion Frequency, Degree, and Betweenness Centrality. We show in Figure illustrative example of linear regression analysis performed on the log-linear plot to estimate the exponential decay rate B.
Complete Results for 3. The box plots of estimated exponential decay rates / for visitation frequency, degree, and betweenness centrality across Base, RI,, SFT, and SFT+RL models AIME24, AIME25, and AMC23 datasets from Table[ are shown in Figure 26 The results demonstrate that RL training substantially increases all metrics relative the Base model, while SFT reduces them.
Edge Distance Distribution. The differences in edge distance (I2 norm of the centroid of sentence vectors for each node) across Base, RI., SFT, and SFT+RL. models are shown in Figure 27 However, No clear differences in edge distance distribution were observed.
Inter-model Similarity in Node Visitation Frequency. To investigate how RL and SFT modify the rea- soning graph, Figure 2 presents scatter plots of node visitation frequencies between pairs of models. Points closer to the line y = x indicate that the two models utilize nodes with similar visitation frequencies.
We employ the symmetric Mean Absolute Percentage Error (sMAPE) quantitative measure:
SMAPE 100
11 xel
where n represents the total number nodes, 2t denotes the visitation frequency node (t 1,..., n2) for the model on the x-axis, and y/: represents the visitation frequency of node for the model the y-axis.
This reveals distinct behavioral patterns: Base vs. RI. and SFT vs. SFT+RL exhibit relatively low sMAPE values, indicating that RL does not substantially alter the of visited nodes compared to the pre-RL models. In contrast, Base vs. SFT demonstrates a considerably higher sMAPE, suggesting that SFT significantly modifies the node visitation patterns relative the base model.
33

---

Published as a conference paper at ICLR 2026
Linear-Linean
Linear-Log
LineLinear
Log-Log
1.5B
Visitation Frequency
0::00
Eosss
330
200
Kank (1ne8r)
200
100
130
Kank (lasaar)
183
Mark Oog)
Bath Hag)
Degree
Betweenness Centrality
044
0.2
206
230
Bank (Bneer)
100
201
Rank (llveer)
300
549
Rant (lgg]
Rank llog)
7B
Visitation Frequency
035
045
100
200
(Bsear)
1X
110
Rani Hlgg
(lg)
Degree
1010
(lonar)
(ivsarr)
Kaok (og)
Rank (egi
Betweenness
Centrality
Mether
045
2024
0.55
0.2
100
200
175
102
200
127
32'
Rank
101 28
(og)
Figure 24: Rank plots for Visitation Frequency, Degree, and Betweenness Centrality. Results for 1.5B (top) and 7B (bottom) models in Table I on AIME24 Problem #1.
34

---

Published as conference paper ICLR 2026
Degree
Visit Frequency
Betweenness Centrality
Node
10-1
02485, R²= 0.944
f== 04215, 0.879
Betwoommess (log)
Frequency (log)
Degree (log)
10-
20-3
Node
2014
-0,06972. R² 0.924
62
60
50
130
120
140
55
100
120
140
40
99
Rank
(linear]
Rank (linear)
Rank (linear)
Qwen2.5-Math-L33
Degree
Visit Frequency
Betweenness Centrality
Nade
10-
Node
0.04407, R 0.981
f -0 08629, 0.973
Frequency (tog)
Degree (log)
Betweenness noweeenness
10-4
12-
p -0.1284, R 0.931
20
35
20
Rank (inear))
Rank (inear)
Rank (lnear)
0wenn25--ath--Ontt-znn---
Degree
Visit Frequency
Betweenness Centrality
Node
Node
10%
132
B 0.007741, A 0.983
A 0.01117, A - 0.990
13-9
quency (tog)
Dagres (log)
12
20-1
224
20
Node
B- 0.01945, 0.975
18P
13
Rank (linear)
Rank (linear)
Rank (linear)
Degree
Visit Frequency
Betweenness Centrality
Node
100
D. 0.009783, 8 0.985
Frequency (log)
10-1
p 0.01595, R²= 00990
13-
Degree (log)
20--
12
Node
13+
B= - -0002599, A² =2 0.968
10-1
100
200 (linear)
200
400
102
202
300
400
:00
200
200
Rank (linear)
Rank (linear))
Nemotron- Cemotron-Researcche Beaeech--Raaninn-OOO Qwaee--eennnia
Figure 25: Example of Exponential Decay Rate Estimation. Linear regression on semi-log plot of degree, visitation frequency, and betweenness centrality in AIME24 Problem #1.
35

---

Published as conference paper ICLR 2026
Visitation Frequency
Degree
Betweenness Centrality
1.5B
380
SFT+RL
SFT+AL
moan
minn
meon
7B
Baise
RL
SFT
SPT+RL
Bane
RL
SFT
SFT+RL
Ba
RL
SFT
SIT-RL
14B
Figure 26: Exponential Decay Rate for Visitation Frequency, Degree, Betweenness Centrality. Box plots show the estimated exponential decay rate B across all problems separated by 1.SB models in Table and datasets, AIME24, AIME25, AMC23.
36

---

Published as conference paper ICLR 2026
Sieen2.5-Mtth-1.BB
DeepSeck-RR---eee
Nemotron-Reseen
Owen2.5 Maath-1.38
Zero
Qwen-1.50
Serboninn-Qwen-1.50
1130
1.5B
6.5
Damety
(.00
1.00
73
15.0
123
21.
128
55.
20.5
183
172
1..
158
314
312
Edge
Edge
Edge
DeepSeek-R1-Diase
Qwen2.5 3-atth-78
Qwen2.5 3-Math-78-Oaa Oat--Zero
Qwen-78
AceReason- PetReason-Nemotrrn--1
7B
Demmyy
Daieity
Edge Distance
Edge Distance
Edge Distance
Edge Distance
DeepSeak-RR1Diaee Qwen-148
Qwen-2.5-14BB SimpleRL-Zoo
BicReason-Nemotrroe
5.5
14B
Denkyy
Danebyy
Edge Distance
Edge Distance
Edga Distance
Edge Distance
Figure 27: Edge Distance Distribution. Frequency distribution of edge distances shown in bins across models Table and datasets, AIME24, AIME25, AMC23.
37

---

Published as conference paper ICLR 2026
Base vs. RL
Base vs. SFT
SFT vS. SFT+RL
(SMAPS:
(SMAPS: 63.00%0
0433
Owen2 1.50 Olt Dah---arr (NL)
3.839
1.58
3:303
1.5B
Hometron
0643
C330
Qwen2.5 Math-1.58 (Ba00)
0wen2255WWth-1..0 (Dmen)
DeepSook RI -1---------------- Sesiil--wwen--a. [SF
06375
0.0075
GPNAEE: 39.00%)
Owenn 5-Noth-P 0at-3are (933
Reofessan Nemotron 7B (SPT++
9435
3.8335
7B
4332
0007
2.005
(390
EASEE CONS GELE OC15 CASS
E.GS case CUE0S J.125 3150
Qwen2.5 Math---ttt--. (Base)
Qwen2.5 Math-M (Dase)
SonpeenkA1-Oiitlloooo (SFT)
0274
0.803
120.655)
6.000
3024
9433
Acoffcaaren Monstrun
C.955
14B
3.84
9203
3353
Qwan2.5-148 (Bane)
Qwend 541 (DAAa)
DeepSenk- R.1 Dil---ietil--- -ww--148 (SFT)
Figure 28: Visitation Frequency Scatter Plot Between Two Models. Each plot represents a node, with coordinates indicating the visitation frequency in the integrated graph across all AIME24 problems for two models in Table [ The closer the plots are to the line y = x, the more similar ther visitation frequency of that node between two models.
38

---

Published as a conference paper ICLR 2026
D.3 STRUCTURAL GRAPH PROPERTIES
Graphlet Analysis. We graphlets (Milo et al. 2004 Pržulj al. 2004 to analyze the local structure of graphs. Graphlets have been extensively applied across diverse domains. including protein interaction networks (Pržul) al.) [006] Pržulj 2007). social network (Janssen et al., 2012), and world trade networks (Sarajlić et al. 201 Since counting 5-node graphlets is computationally hard and 3-node graphlets consist of only two types, which is insufficient to describe graph structures, we focus on 4-node graphlets and count the subgraphs shown in Figure 9a in the reasoning graph integrated across all problems G' for cach dataset. We then calculate the proportions of 4-node graphlets and compare them across models. As shown in Figure 29] across all models and datasets, RL consistently decreases linear graphlets G3 and G4 while increasing cyclic G7 and G8. G5 also shows a slight increase. SFT that imitates DeepSeek-Ro 's reasoning traces exhibits similar increases G7 and G8, suggesting that the teacher model's reasoning graph possesses comparable structural tendencies. Notably, all models except base models show highly similar graphlet proportions. However, their accuracies diverge significantly (see Figure 12). This indicates that despite similar local reasoning graph structures between RL-trained models from Base and SFT or SFT+RL models, substantial performance gaps persist in reasoning capabilities.
AIME24
AIME25
AMC23
Ooeed Natt cul 2ooe
1.5B
Proportion (%)
(wwadd Miaah-- (have)
Ovephess 13 dist owarts (997))
7B
1.0 Rimullats 181)
Acattasoon Hemur an- (int-ee
14B
Figure 29: Poroportion of 4-node Graphlets. Graphlet proportions for models in Table 1 on AIME24, AIME25, and AMC23.
Global Structures of Reasoning Graph. We calculated edge density, clustering coefficient (Watts & Stro-
gatz 1998), assortativity (Newman 2002), modularity (Girvan & Newman 2002), Freeman centralization Freemann 1978), average path length (Watts & Strogatz, 1998), global efficiency (Latora & Marchiori. 2001). and algebraic connectivity (Fiedler 1973) for each model and each problem to examine differences
in complex network structure (Newman 2003), and averaged these metrics within the dataset. We utilized NetworkX library. The results are shown in Figure 31
Here, we consider an undirected graph 9 = (V,E).
39

---

Published as conference paper ICLR 2026
Edge density is given by
2
((
[V((V-1)
Edge density is ratio of the number of observed edges to the maximum possible number of edges in the graph.
The local clustering coefficient (Watts & Strogatz 1998) is given by
t (: - 1)²
C((C)
where & denotes the number of triangles and ky is degree involving node i. The global clustering coefficient is given by
C(g) 1
M EC,
Clustering coefficient the proportion observed connections among the neighbors a node relative to the number of possible connections random the graph. To compare graphs of different sizes, we normalize C(g) by the average length ((rand)) of a random graph.
Assortativity (Newman 2002) is given by
kūKj 2/
R(G) Eisev
Ais
kisk
ki
Σijer
kioōs.j
kg,k
21
where Ar, if there is an edge between and j. ki -,
E, Ac,j is the degree of node i. Assortativity is the Pearson correlation coefficient hetween the degrees of nodes at the ends edges. A highly assortative net- work is one where high-degree nodes connect with other high-degree nodes, and low-degree nodes connect with other low-degree nodes. In contrast, a disassortative network has a hub structure, where high-degree nodes connected low-degree nodes.
For a partition {ci), modularity (Girvan & Newman 2002 is given by
1
k, o(c),,)),
Q(9)
A
2|
2|
IJEV
where Ai. = if there an edge between and j. k; = E, As.j As, is the degree node and 8(cs, ez) if and j belong the same community. Modularity measures the strength of division of network into communities, relative to a random graph.
Freeman centralization (Freeman 1978) is given by
CD(g) = (deya d(i))
11(11)
where we use the denominator (IV) 11(2)))())))))) normalize to ranges between 0 and (0D)-1)(V112) corresponds to the value achieved by star graph. Freeman centralization quantifies the extent to which the network's connectivity organized around a central node.
Average path length (Watts & Strogatz 1998 is given by
L(g)
Σ d(i.j),
&jEViin
40

---

Published as conference paper ICLR 2026
where d(i,j) denotes the length of the shortest path between nodes and j. Average path length is the mean of the shortest path lengths between all pairs of nodes in the network. To compare graphs of different sizes, we normalize L(g) by the average length [(ram)) of random graph.
Global efficiency (Latora & Marchiori 2001) is given by
E(g) 1I(V--1
E
AJEVi dii.j)
where d(i.j) denotes the length of the shortest path between nodes i and and use (V - 1)(V 2) for normalization. Global efficiency is the mean the inverse shortest path length across node pairs, indicating communication efficiency.
Algebraic connectivity is given by the second smallest eigenvalue of the graph Laplacian, which reflects the robustness of network connectivity. Low algebraic connectivity indicates that the graph can be easily disconnected into separate components by removing only a few edges or vertices.
C(9)/C( rand) The small-worts index (Watts & Strogatz. 1998) can be obtained as a L(G))/((Gandd)
Minegishi al.
(2055) analyzed the small-world index of reasoning graphs.
Relationship with Pass@k. Comparing the results in Figure 30 and Figure
observe that Pass @ 1 @ is positively correlated with Global Efficiency and Algebraic Connectivity, while negatively cor- related with Modularity. A higher Pass@1 Pass@k k ratio indicates that the improvement from Best-of-k sampling over single inference is marginal, suggesting that the model can effectively explore the solution space and reach the correct answer in a single attempt. This reasoning capability is associated with graph structures that exhibit low modularity and facilitate traversal across the entire graph, enabling efficient navi- gation between distant nodes.
1.5B
7B
1.5B
Pass @
26
64
Pass @
32
Bone
Bose
SPT-LL
Beae
RL
SFT
SFT-
Figure 30: Pass@1 1/ Pass @ k performence by model. The average Pass Pass @ across AIME24, AIME25, and AMC23 for each model Table
41

---

Published as conference paper ICLR 2026
1.5B Edge Density
7B Edge Density
14B Edge Density
A35
605
Base
RL
SFT
SFT+RL
Base
RL
SFT
SFT+RL
Base
RL
SFT
SFT+RL
Normalized Clustering Coefficient
Normalized Clustering Coefficient
Normalized Clustering Coefficient
SFT+RL
SFT+RL
Assortativity
Assortativity
Assortativity
-4
Modularity
Modularity
Modularity
14
Base
RL
SFT
SPT+
Base
RL
SFT
SFT+RL
Base
RL
SFT
SFT+RL
Freeman Centralization
Freeman Centralization
Freeman Centralization
Normalized Average Path Length
Normalized Average Path Length
Normalized Average Length
134
134
055
Base
RL
SFT
SFT+RL
Base
RL
SFT
SPT+RL
108
Base
RL
SFT
SPT+RL
Global Efficiency
Global Efficiency
Global Efficiency
12
62
12
Algebraic Connectivity
Algebraic Connectivity
Algebraic Connectivity
Base
RL
SFT
SFT+RL
Base
RL
SFT
SFT+RL
Base
RL
SFT
SFT+RL
Figure 31: Graph Structural Metries of Reasoning Graphs. Each model size in Table shows mean values (averaged across AIME24, AIME25, AMC23) for eight core graph structural metrics: edge density, clustering coefficient, assortativity, modularity, Freeman centralization, average path length, global effi- ciency, and algebraic connectivity.
42

---

Published as conference paper ICLR 2026
D.4 ABLATION OF REASONING GRAPH
Visitation Frequency
Degree
Betweenness Centrality
K=1000
A11M
0.5
633
0.211
620
00011
Base
SFT
SFT+RL
Base
SFT
SFT+AL
Base
RL
SFT
SFT+AL
4.00
menn
maa
men
K=3000
0.00
2.41
R1
SFT
SFT-RL
Base
SFT
SFT+RL
Base
RL
SFT
SFT+LL
moon
Cosine
1.94
0.30
1125
0.041
0,001
SFT
SFT+RL
Daxe
SFT
SFT-RL
Bate
SFT+LL
Embedding
4.5
4.0
SPT
SPT-RL
Base
SPT
SFT-RL
Baxe
SFT+RL
Figure 32: Exponential Decay Rate for Visitation Frequency, Degree, Betweenness Centrality. Box plots show the estimated exponential decay rate 3 across all problems in AIME24 for the 7B models in Table [ K=1000 and K=3000 denote the number of clusters in K-means clustering, 'cosine' indicates the use of cosine distance K-means, and 'embedding' refers to the use of GTE-base-en-vi.5 for sentence embeddings.
We conduct an ablation study to assess the impact of different reasoning graph construction methods on our results. In the main results, we construct reasoning graphs by first segmenting generated reasoning traces into individual sentences, obtaining sentence embeddings using BGg---arge----vl.5 (Xiao et al., 2024), and clustering them via K-means with K 2000 and L2 norm to define graph nodes. We systematically vary the number of clusters K, the distance metric, and the sentence embedding model to analyze how these design choices affect the exponential decay rates underlying the "RL squeezes, SFT expands" phenomenon. We estimate the exponential decay rate 3 for the 7B models on AIME24 (Table T under three alternative configurations: (i) varying the cluster count K 1000 and K 3000, (ii) replacing the distance
43

---

Published as a conference paper ICLR 2026
metric with cosine distance, computed via L2 normalization of sentence embeddings, and (ii) substituting the embedding model with GTE-bbaeeen-vi.55 (Zhang et al. 2024). which produces d 768 dimensional representations.
Figure 32 shows that across all ablations, RL consistently increases the mean exponential decay rate B, while SFT decreases it. This pattern remains consistent with the main results presented in Figure 26
D.5 SPARSIPYING REASONING GRAPIIS
As described in Section[4.1 defined nodes by clustering sentence embeddings and analyzed the properties
reasoning graphs. Figure [3 shows that the edge density is approximately 0.1 for the RL model and 0.05 for others, indicating these are not sparse graphs. Nevertheless, we examine the behavior of these metrics graph sparsity increases. We construct reasoning graphs for the 7B models Table ffoloo]oo Section 4.1 and apply distance-basee
sparsification in the sentence embedding space, We sort the edges connected
each node by L2 norm and retain only the sop-10 or top-20 closest edges. Nodes with a degree under 10 or 20 retain all their edges.
SFT
SFT-LL
Bess
AL
SFT
SIT+
Bame
AL
SFT
SIT+AL
SIT
SIT-RL
(a) Degree Top-10 10
(b) Degree Top-20
(c) Centrality Top-10
(d) Centrality Top-20
Figure 33: Exponential Decay Rates of Degree and Betweenness Centrality (Centrality) on Sparsified Reasoning Graphs. The box plots aggregate the estimated decay rate for the 7B models in Table T after sparsifying the graphs using the top- 10 and top-20 distance- based sparcification. Results are combined across AIME24, AIME25, and AMC23.
presents box plots of the exponential decay rates 3, estimated across AIME24, AIME25, and AMC23 for the 7B models in Table Figure 33 shows that when sparsifying based on sentence embedding distance (top-10 and top-20), RI. increases the exponential decay rate while SFT decreases it. These results are consistent with the non-sparsified results Figure 6 An important caveat that our graph construction method (Section[4.) ensures the reasoning graph weakly connected by designating problem x as the initial node. However, sparsification may disconnect the graph and create unreachable nodes whose betweenness centrality becomes zero.
Figure 33
D.6 REPRESENTATIVES OF SFT
Following Appendix C.7
conduct step-level analysis for RL and SFT when performing SFT with one response per problem on slk- dataset(Muenmighoff et al. 2025)
After constructing the reasoning graphs following Section 4.1 and Appendix D.I with M 256 and K = 1000, we estimated the exponential decay rates of Visitation Frequency, Degree, and Berweenness 1-MM-aath55 (Base), Qwen2.5-Math- 11e---..------- (RL), and Swen2.5-Math-1.53 s1k-1.1 (SFT) on AIME24, AIME25, and AMC23. Consistent with Figure RI. exhibits higher decay rates, whereas SFT exhibits lower decay rates.
Centrality for
We then computed the topological metrics of the reasoning graphs and present them Figure 36 The results exhibit trends consistent with Figure g
44

---

Published as conference paper ICLR 2026
Density
Normalized clustering Confficient
Assortativity
Modularity
64
-A1
6:350
EJ05
SIT
OFT-RL
Freeman Contreltzation
Normalized Avarage Path Length
Global Emfiiency
Algebraic Connectivity
S41
SIT+AL
Use*
SF1+L
SFT
54 TelL
28m
8L
SF1
5F1-AL
Edge Density
Normalized Clustering Coofficiont
Assertativity
Modularity
64
6,055
-4
6.05
B3*
3FT
STTHLL
B4.*
SPT+LL
Bade
SFT
SPTHLL
SFT SFT-AL
Freeman Contralization
Normalized Average Path Length
Global Efficiency
Algebraic Connectivity
AF7
Rooe
RL
MFTIL
4PT-BL
Figure 34: Comparison of Eight Graph Metries for the Sparcified Graphs with the Top-10 (Up) and Top-20 (Bottom) Distance-based Sparcification Method across Base, RI, SFT, and SFT+RL Models (7B). Values are averaged across three datasets, AIME24, AIME25, and AMC23.
Visitation Frequency
Degree
Betweenness Centrality
1.0
68
0.021
Base
Figure 35: Exponential Decay Rate for Visitation Frequency, Degree, Betweenness Centrality. Box plots show the estimated exponential decay rate across all problems in AIME24, AIME25, and AMC23 for the 1.5B models in Appendix D.6
9.0
7.
5.0
0,
2,5
Bane
Base
Centrality
Length
Erficienccy
Algebraic
9.4
675
6.59
5.6
Bane
Ba
Base
Figure 36: Comparison of Eight Graph Metrics across Base, RI, and SFT models (1.5B). Values are averaged across three datasets, AIME24, AIME25, and AMC23.
45

---

Published as conference paper ICLR 2026
D.7 CODE DOMAIN
Visitation Frequency
Degree
Betweeness Centrality
Base
SPT
SPT-
Base
SPT
SPT-RL
RL
SPT-RL
Figure 37: Exponential Decay Rate for Visitation Frequency, Degree, Betweenness Centrality Box plots show the estimated exponential decay rate / across all problems in HumanEval on the 7B models in Table
Edge Denaity
Clustering
Assortativity
Modularity
8.3
Efficiency
33
66
800
Figure 38: Comparison Eight Graph Metries across Base, RL, and SFT models (7B) on HumanEval.
In addition to Appendix C.8 to assess the generalizability of our findings across domains, we conduct ex- periments on code generation using Human (Chen et al. (021]. For each problem instance, we generate M = 128 samples with the 7B models in Table [ We set sampling parameters to semperature- 6, top_p=C. .95 ,95 and max_ max.tokens==100 6000.. We use Qwen Template for Qwen2.5-Math-77 and Qwen2.5- Math-Oat-Zero, and R1 Template for DeepSeek- R RR1-Distill-Qwe Qwell-BeeennBBBB and AceReason-Nemene Nemotron-7B.. Then, we construct reasoning graphs following the same methodology described in Section 4.1
Figure 37 shows box plots of the exponential decay rates of the rank plots of the reasoning graphs for each HvmanEval problem. We observe that, for visitation frequency. degree, and betweenness centrality, the exponential decay rate 3 increases with RL, whereas B decreases with SFT.
Furthermore, we compute eight topology metrics of the reasoning graph for each sample and report their averages over all samples in Figure 38 In this domain, obtained results consistent with those in Figure g
46

---

Published as conference paper ICLR 2026
E EXAMPLES OF TRAJECTORY-LEVEL ANALYSIS
An example where incorrect trajectories from Qwen2.5-Math-7B5 (RL) are similar to each other, while those from DeepSeek-RI - -iisiill-QQen----- (SFT) are dissimilar.
Qwen2.5-Math- -B---rrZer (RL) AIME24 Problem #2 m=182 (vs. 230) chrF=91.88 (Incorrect)
To solve this problem, we need use some geometric properties and algebra. Here's the step-by-step solution:
**Identify the Power of a Point:* The power of point A with respect to the circle w can used to find the length AP. The power of point A is given by AB = .P. ADD. 2. **Find the Length *D:** Since D is the intersection of the tangents the symmedian point of triangle ABC. The length AD can be found using the formula for the symmedian point:
B and C, D
2bc
a²
AD
| bc
1
1 + c²
(b + c)2
where BC = 9, AC 10, and
AB 5.
3. *Calculate AD:**
- AD
92
100
81
144
102 + 5
V
10
(10 + 5)2
125
50
1
225
5
50
225
(50
144
7200
4
16
=
225
5
225
5
32
5
4
5
4. #*iind AP:** Using the power of point A:
16
16v2
AB
AP. AD
52 AP.
25 AP
25
125
125
AP
16v2
16v2
32
To rationalize the denominator, we multiply the numerator and the denominator by /2:
125
125
250
125
AP
32
/2 v22
64
32
So. AP 125
5. **Find m and n: # Here, m 125 and n = 32, and they are relatively prime. Therefore,
+ 125 32 157.
The final answer is 157
Owen2.5-Math-7 7B-Oat-Zero (RL) AIME24 Problem #2 m= 182 vs.) 230 chrF=91.81 (Incorrect)
To solve this problem, we need to use some geometric properties and algebra. Here's 's the step-by-step solution:
47

---

Published as conference paper ICLR 2026
1. #*Identify the Power of Point*#; The power of point A with respect to the circle can used to find the length AP. The power of point A is given by AB AP AD.
2. **Find the Length AD**; Since D is the intersection of the tangents symmedian point of ACCC with respect to A. The length AD can be found using the formula for the symmedian length:
B and C, is the
2bc AD
a²
bc
1
62 + c²
(b c)2
where BC 9 AC 10, and
AB
5.
3. *Calcullaee AD**:
2 5
92
100
81
4
144
AD
10a
+
V
10
(10 + 5)²
125
50
1
225
5
50
225
50
144
7200
4
16v2
=
225
5
225
322
5
4
5
4. **Find AP**; Using the power of point A:
16
16v2
AB
AP.
AD
52
AP.
25 AP
25-5 5
125
125
AP
16v2
16v2
32
5. *Rationalize the Denominator
m To express AP in the form
n we rationalize the denominator:
125v2
125
250
125
AP
32
32
v2 v2
64
32
So, AP = 125
32 The integers m and n are 125 and 32, respectively, and they are relatively prime.
Therefore, m + n 125 + 157.
The final answer 157
DeepSeek-Re RR--iitill-Qwent (SFT) AIME24 Problem m=172 (vs. 196) chrF=29.56 (Incorrect)
Okay, so I'm trying to solve this geometry problem. Let me read it again: We have triangle ABC inscribed in circle The tangents to at B and C' intersect point D. Then, AD intersects w again at P. We're given AB = 5, BC = 9, and AC = 10. We need to find coprime, and then compute Alright, let's break this down step by step. First, let me visualize the problem. We have triangle ABC with sides AB cyclic quadrilateral we consider the circle. The tangents at B and C meet at D, so D is the ex-pole or something? Hmm. not sure. Maybe need to recall some properties.
AP in the form where and
BC and AC 10. It's inscribed in a circle w, which means it's a
Since D is the intersection of the tangents at B and C, D must lie on the polar of the line BC with respect to circle w. Wait, maybe should think in terms power of point.
Yes, the power of point D with respect to w should be equal to DB² DC since DB and DC are both tangent lengths from to w. So, DB DC', That might be useful later.
48

---

Published as a conference paper ICLR 2026
Now, AD intersects again at So, AD secant of the circle, starting A and going through P. We need to find the length AP. Given that, perhaps can use the Power of a Point theorem. The Power of Point with respect to circle is AB x AC' because A is the circle, so its power is zero. Hmm, that doesn't teesn't seem immediately helpful.
Wait, actually, A is on the circle, any line through A intersecting the circle again will have power zero. So, the power of is zero, which means AP x AD O. But that's trivial because AD passes through A. AP X AD is just AP X (AP + PD). but since A is the circle, AP is the length from to P. and PD is from P to D. Hmm, maybe need a different approach.
Alternatively, maybe can harmonic division projective geometry concepts, but I'm not sure. Maybe coordinate geometry? That might be messy, but perhaps manageable. Alternatively, maybe using inversion? Since have tangents and circles, inversion could be useful. But that might be a bit advanced for me right now.
Wait, let's think about triangle ABC. We know all three sides: 5, 10. Maybe we can compute some angles or the circumradius.
Yes, that might be good starting point. Let me compute the circumradius of triangle ABC. Using the formula:
R= abe T23 Where b, are the sides, and A is the area of the triangle. First, let's compute the area A using Heron's 's formula.
The semi-perimeter
5+9+10 21 12.
Then, A = ss(s - a)(s (s c) /12(12 5)(2 9) 10) = 112
Calculating that: 12 x 7 = 84, 84 3- = 252, 252 x 2 504. So. A = v504.
Simplify / /504. Let's factor 504: 504 = 252, +2 again 126, +2 63, +3 21, +3 So, 504
23 x 32 x 7. Therefore, v504 2 x3x /7 2 v2x x /7 = 6v14. Wait, let me check
that.
Wait, v V504- = VI6 x 31.5. Hmm, maybe did that wrong. Wait, 504 36 = 14, so /504
/36 x = 6414. Yes, that's correct.
So, A 6414.
Now, compute Rt
alc
Let's assign BC = -, b= AC = 10, == AB = 5.
TA
So, R = 9x 9x10x5
450
4x6v14
24v14'
[continued]
(SFT) AIME24 Problem #2 m=(172 vs.) 196 chrF=29.55 (Incorrect)
DeepSeek-BE
Okay, I've got this geometry problem here about triangle ABC inscribed in circle w. The tangents at B and C meet at point D, and then AD intersects again at P. The sides AB, BC, and AC are given 9, and 10 respectively. I need find the length of AP, express it a reduced fraction m/n, and then find m n. Hmm, okay, let me try to figure this out step by step.
First, I should probably sketch the problem visualize better. So, triangle ABC with sides AB=5, BC=9, AC=10. That seems like a scalene triangle. The circle w is the circumcircle of ABC, which means all three vertices lie on w. The tangents at B and C intersect D. So, is the ex-pole of BC with respect to w? Or maybe it's just the ex-pole. Hmm, tangents from common external point to a circle are equal in length, so DB DC, That might be useful later.
49

---

Published as conference paper ICLR 2026
Then, the line AD intersects w again at P. So, AD a secant line from A through intersecting the circle again at P. So, AP is the segment from A to P, which is part of AD. So, I need to find the length AP.
Given the sides of the triangle, maybe can use some properties of circles, power of a point, similar triangles, or something like that. Let me think.
Given that D is the intersection of the tangents at B and C, D is the ex-pole opposite A, right? Or maybe it's the ex-pole? Wait, actually, the point where two tangents meet called the ex-pole it's outside the circle, which it is because tangents So, point D is outside the circle w. and DB and DC are tangent to w. So, by the power of point with respect to w, have DB DC DA DP. That might come in handy.
drawn from outside the circle.
Power of point says that for a point D outside a circle, the square of the tangent length is equal to the product of the lengths of the secant segments. So, in this case, DB DA DP.
So, I can find DB, then can relate it to DA and DP. But I don't 't know DB yet. Maybe can compute it somehow.
Alternatively, maybe can use coordinate geometry. Assign coordinates to the points and compute everything. But that might get complicated. Let me see if can find more synthetic approach. Wait, perhaps using triangle similarity some theorem related to harmonic division or something. Let me recall that in triangle ABC, with D being the ex-pole, AD is called the symmedian. So, AD is the symmedian of triangle ABC. That might be useful property.
Symmedian reflects the median over the angle bisector. So, can find the median from A, then reflect it over the angle bisector to get the symmedian.
But I don't 't know if that's 's directly helpful here. Maybe using power of point is still a better approach. So, let's try to find DB. Since DB is the tangent from D w, and D is the intersection of tangents B and C. So, is the ex-pole, and DB = DC. [continued]
50


Reviews:
[{"review_id": "oSVYqYqSK2", "reviewer": "Reviewer_oB2v", "summary": "The authors examine how two training strategies, reinforcement learning with verifiable rewards (RLVR) and supervised fine-tuning on reasoning traces (SFT), reshape LLM reasoning. Their central result: RLVR compresses the reasoning space, reducing path diversity and concentrating effort on a small number of high-impact steps, whereas SFT expands it, introducing many correct solution paths and distributing functionality across more steps. This contrast clarifies why the standard SFT + RL pipeline succeeds: SFT first generates a richer set of good paths; RL then trims the weak ones to raise Pass@1.", "questions": "1. Can we test on other domain datasets to see if the observed effects replicate?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The trajectory and step level analysis for both SFT and RL are interesting to me.  \nTrajectory clustering shows RL compresses incorrect paths while SFT expands correct ones, motivating the common two-stage recipe (SFT then RL).\n\n2. The paper builds “reasoning graphs” and profiles them with edge density, modularity, assortativity, centralization, global efficiency, etc., revealing RL concentrates functionality into hubs while SFT distributes it.\n\n3. The paper is easy to follow.", "weaknesses": "1. Even though the analysis is interesting, the paper’s scope and novelty are not sufficient for ICLR.\n\n2. The domain and model scope are narrow. The evidence centers on verifiable math datasets and a limited set of public checkpoints, leaving generalization to coding/science tasks and broader architectures untested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "04Ius8XLtf", "reviewer": "Reviewer_YntV", "summary": "This paper analyzes how SFT and RL on Verifiable Rewards shape the LLM reasoning process. They perform a two-level analysis: (1) trajectory-level clustering of sampled solutions, and (2) step-level reasoning graphs built from sentence embeddings of the models' CoT. The core findings are the following: (1) RL compresses incorrect trajectories while SFT expands correct trajectories, helping explain why SFT --> RL is the prevailing training recipe; (2) at step level, RL also consolidates reasoning graph functionality into fewer steps. Experiments are shown for 1.5B/7B/14B models on AIME24/25 and AMC23 with controlled sampling.", "questions": "1. Any preliminary results on code or scientific reasoning to test whether RL still weakens community structure and SFT boosts global efficiency?\n2. I'd suggest the Figure should provide some insight, and the caption should explain it. For instance, what's the value of the 4 versions of the same plot in Figure 4? Or similarly, what is the value of each metric in Figure 7?\n3. The findings in the paper are interesting, but the authors fail to point out how they are useful for improving them, do you have some comments on how we can leverage these insights?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It presents a timely analysis for a RL/SFT fine-tuning on LLMs, which helps explaining and understanding  the use of these techniques for fine-tuning. \n2. The authors support their insights with a solid set of experiments which include multiple models. \n3. The authors perform a detailed analysis at the whole reasoning trajectory, and at the step level combined with fundament computer science analysis based on graphs.", "weaknesses": "1. Reasoning graphs depend on (a) sentence segmentation, (b) embedding model, (c) k=2000, and (d) Euclidean distances. Some ablation on of these points can help strengthen the derived insight. Some of these ablations should be easy to run. \n2. Local graphlet proportions can look similar across models while accuracies differ; more analysis is needed to connect which topological features track Pass@1/Pass@k\n3.  In the graph analysis, step-level graphs aggregate all sampled responses for a problem without separating by correctness, so topology may conflate successful and failing behaviors.  Similarly, edge distance is Euclidean between K-means centroids from a single embedding model; no evidence is given that this correlates with semantic step difference, and cosine/other spaces aren’t tested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0vqNelo2gd", "reviewer": "Reviewer_pCKm", "summary": "This work provides a novel framework to analyze the properties of two post-training techniques of LLM - SFT and RL.\nIt investigates the reasoning processes of base, SFTed, RLed and SFTed+RLed models at two levels of granularity: the trajectory-level which examines complete reasoning outputs, and the step-level, which analyzes reasoning graphs whose nodes correspond to individual reasoning steps.\n\nBased on the analysis existing models, covering Qwen2.5-Math-1.5B, Qwen2.5-Math-7B and Qwen2.5-14B, the authors get the conclusion that RL compresses incorrect trajectories, whereas SFT expands correct ones (trajectory-level); RL concentrates reasoning functionality into a small subset of steps, while SFT homogenizes it across many steps (step-level).", "questions": "1. Do the properties and construction methods of SFT datasets influence the resulting SFT models? If so, could this affect the validity or generality of the paper’s conclusions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work proposes a novel and solid analysis framework based on reasoning paths to better understand the effects of SFT and RL in LLM post-training, particularly from the perspective of reasoning diversity (e.g., pass@k vs. pass@1).\n\n2. The framework design is rigorous and well-grounded in theory.\n\n3. The paper is clearly written and well-presented, making the methodology and findings easy to follow.\n\n4. The experimental analysis is comprehensive and covers multiple model scales and post-training settings.", "weaknesses": "## Representativeness of the SFT Models\n\nI am concerned about whether the models chosen in this paper accurately represent conventional SFT. \n\nThe authors use the DeepSeek-R1-Distill-Qwen series as SFT models; however, these models are trained via knowledge distillation (KD), which differs from standard SFT practices. In KD, for each prompt, multiple responses from teacher models are used for supervision (as described in Sec. 2.3.3 of Guo et al., 2025). This setup might make properties diverge from a pure SFT setting and potentially enhance the diversities by incorporating richer COT paths. (In fact, it is somehow close to offline RL).\n\nTherefore, the behavior of KD-based models may not truly reflect that of conventional SFT models. If this is the case, some of the conclusions in the paper may not fully hold.\n\n\n\n\n--------------_\n\n- Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bUsVqImnb", "reviewer": "Reviewer_mxYj", "summary": "Summary:\nThis paper compares how reinforcement learning (RL) and supervised fine-tuning (SFT) shape reasoning in large language models. RL “squeezes” reasoning by concentrating paths, while SFT “expands” it by diversifying correct trajectories. Graph-based analyses explain why the SFT to RL pipeline works effectively.", "questions": "The main findings rely heavily on clustering reasoning trajectories and sentences, yet the paper reports no clustering-quality metrics such as ARI, NMI, or Silhouette scores are reported. Without such validation, it is difficult to assess whether the clustering results genuinely reflect reasoning structure or are artifacts of algorithmic choices.\n\n\nThe paper adopts BGE-large-en-v1.5 (Xiao et al., 2024) for sentence embeddings instead of using the LLM’s own internal representations. What is the rationale for this choice? Given that the goal is to analyze reasoning behaviors within each model, it seems more natural to extract embeddings directly from the model under analysis, since external embeddings may not align with the model’s latent reasoning space.\n\nThe paper states that edge weights are defined by the Euclidean distance between cluster centroids, which suggests that every pair of nodes could be connected, effectively creating a dense graph. Could the authors clarify whether any thresholding, sparsification, or top-k selection was applied when constructing edges? If not, how should we interpret structural metrics such as degree, modularity, or centrality that assume sparse connectivity?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper shows a novel analytic framework for The trajectory- and step-level “reasoning graph” approach provides a creative and rigorous lens for understanding LLM reasoning beyond metrics like Pass@k.\n\nThe work sheds light on why SFT+RL remains the dominant fine-tuning recipe for reasoning LLMs and offers potential implications for efficient data curation and model design.", "weaknesses": "The evaluated datasets (AIME24, AIME25, AMC23) are all from the same competition series and share similar problem distributions. This narrow scope makes it difficult to generalize the conclusions. For example, the claim that “applying SFT to the Base model increases the number of correct trajectories, showing that SFT teaches new solution strategies absent in the Base model” is not consistently significant, particularly for the AMC23 dataset and the 14B model.\n \nThe experiments use pretrained and post-trained models from the same family (e.g., Qwen and DeepSeek-based models). This raises concerns that the authors may not have controlled for shared training samples or overlapping data distributions, which could confound the observed “squeeze–expand” effects.\n \nThe paper omits several key implementation details required for replication. See question for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)