Title: Topology of Reasoning: Understanding Large Reasoning Models through Reasoning Graph Properties
Venue: NeurIPS 2025
Accepted: yes
Keywords: Interpretability

Reasoning

LLM
Abstract: Recent large-scale reasoning models have achieved state-of-the-art performance on challenging mathematical benchmarks, yet the internal mechanisms underlying their success remain poorly understood. In this work, we introduce the notion of a reasoning graph, extracted by clustering hidden‐state representations at each reasoning step, and systematically analyze three key graph-theoretic properties: cyclicity, diameter, and small-world index, across multiple tasks (GSM8K, MATH500, AIME~2024). Our findings reveal that distilled reasoning models (e.g., DeepSeek-R1-Distill-Qwen-32B) exhibit significantly more recurrent cycles (about 5 per sample), substantially larger graph diameters, and pronounced small-world characteristics (about 6x) compared to their base counterparts. 
Notably, these structural advantages grow with task difficulty and model capacity, with cycle detection peaking at the 14B scale and exploration diameter maximized in the 32B variant, correlating positively with accuracy.  
Furthermore, we show that supervised fine-tuning on an improved dataset systematically expands reasoning graph diameters in tandem with performance gains, offering concrete guidelines for dataset design aimed at boosting reasoning capabilities. By bridging theoretical insights into reasoning graph structures with practical recommendations for data construction, our work advances both the interpretability and the efficacy of large reasoning models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces reasoning graphs, which are constructed by clustering hidden states of LLMs during reasoning tasks. It analyzes their graph-theoretic properties (cyclicity, diameter, small-world index), demonstrating that reasoning models exhibit significantly more cycles, larger exploration diameters, and stronger small-world characteristics compared to base models. The work provides new insights into the structural mechanisms behind advanced reasoning capabilities and practical guidance for data design.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. This paper is well-organized and easy to follow.\n2. The idea of extracting reasoning graphs from LLM and exploring their properties is interesting, offering a new perspective for understanding the model's reasoning mechanisms\n\n**Weaknesses:**\n1. The conclusions of this paper are only validated by three math reasoning datasets, leaving their generalization to a broader domain (such as logical reasoning, code reasoning, and general tasks) uncertain.\n2. The paper emphasizes that the proposed method can guide the design of training data to enhance reasoning capabilities, but it fails to explicitly specify how to construct such data in practice. Moreover, this approach resembles a post-hoc analysis that incurs prohibitive computational costs—since it requires inference across multiple datasets before yielding actionable insights—severely limiting its real-world applicability.\n3.  I am not convinced that the approach proposed in the paper—selecting better datasets solely based on the diameter of the reasoning graph—is practically effective. Taking the same scenario as presented in the [S1 paper](https://arxiv.org/abs/2501.19393) as an example: when a large amount of supervised data has been collected, how can the authors' method be applied to select samples for constructing an optimal dataset?\n4. The logic underlying the paper’s proposed method for constructing SFT data is flawed. The paper only empirically shows that SFT data with more cycles and larger diameters in the reasoning graph tend to yield better performance. However, the reverse does not necessarily hold: SFT data with more cycles and larger diameters do not inherently lead to better performance. Intuitively, data that reflects overthinking or unnecessarily long reasoning paths may result in graphs with more cycles and larger diameters, but such data does not guarantee improved outcomes. As noted in the S1 paper, selecting data with the longest reasoning traces did not lead to optimal performance.\n5. The attribution of redundant cycles to overthinking (Lines 296-298) conflicts with the empirical analysis in Section 4.4, which conclusively traces their origin to language-mixing effects.\n6. The paper has many minor flaws that may affect the reader's impression:\n    * There is a mismatch between the question and the corresponding response in Figure 1, that is, the generated answers do not address the displayed question.\n    * There is a mixed use of notations, such as $K$ in lines 118, 152, and 182, which may lead to reader confusion/misinterpretation.\n    * The quality of Figure 3 may hinder proper interpretation, as it is not provided in vector format, resulting in potential pixelation or blurring when scaled.\n    * There is an inconsistency between the textual description (Line 641) and Figure 14's caption: The text states the model was fine-tuned using \"100 and 500 steps\". However, Figure 14's sublabels show results for \"200 steps\" and \"400 steps\".\n    * There appears to be an issue with invalid citations in Lines 856 and 886, where the references are displayed as question marks (\"?\").\n\nQUESTIONS:\n1. The authors should conduct additional experiments beyond mathematical reasoning datasets to demonstrate the broader applicability of their findings. (W1)\n2. The authors must resolve the issues stated in W2-W4.\n3. The authors should explain why they chose the hidden states from the layer of 90% depth as the default rather than the last layer. The hidden states of the final layer are used to generate responses, which seems significantly more reasonable.\n4. The authors could demonstrate the statistics (e.g., average number of nodes and edges) of the reasoning graphs for each dataset to help readers better understand the changes in graph properties.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

comments: SUMMARY:
The paper analyzes the reasoning steps of CoT LLMs as graph, and find interesting correlations between graph properties and the reasoning capability of the LLMs.

STRENGTHS AND WEAKNESSES:
Strength:
1. This is a interesting perspective to view each type of reasoning steps as node and analyze the graph patterns between reasoning node.   The graph properties shows strong correlation to the reasoning capability of the Qwen model family.

Weakness:
1. The paper is a bit hard to read with some grammar issues. For example in the first paragraph "These recent reasoning models are characterized with to think and reason for longer before responding". Also the choice of math notations are also not helping the readers to better understand the paper. For example, the author used "T" to define the number of reasoning steps, then the readers will naturally expect  to use "t" to index the step, but then the author used "t" to index the token within each step. Same for "l" and "L". In general I find much can be improved in the writing.
2. For the clustering of segments, there are many ad hoc choices that are not explained well. For example, the choice of K=200. Also the authors only showed 3 representative centroids among the 200. Can the authors add the others in the Appendix? I am curious if the other centroid make sense.
3. It seems the experimental results are restricted to the Qwen family of models. The results will be stronger if more model families can be included.

QUESTIONS:
None

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the concept of reasoning graphs to analyze the internal mechanisms of large reasoning models. The paper systematically analyzes key graph-theoretic properties such as cyclicity, diameter, and small-world index across multiple tasks to provide insights into the success of reasoning models. The empirical findings suggest that large reasoning models exhibit unique structural properties in their reasoning graphs, which correlate with enhanced performance. The paper also reviews related works and discusses the implications of these findings for understanding large reasoning models and improving SFT-Data construction.\n\nSTRENGTHS AND WEAKNESSES:\n## Paper Strength\n1. Innovative Methodology: The introduction of reasoning graphs to analyze large reasoning models is innovative and provides a new perspective on understanding their internal mechanisms.\n2. Insightful Empirical Findings: The paper offers insightful empirical findings on the structural properties of reasoning graphs in large reasoning models and how they correlate with enhanced performance.\n3. Comprehensive Analysis: The paper systematically analyzes key graph-theoretic properties across multiple tasks, providing a comprehensive understanding of the reasoning processes in large reasoning models.\n\n## Paper Weakness\n1. Lack of Implementation Details: The construction of reasoning graphs needs further clarification. Are the nodes in the graphs constructed separately for each evaluated LLMs? or are they pre-constructed from the outputs of a single LLM such as Qwen2.5-32B-Instruct? If so, would it lead to a bias towards similar large reasoning models, as they tend to generate similar outputs? Besides, when evaluating the reasoning graphs in different layers, would the nodes in the graphs be constructed from the outputs of the same layer of the LLMs? These details are crucial for judging the validity of the findings in the paper.\n\n2. Concerns regarding the reasoning graphs: Are the reasoning graphs really reflecting the true reasoning process of large reasoning models or just showing the statistics for characteristics of the output tokens? Because the reasoning step is broken down by empirically checking the `\\n` and grouping the hidden states of output tokens. Each node in the reasoning graph is a composition of reasoning steps with similar semantics. **Would this really reflect the internal reasoning process of large reasoning models or just a statistical analysis of the output tokens?** For example, the paper mentions that the reasoning graphs exhibit cycles, which could be related to the repetition of the `wait` token in the LLMs outputs. \n\nConsidering the recent works [1] argue that the intermediate tokens in the reasoning process are just some statistically generated tokens and lack any real semantic content or algorithmic meaning (e.g., language mixing in DeepSeek-R1-zero). They could not be treated as reasoning traces of LLMs. This further raises my concerns about the validity of the reasoning graphs in the paper. \n\n3. Implications for Reasoning SFT-Data Construction: The implications for reasoning SFT-Data construction are not well justified. The experiments in Section 5 only show that the model trained with the s1-v1.1 data can generate larger diameter reasoning graphs. This just reflects the reasoning patterns of the finetuned LLMs and is not a direct indication of the quality of the SFT-Data. Can we apply the proposed method to construct the reasoning graph offline and then analyze the graph characteristics to directly judge the quality of the SFT-Data? A naive solution I can think of is to feed the SFT-Data into the forward process of the LLMs in a teaching-forcing manner and extract the hidden states to construct the reasoning graph. By analyzing the cycles and the diameter of the reasoning graph, we can judge the quality of the SFT-Data to see if it is suitable for training the LLMs (i.e., more cycles and a larger diameter are better).\n\n[1] Kambhampati, Subbarao, et al. \"Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!.\" arXiv preprint arXiv:2504.09762 (2025).\n\nQUESTIONS:\n1. Can you provide more details on the construction of reasoning graphs, especially regarding how the nodes are constructed and whether they are biased towards specific large reasoning models?\n2. How do you justify the validity of the reasoning graphs in reflecting the true reasoning process of large reasoning models?\n3. Can you elaborate on the implications for reasoning SFT-Data construction and how the proposed method can be used to directly judge the quality of SFT-Data?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an interpretability method that extracts reasoning graphs from reasoning traces generated by large language models when solving complex tasks. These graphs are created by clustering the hidden state representations of models at each intermediate reasoning step when solving problems across an evaluation set. The centroids of these clusters are nodes, and transitions between reasoning steps for a given solution define the directed edges in the graph. The authors compare graph-theoretic properties of graphs constructed from traces taken from reasoning models and their non-reasoning, “base” counterparts to show how these metrics correlate with reasoning ability. They claim that this method illuminates how reasoning models approach complex problems, and they argue that these metrics can be used to build better reasoning datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The authors test their method across a wide range of model sizes and evaluation datasets. They sweep across different hyperparameters like k-values for clustering and the layer at which hidden state representations are extracted, showing that relationships presented are not artifacts of cherry picked graph construction. Additionally, the authors demonstrate how these metrics change at different depths within the model.\n* The experiments are comprehensive and thoroughly demonstrate how different graph properties correlate with reasoning performance at different model sizes and target layers.  The examination of the nodes in the graphs, along with example reasoning steps, was very helpful for understanding what the reasoning graph might look like.\n* One question while reading the paper is whether increases in these metrics are merely correlated with superior reasoning or whether reasoning models actually learn to leverage relevant techniques that are measured by these graph properties. The intervention study of training on s1 and iteratively measuring diameter distribution helps answer this question. Additionally, the comparison between s1 v1 and s1 v1.1 demonstrates that graph metrics not only increase with supervised finetuning on reasoning data but that they increase more on better data, supporting the authors’ claim that this method can be used for dataset curation.\n\nWeaknesses:\n* One possible confound is that these graph metrics may increase with longer reasoning chains in general. An experiment controlling for token length/flops is needed.\n* The reasoning steps are segmented at newlines. Some models may have a propensity to generate longer sequences. Is comparing the graphs generated from these models fair?\n* Figures 7b and 7c show a massive increase in AIME24 accuracy between Distill Qwen-7B and Qwen-14B. However, the cycle counts are similar between the two, and graph diameter is noticeably larger for Qwen-7B. Is there any explanation for why this massive increase in accuracy is not correlated with these metrics?\n* Figure 3 demonstrates how DeepSeek-R1-Distill Qwen 32B exhibits more exploration and cyclicity in its reasoning graph compared to the corresponding base model. Were the examples presented in this figure solved by both models? If not, are all the nodes explored by the reasoning model essential to solving the answer. Recent papers like [1] claim that reinforcement learning with verifiable rewards improves pass@k at small k but not at large k, suggesting that RLVR does not expand the coverage of the base model. Do you think that overlaying graphs from multiple attempts from a base model would results in similar graph coverage to that of the reasoning model? Would studying the trajectory of the reasoning model in these problems help confirm or disprove the claim above?\n* Additional insight into what the cycles accomplish would be helpful. The discussion around figure 7a claims that cyclic reasoning generally enhances reasoning effectiveness except in cases like language mixing. The authors later state in section 6 that “redundant cyclic structures may explain overthinking.” Is there any way to differentiate between trivial and non-trivial cycles that harm or help reasoning performance? In addition, if the nodes are tied to general math operations as suggested in figure 2, would something relatively simple like adding and multiplying numbers in sequence manifest as a cycle?\n\n\n[1] [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?\n](https://arxiv.org/abs/2504.13837)\n\nQUESTIONS:\n(See questions above)\n\n1. Figure 2 lists a small set of nodes taken from the reasoning graph as well as representative example reasoning steps for each node. The three nodes included are labelled “multiplicative,” “additive,” and “wait.” Are the nodes in the reasoning graphs studied in the paper always directly tied to concrete reasoning techniques? If we set K=200, then are all 200 nodes clearly associated with these techniques?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}