Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts
Authors: Ammar Ahmed
Azal Ahmad Khan
Ayaan Ahmad
Sheng Di
Zirui Liu
Ali Anwar
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient Reasoning
Large Reasoning Models
Retrieval Augmented Language Models
Primary area: foundation or frontier models, including LLMs
TL;DR: Retrieval-of-Thought (RoT) improves LLM reasoning efficiency by reusing prior reasoning steps as dynamic templates, cutting tokens, cost, and latency while preserving accuracy.

Abstract: Large reasoning models improve accuracy by producing long reasoning traces, but this inflates latency and cost, motivating inference-time efficiency. We propose Retrieval-of-Thought (RoT), which reuses prior reasoning as composable ``thought" steps to guide new problems. RoT organizes steps into a thought graph with sequential and semantic edges to enable fast retrieval and flexible recombination. At inference, RoT retrieves query-relevant nodes and applies reward-guided traversal to assemble a problem-specific template that guides generation. This dynamic template reuse reduces redundant exploration and, therefore, reduces output tokens while preserving accuracy. We evaluate RoT on reasoning benchmarks with multiple models, measuring accuracy, token usage, latency, and memory overhead. Findings show small prompt growth but substantial efficiency gains, with RoT reducing output tokens by up to 40%, inference latency by 82%, and cost by 59% while maintaining accuracy. RoT establishes a scalable paradigm for efficient LRM reasoning via dynamic template construction through retrieval.

Published as a conference paper at ICLR 2026
RETRIEVAL-OF-THOUGII VIA REUSING THOUGHTS
EFFICIENT REASONING
Ammar A Azal Ahmad Khan',, Ayaan Ahmad', Sheng Di Zirui Liu', Ali Anwar
Department of Computer Science and Engineering, University of Minnesota
2 *Department of Computer Science and Engineering, University of California Santa Cruz
Argonne National Laboratory
* Equal Contribution.
aahme khan1069, zrliu, aanwar @umn. edu ay ayahmad@ucsc edu, sdil@anl, gov
ABSTRACT
Large reasoning models improve accuracy by producing long reasoning traces, but this inflates latency and cost, motivating inference-time efficiency. We pro- pose Retrieval-of-T ThoTuhht (RoT), which reuses prior reasoning as composable problems. RoT organizes steps into thought graph
"thought" steps to guide with sequential and semantic edges to enable fast retrieval and flexible recombina- tion. At inference, RoT retrieves query- melevate nodes and applies eeward-guided traversal to assemble a problem- sppccffcc template that guides generation. This dy- namic template reuse reduces redundant exploration and, therefore, reduces output tokens while preserving accuracy. We evaluate RoT on reasoning benchmarks with multiple models, measuring accuracy, token usage, latency, and memory overhead. Findings show small prompt growth but substantial efficiency gains, with RoT reducing output tokens by up 40%, inference latency by 82%, and cost by 59% while maintaining accuracy. RoT establishes scalable paradigm for efficient LRM reasoning via dynamic template construction through retrieval. The code for RoT is available at https//ggiibbboommmmmmf9999e Thhouhh
INTRODUCTION
Large Reasoning Models (LRMs) have demonstrated impressive capabilities in solving complex tasks by producing outputs accompanied by detailed reasoning trajectories (Xu et al., 2025a). Pro- prietary models such as OpenAl's ol, o3, and 04 (Jaech et al., 2024), Google's Gemini 2.5 (Comanici et al., 2025), and Anthropic's 's Claude Opus 4, as well as open-source models like Qwen-QwQ (Yang et al., 2025a) and DeepSeek-PR RI (Guo et al., 2025), exemplify this trend. These models adopt in- tentionally slower and more deliberative inference process, mimicking human-like reasoning. This approach typically involves generating longer outputs and consuming increased inference-time com- pute to effectively address reasoning-intensii queries.
Recent efforts improve reasoning in LLMs have primarily focused generating more output to- kens to simulate thoughtful, multi-step reasoning (Snell et al., 2024). A common approach involves guiding generation using external reward models Zhang et al. (2024). These include outcome-basse reward models, such as Best-of-N (BoN) sampling. Other methods employ process-based reward models that supervise intermediate reasoning steps through search strategies like beam search or Monte Carlo Tree Search (MCTS) Xie et al. (2024); Zhang et al. (2025b). More recently, rein- forcement learning bRse)-based techniques, such as Group Relative Policy Optimization (GRPO), have been explored to train models toward trajectories that yield high reasoning quality (Shao et al., 2024). While increasing test-time compute has improved reasoning performance, this often relies on generating longer outputs, which introduces key inefficiencies (Sui et al., 2025; Feng et al., 2025). First, generating longer outputs inherently increases latency, as each token must be decoded se- quentially. Second, API-based model providers typically price output tokens 2-5x higher than input tokens, making such methods cost- cost- post-prohibitive at scale. These limitations highlight the need for more efficient reasoning techniques that retain performance while reducing generation cost and latency.

---

Published as a conference paper at ICLR 2026
Chaln-of-Thought
Wrrong Fath
Reasoning Question
LRM
Winnng Path
Answer
Righn
Retrieval of Thought
Thought Graph w/ Intaal Node
Reward Guided Traversal
Retrieved Templite
Retrieval of Intial
Reasoning Question
LRM
Figure 1: The figure contrasts Cfain-of-Thowee Thought (CoT) inference in LRMs with our Retrieval- of-Thought (RoT) approach. In CoT (top), models sequentially explore multiple wrong paths, causing inefficiency and high token usage. RoT (bottom) builds on structured thought graph where reasoning steps are stored as nodes. First, RoT retrieves relevant nodes and performs reward- guided traversal to assemble a problem-specific template, reducing redundant exploration and directing the model toward correct reasoning. This yields more efficient inference and fewer tokens.
To enable more efficient reasoning without excessive token generation, promising direction is to reuse prior knowledge in the form of thought templates. Recent work has pursued this via retrieval- based reasoning, Buffer of Thought (BoT; Yang al. (2024a)) and SuperCorrect (Yang et al., 2024b) augments this with both coarse and fine-gen lineggranedd templates to bolster smaller LLMs. However, these approaches operate largely with static templates fixed before generation, limiting their ability to adapt or compose new reasoning pathways on the fly. In contrast, human problem solving relies not only on recalling past experiences but also on "connecting the dots", recombining fragments of prior solutions into novel configurations that make knowledge broadly reusable (Gick & Holyoak, 1980). Translating this to LLMs, existing methods lack the ability to dynamically assemble templates across reason
problems, leaving them unable exploit the kind of flexible memory that humans efficiently. To address this gap, we propose RoT, a framework that equips LRMs with a structured memory of individual thoughts and enables the dynamic construction of new templates through knowledge graph, thereby supporting more efficient and generalizable reasoning.
The foundation of RoT is built upon three key observations related performance optimization. (01) Queries within given dataset often exhibit recurring patterns in the underlying reasoning or thoughts used solve them. (02) Retrieving relevant information from large vector database is significantly faster than generating text from a LM. (03) When presented with a correct reasoning path or solution a similar query, language models can solve new queries more efficiently, requiring fewer tokens to reach correct or complete answer. Yet fully exploiting these observations exposes a core challenge to design system that can dynamically generate reasoning templates during in- ference by observing the input, rather than reusing fixed scaffolds. This requires not only storing a rich repository of individual reasoning thoughts but also developing mechanisms to identify, re- trieve, and synthesize the most relevant patterns into coherent, query-specific templates. Unlike prior approaches, RoT aims to provide LRMs with the ability to construct contextually appropriate reasoning scaffolds on-demand by flexibly combining and adapting prior thoughts, thereby enabling efficient, reusable, and generalizable reasoning.
As shown in Figure 1, introduce RoT's novel architecture, which organizes reasoning steps into a thought graph. This graph leverages metadata-base retrieval to quickly narrow the search space and then applies reward-guided traversal algorithms dynamically assemble thought templates by exploring interconnected reasoning patterns. The resulting templates are generated at inference
'Throughout this paper, we use the term thought interchangeably with reasoning step, referring the individual steps used salve reasoning problem.
2

---

Published as a conference paper at ICLR 2026
time, ensuring both contextual relevance to the query and computational efficiency. We further demonstrate how these templates can be applied to guide reasoning, reducing token generation and establishing RoT more efficient framework for LRM problem solving.
In CoT method LRMs frequently explore through multiple incorrect reasoning paths before con- verging, whereas RoT uses the retrieved template a guide that anchors the model to promising reasoning paths. This substantially reduces path switching, enabling the model to reach the correct solution with fewer detours. As result, RoT lowers the number of output tokens generated, directly translating to improved efficiency in both cost and latency.
Our work makes three key contributions to efficient inference in reasoning models. First, we pro- pose a new paradigm for dynamic template construction, introducing the first framework that equips LRMs with structured memory of reasoning and dynamically assembles context-specific tem- plates at inference time, unlike prior methods that rely on static templates. Second, we design thought graph with a heward-guided traversal algorithm that organizes reasoning traces and effi ciently retrieves and composes templates while preserving contextual relevance. Finally, through comprehensive experiments and ablations across AIME and AMC benchmarks, we show that RoT consistently achieves comparable higher accuracy than Chain-of-Thouu while reducing output tokens by up to 20-40% and cutting inference latency and cost substantially. These results establish RoT as a scalable and efficient solution for LRM reasoning.
and retrieval baselines,
RELATED WORKS
Test-Time Scaling. Scaling test-time compute has emerged as a key strategy for improving rea- soning, with methods such as Best-of-N sampling, beam search, and more recently RL approaches like GRPO enabling models to generate longer reasoning traces and improve accuracy (Snell al., 2024; Yao et al., 2023; Guo et al., 2025). While effective, these methods come at the expense of efficiency as longer outputs substantially increases latency and cost, motivating recent work on inference-ting efficiency through adaptive computation, parallel decoding, and early termination strategies (Pan et al., 2025; Chen et al., 2025; Fang et al., 2025). Together, these efforts highlight a growing demand for techniques that preserve reasoning performance while reducing computation.
Retrieval-based Reasoning. Retrieval-based methods represent complementary direction, in- jecting external knowledge or pre-computed reasoning to guide inference. Prior approaches such as BoT (Yang al., 2024a), SuperCorrect (Yang et al., 2024b), and RAT (Wang et al., 2024) have shown that reusing thought templates can improve reasoning, though they typically rely on static scaffolds distilled from larger teacher models. In contrast, work targets caasoningccapable LMs directly and introduces a dynamic alternative: decomposing prior solutions into reusable thought templates inference time. This fine-grained, on-demand
steps and assembling problem-specine reuse enables greater adaptability than static retrieval, establishing a new paradigm for efficient rea- soning. A complete discussion of related work is provided in Appendix A.
MOTIVATION
33.7%
-8.2%
Dreeration Somer
539
-2.%%
WAbrEO 0222
332
Thought Sinitarit (N)
Dalaseda
Figure 2: Key observations motivating the RoT framework. (Left) Semantic similarity between Thought Store and steps to solve reasoning datasets. (Middle) Comparison of retrieval and genera- tion showing retrieval is faster than generation. (Right) Thought Caching yields token savings.
Our framework builds three central observations about reasoning efficiency.
3

---

Published as a conference paper at ICLR 2026
(01) Thoughts required to solve problems across same domain highly similar. Problems in reasoning datasets often require overlapping steps. For instance, many AIME or AMC questions share algebraic transformations or simplification patters.. When analyze solution traces, that the same types of steps appear again and again, as shown in Figure (left). This suggests that a identify and store these thoughts.
large portion of reasoning can be reused if
(02) Retrieval is faster than generation. Retrieving information from a vector database is much faster than asking language model generate new text. While generating tokens requires sequen- tial decoding on GPUs, retrieval simply fetches already-stored thoughts from memory. As shown Figure 2 (middle), fetching from a large repository takes only a few milliseconds, whereas gen- erating even a short piece of text takes seconds. Since output tokens are also several times more expensive than inputs, retrieval is both cheaper and faster.
(03) Reusing cached steps reduces tokens. When correct steps are cached and reused, the number of tokens that the model must generate decreases substantially, as shown in Figure 2 (right). This is important because output tokens are both slower and more costly than input tokens. While adding retrieved steps slightly increases the prompt length, the savings in output tokens are much larger. As a result, reuse improves both cost and latency on datasets where problems share similar reasoning.
Together, these observations show that many reasoning steps are repeatable, retrieval significantly more efficient than generation, and caching directly cuts down token usage. This motivates our design of a system that can store and reuse thoughts to make reasoning more efficient. Full experimental details supporting these observations provided in Appendix B.
4
RETREEVA--OF-THOUG
4.1 PRELIMINARIES
Before introducing our RoT framework, we formalize the key objects used throughout the paper: templates and reasoning steps, the graph structure that encodes their relationships, and the procedure for constructing semantic connections between steps. These preliminaries establish the foundation for representing and organizing reusable reasoning behaviors across problem-solving trajectories.
Definition 1 (Templatees and Steps). Let D denote finite set of problem- templates. Each template & D is an ordered sequence of reasoning steps St = (s s1: .......,, .); where le E N is the template length. For index E (0....:4t11) the pair (1, denotes the i-th step of template t. Definition 2 (Thought Graph). The structure of reusable reasoning steps is encoded directed, weighted, disconnected multi graph
G (V, ESequential U Esemonticc w)
where V the set of nodes, & S ((11,12))(11 ([(([11,,) U2) V}. : E - R is weight function. Each node it E V corresponds to a reasoning step from template t at index i. Specifically, the set { (1, TE 0
(1)
The edge set E is partitioned into two disjoint types:
Esequenial U Eeemantic-
Sequential Edges. The set Esequentil contains the sequential edges, which capture the within- template ordering of steps. For every t E 'D and every pair of adjacent steps (i, i+1) within it, we include a directed edge:
Esequential
{ ((.,).(() te
Semantic Edges. The set Esemantic contains the semantic edges, which connect semantically anal- ogous steps across different steps and templates. These edges are defined subset:
Esemantic ((t, (t' ()) E V x v: (t, * (t'j)), sim(u, Ung) 2 7
where u4 is the embedding of node (t, i). (0, 1) is similarity threshold. and sim is malized similarity measure (defined below). Semantic edges are treated as bidirectional: if (t. connects to (f, then (L'j) also connects to i).
4

---

Published as a conference paper at ICLR 2026
Construction of Semantic Edges. To construct semantic edges, begin by measuring the sim- ilarity between node embeddings using cosine similarity. All embeddings i are &2 n-normalized so that cosine similarity reduces to the dot product (U,i j) and rescale the result to lie within [0, 1] using sim(a, b) sim(a.b)+1 The similarity threshold T E (0, 1) decide which node pairs are semantically similar enough be connected and use 7 0.85 for our study. Specifically, include a semantic edge between two distinct nodes (t, and (t', (t', their normalized similarity score exceeds the threshold:
2
To ensure bidirectional connectivity, we insert both directions of each edge: whenever a pair iL, (t) satisfies the similarity condition, both ((t,i), (t) and ((t,.),, (t) are added to the graph, each with the same weight.
Edge Weights. The edge weight function w E is defined as
if ((t, i), (f1,i) E Esequential:
((t, sim je. if ((t, i), (f%.j) E &semantic
In practice, to make this process scalable and efficient, precompute and cache all embeddings.
4.2 THOUGHT GRAPH CONSTRUCTION
We construct a directed thought graph that captures problem-solving thoughts from curated set of mathematical reasoning templates. Each node corresponds to a single reasoning thought. Nodes contain metadata such as the template type and knowledge tags along with the reasoning text. Our graph is built from 3.34k templates drawn from the ReasonFlux-v2 dataset We manually verified that none of the templates overlaps with benchmarks used in evaluation, to ensure fair assessment.
The graph contains two types of edges. Sequential edges connect consecutive steps within the same template, preserving the natural logical flow. Semantic edges connect analogous steps across templates, supporting cross-template knowledge transfer. To establish semantic edges, we generate step embeddings using the jina-embedd ings nggnnngsg -vas-vv2--- 1122smalll semall-ce ln-en model (Günther et al., 2023) and compute cosine similarity. All embeddings are pre-computed and cached for efficiency, with data sanitization ensuring XML-safe graph serialization.
Given a reasoning query from the user and a pre-constructed thought graph, the RoT framework operates three-step process: (1) retrieval of the initial node, (2) reward guided-traversal, and (3) template integration into the model. These steps together enable RoT to dynamically assemble contextually relevant reasoning templates from prior knowledge while preserving logical flow. We discuss each step in detail in the following subsections. The overall procedure is summarized in Algorithm 1, and discuss each step in detail in the following subsections.
4.3 RETRIEVAL OF THE INITIAL NODE
The first step in constructing a reasoning template is to select an appropriate entry node from the thought graph. This is performed through multi-stage filtering and scoring procedure.
Filtering. Candidate nodes are first restricted by metadata: only nodes whose template type matches the problem class (e.g., algebraic or geometric) are considered. Knowledge tags are then used to further refine the candidate set, retaining only nodes annotated with domain-specificc concepts such as geometry, calculus, or number theory. Finally, validity checks ensure that each candidate node possesses pre-computed embedding, guaranteeing consistency in similarity evaluation.
In our experiments, we manually annotated metadata tags (e.g., algebraic, geometric) for AIME and AMC problems. While this process could be automated using smaller encoder-only models such as BERT, well known for their effectiveness in text classification (Peng et al., 2021; Piçkos et al., 2021), we did not train a specialized classifier here, since the goal of this work not to propose new tagging model but to demonstrate the impact of retrieval and traversal given such tags.
"The full version of the ReasonFlux-v2 dataset can he accessed using the link: https:// huggingface,
www.e/ Reason 1o----
5

---

Published as a conference paper at ICLR 2026
Algorithm RRetiieal-of--houghh (RoT) Inference
1: Input: Query Q: Thought Graph (V, Esequantal U Semaniic w); Reasoning Model M
2: System executes:
3: Run metadata and knowledge-tag filtering V to obtain candidate set of nodes.
4: Compute first-step reward Initial (Equation 1); select Vo = arg max Riitial
5: Initialize template T [co]
6: while termination condition is not met do
7: Evaluate neighbors of current node using (Equation 2)
8: Select v* arg max R
9: Append v to T
10: end while
11: Construct prompt P = (2, T) with <think> tags
12: Call M API with P: f M(P)
13: return y
Scoring. The initial node is selected by maximizing reward function that combines semantic relevance and structural validity:
Rinitial Ro + (1 -1-08 (1)a) a) Rs.
(1)
where Ro is the cosine similarity between the problem query and the node embedding, and Rs is structural indicator reward defined as
(1, if ((.)) E EV, = 0,
Rs
(o,
0. otherwise.
This criterion encourages the selection of nodes that are both semantically aligned with the query and structurally consistent valid entry points. In implementation, we a 0.8 prioritize semantic relevance while retaining a bias toward legitimate step-0 nodes.
4.4 REWARD-GUIDER TRAVERSAL
After initialization, reasoning template expanded through iterative graph traversal. At each step, candidate neighbors of the current node are evaluated by reward function that balances se- mantic alignment with sequential flow.
Graph Traversal.
The traversal reward is defined
R Rg + RF
where Ro denotes the reward given by
relevance of the candidate node, and Rp is a structural flow
RF = {1
if ((t, (t,i+)) Esequential,
otherwise.
The traversal reward R assign equal weight to semantic alignment and structural consistency. Weighting factors can be introduced to adjust the traversal to emphasize either query-semantic rele- vance or structural flow.
Template Termination. Traversal halts when template expansion is longer beneficial. Termi- nation is also defined through a reward-based criterion, where continuation penalized if quality or structural limits violated: (max(R)) < T) V (Temmllte Imax) V (NCandidates 0) where Template the current template length, Imax = is the maximum allowed length, and Ncandidanes is the number of valid candidate nodes. This termination reward ensures that traversal halts when relevance falls below threshold, templates grow too long. or no candidates remain, thereby preventing unnecessary expansion and maintaining reasoning quality.
The specific parameter choices for thresholds and weighting factors are justified through sensitivity analysis in Appendix C.

---

Published as a conference paper at ICLR 2026
4.5 TEMPLATE INTEGRATION
Once a template is retrieved, it is directly inserted into the prompt without additional processing. Prior work has shown that reasoning models often fail reliably follow explicit instructions (Fu et al., 2025; Li et al., 2025a). To ensure that the model adheres to the retrieved reasoning during its internal deliberation, we place the template inside the <think> and <\think> tags, as introduced by Thinking Intervention (TI; Wu et al. (2025)). Importantly, we did not finetune the model specifi- cally for template adherence, as such finetuning could risk catastrophic forgetting of core reasoning abilities. Empirical validation in Section 5 demonstrates that this simple integration is effective and usable in guiding the model's reasoning process.
EVALUATION
5.1 EVALUATION SETUP
We evaluate our approach on four mathematical reasoning benchmarks: AIME 2025, AIME 2024, AIME 2023, and AMC 2023, following the standard evaluation for reasoning techniques (Zhang al., 2024; Snell et al., 2024; Yang et al., 2025b). We focus exclusively mathematical datasets enable reusability of a single thought graph across all evaluations, taking advantage of the consistent problem-solving patterns inherent mathematics. Although our method is domain-agnostic, with potential for broader applications, we defer details on other domain reusability to Appendix B.
Our experiments use five models from the Qwen3 family: 0.6B, 1.7B, 4B, 8B, and 14B parame- ters. We choose Qwen3 for its thinking mode, which separates reasoning and output tokens using structured <think> and </think> tags. We evaluate two variants of our method: RoT, where the retrieved template is directly placed in the prompt, and RoT+TI, where the template is inserted inside the <thinkk tags following the Thinking Intervention mechanism. This distinction allows us to assess thought reuse both at the prompt level and within the model's internal reasoning pro- cess. All experiments are conducted with vLLM for efficient batch inference on NVIDIA A100 GPUs (40GB). For state-of-the-art comparison, et al., 2022), RAG (short for RAG+CoT), where we retrieve static templates same as those used to construct our Thought Graph combined with CoT, and BoT (Yang al., 2024a).
include CoT (Wei al., 2022), CoT-SC (Wang
CUTSCC
RAG
BOT
BoT
RoT
Qwee3:1177
Owend 48
Qweed-0B
200
Avenge Accuracy 150
Avenage Accuracy
BAG
boT
1B67
33t
100%
213%
3031
25.0
35t 311
55%
2331
250% 360
36.
25k 206x
1351
12.02000 2008 2536
Average Output
Output
Figure 3: Average accuracy versus output tokens across Qwen3 models (1.7B, 4B, 8B). Each panel reports COT, CoT-SC, RAG, BoT, RoT, and RoT+TI (star). The shaded top-left region denotes the Efficient Reasoning Zone, corresponding to higher accuracy with fewer tokens. RoT+TI consistently lies within this region, matching the accuracy of other methods while using way fewer tokens.
5.2 PERFORMANCE AND EFFICIENCY TRADE-OFES
Figure reports the trade-off between reasoning accuracy and token efficiency across Qwen3 mod- els (1.7B, 4B, 8B). We find that RoT+TI consistently achieves comparable or better accuracy than baselines while substantially reducing output tokens, positioning it inside the Efficient Reasoning Zone. On Qwen3-1.7B, RoT+TI reaches roughly 44% accuracy versus 46% for CoT, while cut- ting output length by nearly 3k tokens (8.0k vs. 11.0k). At 4B scale, RoT+TI sustains 72% accu- racy compared to 74% for CoT while lowering token usage by about 800 tokens (9.1k vs. 9.9k). Standard RoT also outperforms RAG by achieving similar accuracy with about 400 fewer tokens. On the larger 8B model, RoT+TI delivers 83% accuracy, slightly higher than RAG (80%), while using nearly 500 fewer tokens (9.6k 10.1k). Importantly, the comparison between RAG and
7

---

Published as a conference paper at ICLR 2026
RoT+TI underscores the benefits of dynamic templates: RoT+TI leverages reusable but context- sensitive structures, which generalize better than static templates in RAG or BoT, validating our hypothesis that dynamic template integration offers broader applicability when retrieving reasoning solutions. These results highlight that RoT++TI not only preserves reasoning performance but also systematically cuts down token usage across scales, demonstrating that integrating templates within <think> tags provides robust pathway to efficient reasoning. Detailed results across all models and datasets, along with input-output example, are provided in Table 7 and Appendix D.
Thought Graph Scalability Analysis. We further verified scalability by comparing RoT with a smaller subgraph of randomly sampled 0.9k templates against the full 3.34k 3.34k-template graph (Fig- ure 10). Results show that increasing the number of templates generally improves accuracy across Qwen3 models, with particularly large gains higher model scales (e.g., +17.0% Qwen3-8B). These findings underscore that RoT is highly scalable: as LLM serving platforms accumulate more user data, the resulting larger thought graphs can yield systematically higher reasoning performance.
5.3 TOKEN USAGE AND COST IMPLICATIONS
We next analyze how thought reuse affects token distribution and mon- etary cost. While RoT slightly
CoT
CoT-SC
RAG
BoT
RoT
K22
RoT
Avg cost/sample (USD)
increases the number of input to- kens due to template insertion, it consistently reduces output tokens by roughly 20%, yielding mean- ingful efficiency gains. Figure 4 reports the average per-sample in- ference cost across Qwen3 mod- els using Alibaba Cloud token pric- ing and measured input/output to- ken counts. RoT+TI delivers the largest savings at smaller scales, cutting costs by 59.0% on Qwen3- 0.6B and 39.3% on Qwen3-1.7B relative to COT. At mid-scale, the gains remain notable, with reduc- tions of 20.3% on 4B and 3.8% 8B. Even at 14B parameters, RoT+TI still achieves an 8.5% reduction. These results highlight that the efficiency benefits of thought reuse are strongest at small and medium scales but remain consistently positive across all model sizes. Full token and cost breakdowns, including pricing details, are provided in Tables and (Appendix D).
0.04
0.03
-8.5%
0.02
-3.8%)
0.01
-59.0%
-39.3%
-20.3%
0.00
Qwen3 0.68 Qwen3-1.78 Qwen3 Qwen3-88 Qwen3-144 148
Figure 4: Average per-sample inference cost (USD) across Qwen3 models, comparing CoT, Cot-SC, RAG, BoT, RoT, and RoT+TI. Costs are computed using Alibaba Cloud prices with average input/output token over AIME 2023/2024/2025 and AMC 2023. Arrows above RoT+TI indicate the percent cost reduction relative to CoT.
5.4 GENERALIZATION ACROSS MODEL FAMILIES
To test whether RoT specific to the Qwen3 family, we evaluate RoT on two additional reasoning models from differ- model families: DLER-7B Liu et al. (2025) and DeepSeek-R1-Lan B Guo al. (2025). We keep the same RoT pipeline and evaluate on two math bench- marks (AMC 2023 and AIME 2024). As shown in Table 1, RoT and RoT+TI re- main competitive across both model fam- ilies. RoT often improves accuracy some- times at the cost of longer outputs, while RoT+TI frequently recovers token effi- ciency and yields Pareto-efficient points on the ccuracy-efficience trade-off.
Model
Data
Method
Acc. (%)
Out Tok. (A va Col)
CoT'
85.00
1302 16 (-)
AMC
RoT
90.00
1595.75 (+22.5%)
RoT+I1
90.00
1077.53 (-17.3%)
DLER-7B
CoT
37.04
2167.78 (-)
AIME'24
RoT
37.04
2434.22 (+12.3%)
RoT+TI
51.85
2008.33 (-7.3%)
CoT
77.50
4372.45
AMC
RoT
87.50
5419.40 (+23.998)
R1-L.B
RoT+TI
72.50
2949.72 (-32.6%)
Col'
40.74
9582.00 00 (-)
AIME'24
RoT
51.85
9397.11 (-1.9%)
RoT+II
40.74
8647.00 (-9.8%
Table 1: Generalization across model families AMC'23 and AIME 24. DeepSeek-RI-Llamase
DS-RI-L8B abbreviates
8

---

Published as a conference paper at ICLR 2026
5.5 GENERALIZATION BEYOND MATHEMATICS
While our main evaluation emphasizes math benchmarks for maximal reuse of single thought graph, we also evaluate RoT on non-math, sci- entific reasoning benchmark: GPQA. We con- struct a scientific thought graph from GPQlestyle explanations and apply the same RoT procedure. Table 2 shows that RoT remains effective outside mathematics for mid/large models. RoT matches or improves accuracy while reducing output to- kens, while RoT+TI yields particularly large to- ken savings achieving up ~80, highlight- ing that the efficiency benefits extend to domains where reasoning is less formulaic than math. This setting stresses whether RoT can reuse ab- stract reasoning moves (e.g., selecting relevant evidence, eliminating distractors, and composing a structured explanation) rather than relying on repeated algebraic patterns. Overall, these results suggest RoT is not inherently tied to mathemati- cal formalism, but instead leverages transferable reasoning structure across domains.
Model
Method
Acc. (%)
Out Tok. (A vs CoT)
CoT
27.94
4039.23 (-)
Qwen3-0.6B
RoT
29.41
4158.04 (+2.96)
RoT+TI
20.59
797.34 (-80.3%)
CoT
30.23
$615.28 (-)
Qwen3-1.7B
RoT
25.58
5496.93 (-2.1%)
RoT+TI
30.23
1448.26 (-74.2%)
Coll
58.21
5443.16 (-)
Qwen3
RoT
61.19
4773,43 (-12.3%)
ReT+11
53.73
3021.03 (-44.5%)
Col
53.97
5169.84 (-)
Qwen3-88
RoT
57.14
4165.19 (-19,4%)
RoT+TI
47.62
1646.27 (-68.2%)
CoT
57.69
3999.00 (-)
Qwen3-14B
RcT
59.62
3336.25 (-16.6%)
RoT+TI
42.31
1954.94 (-51.1 1%)
Table 2:
Experimental results of RoT and RoT+TI on GPQA scientific-reasoning results.
5.6 ROBUSTNESS OF ROT ON
VARIANTS
A potential concern for retrieval-based reason- ing is confounding under near-duplicate inputs: two problems may share the same structure but differ in numerical constants, raising the possi- bility that retrieval could inject incorrect values. RoT avoids this failure mode because nodes in the thought graph encode abstract reasoning op- erations (e.g., "apply substitution" or "reduce to a standard form"") rather than copying concrete nu- meric instantiations. To stress-test this setting, we construct a new near- cear-duplicate dataset by gener- ating structurally identical AIME and AMC vari- ants via systematic changes to constants/bounds, build a thought graph using only these variants, and evaluate RoT on the original problems from AIME and AMC. Table shows that RoT remains robust and RoT+TI provides substantial token reductions, demonstrating that RoT benefits further when reasoning structure repeats across queries.
Model
Method
Acc. (%)
Tok. (A CoT)
CoT
24.24
7916.44 (-)
Qwen3-0.6B
RoT
31.82
7827.91 (-1.1%
RoT+I1
21.21
3384.47 (-57.2%)
CoT
59.09
7747.24
Qwen3-1.78
RoT
66.67
7131.58 (-7.9%
RoT+TI
50.00
4991 (-35.6%)
Coll
86.36
6754.41 (-)
Qwen3-4B 4B
ReTT
83.33
6818.22 23 (-0 9%)
RoT+TI
80.30
5087.35 (-24.7%)
CoT
77.27
7368.00 0388,0336 (-)
Qwen3-818
RoT
89.39
7371.17 (+0.04%)
RoT+TI
77.27
5828.44 (-20.9%)
Table 3:
Experimental results of testing RoT with Nepr-duplicate AIME/AMC questions.
5.7 PATI SWITCHING ANALYSIS
How does RoT reduce unnecessary exploration during reasoning? We measure this effect by analyzing path switching, defined as the number of times a model aban- dons one reasoning trajectory and begins another within a single response. In practice, we approximate path switches by counting discourse markers that typically indicate a change of direction in reasoning, such as "however". "a selteernaiiee ivel",, or Minstead". Figure 5 reports the average number of path switches across Qwen3 models. We find that CoT frequently re- vises its trajectory, leading to longer outputs and wasted tokens. By contrast, RoT+TI reduces path switching by up to 81.8% (Qwen3-0.6B) and consistently lowers it across larger models as well. These results confirm that complate-guided reasoning anchors the model promising trajectories, minimizing detours and thereby improving token efficiency.
-8..0%
+14.1%)
Awarage Path Switches
25
(:47.0%
(233.15)
20
1.5
2.0
0.0
ONA015-00 20.00
O.ord-1.75
Gwa15455
Owe-5 50
Ower5-145
Figure 5: Average path switches across Qwen3 models comparing CoT with RoT+TI. RoT+TI consistently reduces unnecessary path exploration, achieving up to 81.8% fewer switches.

---

Published as a conference paper at ICLR 2026
5.8 LATENCY AND RETRIEVAL OVERHEAD ANALYSIS
We further examine the end-to-end latency of RoT, decomposing the cost into three components: re-
trieval and traversal, prefill, and decoding. While introducing re- trieval incurs small constant over- head, the overall effect is strongly positive, as reduced output tokens lead to significantly faster decod- ing. Figure 6 shows average per- sample latency across Qwen3 mod- els. RoT+TI achieves the largest re- ductions smaller scales, with la- tency drops of 72.9% on Qwen3- 0.6B and 29.7% on Qwen3-1.7B compared to COT. At mid-scale, la- tency decreases by 15.6% on 4B and 1 11.6% on 8B, while even at 14B, RoT+TI still yields an 8.5% reduction. These results highlight that retrieval-guided reasoning scales efficiently: although retrieval adds minor front-end cost, the savings from reduced decoding dominate, yielding consistent latency improvements across all model sizes. detailed breakdown of retrieval and traversal overheads is provided in Appendix D.
20000
Avg E2E latancy
15000
10000
5000
-72.9%
-29.7%
-1.6%
-11.6%
+8.5%
Bween--..BQwenn--1.. Owen3-48 Owen3-88 Owen3-148
Figure 6: Average end-to-end latency (seconds) per sample across Qwen3 models, comparing CoT, Cot-SC, RAG, BoT, RoT, and RoT+TI. Arrows above RoT+TI indicate the percent latency reduction relative to COT. RoT+TI achieves large gains on small and medium models (upto 82% reduction).
Retrieval Overhead. To quantify retrieval cost, measured the latency of retrieval and traversal. As reported in Table 8 (Appendix D), retrieval takes on average only 0.038 per query, with all models falling in the narrow range of 0.034 0.044.. This overhead is negligible relative to the time spent on decoding, and thus does not contribute to total latency. Instead. the substantial reductions output tokens dominate, making RoT+TI consistently faster than baseline methods.
DISCUSSION
How much GPU memory overhead does the RoT have? On an NVIDIA A100 GBB with Qwen3-4B and batch size 12, the dominant consumers are KV cache and batching (25.0 GB) and LM weights GB. The Thought Graph and the embedding model occupies only 1.7 GB, about 4.3% of total memory. Since the graph is stored once and shared across queries, its footprint remains small across models and batch sizes. Thus, the added cost is negligible compared to the KV+Batch bottleneck, confirming that RoT can be integrated without straining GPU memory budgets.
Why do smaller models benefit more from RoT? We find that smaller models exhibit larger gains with RoT, likely due to their stronger instruction-follwinn ability. Since GRPO fine fine-tuning is typ- bases, smaller models trained for fewer GRPO epochs, retain ically performed on instruction-tunal following behavior. Larger models, in contrast, undergo more more of their supervised instruction- instruction-folloiingg
extensive reinforcement training that amplifies exploration and raw reasoning ability but diminishes adherence external templates, even after prompt tuning (Nimmaturi et al., 2025). This trade-off between reasoning and instruction following has been observed in prior studies (Fu et al., 2025), and explains why RoT's guidance is more effective at smaller scales. Looking forward, believe efficiency gains will extend to larger models as future reasoning- focused checkpoints better preserve instruction-followww alongside reasoning capacity.
7 CONCLUSION
We presented RoT, a framework that reuses prior reasoning through a thought graph and reward- guided traversal to construct problem-specific templates. RoT achieves substantial efficiency gains, reducing output tokens by up to 40%, latency by 82%, and inference cost by 67.5% while preserving accuracy across benchmarks. These results demonstrate that retrieval-guided template reuse offers practical solution to the growing cost of reasoning heavvy inference. Looking forward, RoT provides scalable foundation for efficient reasoning that extends beyond mathematics broader domains.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work focuses exclusively on mathematical reasoning datasets (AIME 2023/2024/2025, AMC 2023), which do not involve human subjects, personal data, sensitive attributes. The proposed Retrieval-of- Retrieval-ofTToouhh framework is designed to improve inference-tinge efficiency and reduce com- pute and cost, with no foreseeable negative societal impacts such as privacy risks, misuse personal information, or harmful deployment, We have carefully avoided data contamination by ensuring no overlap between the templates used in the thought graph and the benchmarks used for evaluation.
REPRODUCIBILITY STATEMENT
We have taken deliberate steps to ensure the reproducibility of our results. All datasets used in this study are publicly available, and the construction process of the Thought Graph, including metadata annotation, semantic edge building, and caching procedures, is fully documented in the paper and appendices. Detailed algorithm descriptions, hyperparameter settings, and evaluation protocols are provided, along with sensitivity analyses for key parameters. We will release code, configuration files, and processed templates to enable researchers to reproduce and extend our findings.
ACKNOWLEDGMENTS
The work of Azal Ahmad Khan was supported in part by the Amazon Machine Learning Systems Fellowship and the UMN GAGE Fellowship. The work Shend Di was supported by the U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research (ASCR), under contracts DE-ACO2-0CCHI 1357. The work Ali Anwar was supported by the Samsung Global Research Outreach Award and the National Science Foundation Privacy-Preservve Data Sharing in Practice (PDaSP) program under grant number 2452817.
REFERENCES
Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, Dmytro Mo- zolevskyi, Muayad Ali, and Waseem AlShikh. Reflect, retry, reward: Self-improving Ilms via reinforcement learning. arXiv preprint arXiv:2505. 24726, 2025.
Keyu Chen, Zhifeng Shen, Daohai Yu, Haoqian Wu, Wei Wen, Jianfeng He, Ruizhi Qiao, and Xing Sun. Aspd: Unlocking adaptive serial-paralle decoding by exploring intrinsic parallelism in lims. arXiv preprint arXiv:2508.0885 2025.
Seyyed Saeid Cheshmi, Azal Ahmad Khan, Xinran Wang, Zirui Liu, and Ali Anwar. Accelerating Ilm reasoning via early rejection with partial reward modeling. arXiv preprint arXiv:2508.0 019669
2025.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint urXiv:2507.0626h 2025.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Thinkless: Llm learns when to think. arXiv preprint MXXi::2005.13379 2025.
Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang, Efficient reasoning models: A survey. arXiv preprint arXiv:2504.1090) 2025.
Tingchen Fu, Jiawei Gu, Yafu Li, Xiaoye Qu, and Yu Cheng. Scaling reasoning. losing control:
Evaluating instruction following in large reasoning models. arXiv preprint arXiv:2505. 14810, 2025.
Mary Gick and Keith J Holyoak. Analogical problem solving. Cognitive psychology, 12(3): 306-355, 1980.
11

---

Published as a conference paper at ICLR 2026
Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Moham- mad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang. et al. Jina embeddings 2: 8192-token general-purye purerse text embeddings for long documents. arXiv preprint arXiv:2310. 19923, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12445 2025.
Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning. arXiv preprint arXi::2412.18547 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
Guochao Jiang, Guofeng Quan, Zepeng Ding, Ziqin Luo, Dixuan Wang. and Zheng Hu. Flashthink: An early exit method for efficient reasoning. arXiv preprint arXiv:2505.13949 2025.
Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, et al. Enhancing Ilm reasoning with reward-guided tree search. arXiv preprint arXiv:2411.11644 2024a.
Yikun Jiang, Huanyu Wang, Lei Xie, Hanbin Zhao, Hui Qian, John Lui, et al. D-llm: A token adaptive computing resource allocation strategy for large language models. Advances in Neural Information Processing Systems, 37:1725-1749, 2024b.
Dayoon Ko, Jihyuk Kim, Haeju Park, Sohyeon Kim, Dahyun Lee, Yongrae Jo, Gunhee Kim, Moon- tae Lee, and Kyungjae Lee. Hybrid deep searcher: Integrating parallel and sequential search reasoning. arXiv preprint arXiv:2508. 19113, 2025.
Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal. When thinking fails: The pitfalls of reasoning for instruction- following in llms. arXiv preprins arXiv:2505.1 11223, 2025a.
Zhing-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, et al. From system system 2: A survey of reasoning large language models. arXiv preprint 1XXx::5502..74199 2025b.
Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, et al. Dler: Doing length penalty iightiincentivziing more intelligence per token via reinforcement learning. arXiv preprint 1rxiv:25/0.15/10 2025.
Matthew Macfarlane, Minseon Kim, Nebojsa Jojic, Weijia Xu, Lucas Caccia, Xingdi Yuan, Wanru Zhao, Zhengyan Shi, and Alessandro Sordoni. Instilling parallel reasoning into language models. In 2nd Al for Math Workshop@ ICML 2025, 2025.
Datta Nimmaturi, Vaishnavi Bhargava, Rajat Ghosh, Johnu George, and Debojyoti Dutta. Pre- dictive scaling laws for efficient grpo training of large reasoning models. arXiv preprint arXiv:2507. 18014, 2025.
Jayy Pan, Xiuyu Li, Long Lian, Charlie Snell, Yifei Zhou, Adam Yala, Trevor Darrell, Kurt Keutzer, and Alane Suhr. Learning adaptive parallel reasoning with language models. arXiv preprint arXiv:2504. 15466, 2025.
Md Rizwan Parvez. Evidence to generate (e2g): A single-agent two-step prompting for context grounded and retrieval augmented reasoning, arXiv preprint arXiv:2401.0578) 2024.
Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. Mathbert: A pre-trained model for mathematical formula understanding. arXiv preprint Niv:2105.00377, 2021.
Piotr Piçkos, Henryk Michalewski, and Mateusz Malinowski. Measuring and improving bert's math- ematical abilities by predicting the order of reasoning. arXiv preprint arXiv:2106.0390 0092032 2021.
12

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test- compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv: 2024.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang. Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
Zhongxiang Sun, Qipeng Wang. Weijie Yu, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Yang Song, and Han Li. Rearter: Retrieval-augmmened reasoning with trustworthy process rewarding. In Proceedings of the 48th International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 1251-1261,2025
Kaiwen Wang, Jin Peng Zhou, Jonathan Chang. Zhaolin Gao, Nathan Kallus, Kianté Brantley, and Wen Sun. Value-guided search for efficient chain-of-thought reasoning. arXiv preprint (tt.:2505./73733 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint ar arxiv:2203. 11171, 2022.
Zihao Wang, Anji Liu, Haowei Lin, Jiaqi Li. Xiaojian Ma, and Yitao Liang. Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation. arXiv preprint arXiv:2403.05313. 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougg thouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Tong Wu, Chong Xiang, Jiachen T Wang. G Edward Suh, and Prateek Mittal. Effectively controlling reasoning models through thinking intervention. arXiv preprint arXiv:2503.24370, 2025.
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Process- ing Systems, 36:41618-4111 2023.
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min- Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo tree search boosts reasoning via iterative preference learning. arXiv preprint arXiv:2405.004: 2024.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint arX1v:2501.00666, 2025a.
Zenan Xu, Zexuan Qiu, Guanhua Huang, Kun Li, Siheng Li, Chenchen Zhang, Kejiao Li, Qi Yi, Yuhao Jiang, Bo Zhou, et al. Adaptive termination for multi-round parallel reasoning: An univer- sal semantic nutropy-guided framework. arXiv preprini arXiv:2507.0688 06829, 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 0988888 2025a.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: ahgought-augmented reasoning with large language models. Advances in Neural Information Processing Systems, 37:1 13519-1 13544, 2024a.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan. Supercorrect: Advancing small Ilm reasoning with thought template distillation and self- correction. arXiv preprint or arXiv:2410.09008 2024b.
13

---

Published as a conference paper at ICLR 2026
Ling Yang, Zhaochen Yu, Bin Cui, and Mengdi Wang. Reasonflux: Hierarchical Ilm reasoning via scaling thought templates. arXiv preprint arXiv:2502.06772, 2025b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36: 1809-11822, 2023.
Adam Younsi, Abdalgader Abubaker, Mohamed El Amine Seddik, Hakim Hacid, and Salem Lahlou. Accurate and diverse Ilm mathematical reasoning via automated prm-guided gflownets. arXiv preprint arXiv:2504. 19981, 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-6647 2024.
Junyu Zhang, Runpei Dong, Han Wang, Xuying Ning. Haoran Geng, Peihao Li, Xialin He, Yutong Bai, Jitendra Malik, Saurabh Gupta, et al. Alphaone: Reasoning models thinking slow and fast at test time. arXiv preprint (XXi::5555.24633 2025a.
Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301, 2025b.
14

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Test-Time Scaling, Scaling test-time compute has emerged as a prominent strategy for enhancing the reasoning capabilities of LLMS (Snell et al., 2024). This approach mimics the human tendency to dedicate more cognitive effort to more challenging problems (Li et al., 2025b). Early implemen- tations of this idea included methods like BoN sampling and various search-based strategies that explore multiple reasoning paths (Yao et al., 2023; Xie et al., 2023; Jiang et al., 2024a; Wang et al., 2025; Younsi et al., 2025). More recently, since the introduction of models like DeepSeek-R1, the focus has shifted towards RL, particularly using techniques like GRPO for fine-tuning (Guo et al., 2025). These methods encourage models to engage in multiple steps of self-reflection, leading to the generation of huge number of output tokens during inference Bensal et al. (2025). By producing more extensive outputs, these models scale their compute at test-time, often leading to improved performance on complex tasks. However, this increased compute introduces two critical challenges: latency and cost (Cheshmi et al., 2025). The sequential generation of numerous tokens results in significantly higher latency, making real-time applications difficult. Furthermore, the associated costs are substantial, as providers typically price output tokens at a higher rate than input tokens, rendering these methods prohibitively expensive for widespread use (Han et al., 2024).
Efficient Reasoning. High compute and latency overheads of reasoning workloads is creating an urgent need for inference-time efficiency. Recent works address this challenge through adaptive compute. Recent works that tries to parallelize inference achieving higher accuracy at the same la- tency compared to purely serialized reasoning approaches (Pan et al., 2025; Macfarlane al., 2025; Ko et al., 2025; Chen et al., 2025; Xu et al., 2025b). At the architectural level, methods introduce token-level decision modules that selectively skip computation for less informative tokens, reducing FL.OPs and VV-cache (Jiang et al., 2024b). Other approaches have addressed it by finetuning model to minimize unnecessary computational load, including techniques that adaptively select reasoning modes and early termination strategies that halt the thinking process once sufficient information has been generated Fang et al. (2025); Jiang et al. (2025); Zhang et al. (2025a).
Retrieval Augmented Language Models. Retrieval-Augmente been explored as a way to inject external information into the reasoning process. BoT (Yang et al., 2024a) proposed storing distilled reasoning templates and retrieving them inference time guide problem solving. SuperCorrect extended this by distilling multi-granular thought templates from larger teacher models and using them to supervise smaller student models (Yang et al., 2024b). Setrieval-Augmenter Thoughts (RAT; Wang et al. (2024)) advanced this by iteratively revising chains with retrieved exemplars, improving long-horizon reasoning tasks such as math and pro- gram synthesis. Other works such as Chain of Evidences (CoE; Parvez (2024)) and ReARTeR (Sun al., 2025) refine retrieval by grounding generation in evidential sequences or process rewards.
LMs (RALMs) have recently
Our Approach. However, prior approaches primarily target instruction-tuneee models and rely on static templates that are only useful when the new problem closely resembles a previously solved variant. Such static design restricts adaptability and limits efficiency gains. In contrast, our work focuses on models already equipped with strong reasoning capabilities, aiming to make their infer- ence more efficient. Rather than storing entire templates, we decompose them into reusable thought steps that act as atomic reasoning primitives. Given a new query, our system dynamically assembles these steps into fresh template tailored to the problem. This dynamic, fine-grained reuse makes our method substantially more flexible and widely applicable than previous retrieval- based approaches.
B ADDITIONAL MOTIVATION RESULTS
(01) Similarity of steps across datasets. We measured how similar reasoning steps are across range of datasets. First, we solved the datasets using several LMs, prompted for stepwise an- swers, and parsed the outputs into discrete steps. For each step, we computed an embedding us- ing jina- aetaaa-meediiiggg -2-amalle thought graph using cosine similarity. We retrieved the top-10 10 most similar thoughts per step and used the best-match score to summarize similarity. As shown in Figure 2 (left), datasets such as AIME and AMC exhibit strong overlap in the 40-80% similarity range, confirming that many solu- tion steps reused across problems.
model and compared it against stored thoughts in the
15

---

Published as a conference paper at ICLR 2026
Gameer24
Crossword
Txxt
Math
73.5
70
57.4
50
49.0
40
37.1
Parcent
20
21.9
21.4
20
8-20
21-40
41-63
61-80
81-103
Similarity Range
Figure 7: Semantic similarity of steps to solve reasoning questions across datasets.
We further confirmed this trend additional datasets, shown in Figure 3. Here, tasks like Gameof24 and Crossword display high concentrations in the 61-80% range (73.5% and 56.2% of samples respectively). while Text and Math datasets skew toward lower ranges but still exhibit non-trivial similarity. These results demonstrate that reasoning problems across diverse domains frequently reuse similar thoughts, reinforcing that a substantial portion of reasoning can be efficiently supported through retrieval.
(02) Retrieving relevant information from a large vector database is significantly faster than generating text from language model. Reasoning models generate large token volumes. In the QWO-LONGCOT-SS
dataset, QwQ-32B averages 2,185 tokens per response, with 75% exceed- ing 8,000, driving up costs and latency as output tokens cost 4x more than input. Thought reuse reduces costs, as storage and DRAM are far cheaper than GPU-based token generation. Cached thoughts can swiftly retrieved, fetching five matches from a 10,000-document vector DB takes 0.02s. In contrast, Qwen2.5-7B generates 128 tokens in 1.62s, making retrieval quite faster than new tokens generation.
(03) Correctly cached reasoning steps reduce the reasoning tokens required for generation. Thought reuse cuts output tokens by up to 42%, aligning with past studies. Our preliminary results from Figure 2 (right) show its effectiveness varies by dataset, high-similarity tasks like Gameof24 and Crossword benefit more than independent text problems. Thought reusability drives our project, caching slightly increases prompt size but significantly reduces costlier output tokens 4x input cost), Since input tokens process in parallel while output tokens are sequential, reducing them also lowers latency. These results confirm that reusing thoughts improves both cost and efficiency.
C PARAMETER SELECTION DETAILS
We justify each parameter of RoT introduced in Section 4 using analysis experiments on the thought graph that used in the paper. We compute query-step similarities using the jinaai/ http//jjinaeemeediiieee
enn sentence embeddings (with L2 normalization).
Semantic Edge Threshold. We sweep the seman- tic threshold - that determines which semantic edges are retained in the thought graph. The mean seman- tic degree (y-axis) vs. 7 (x-axis) exhibits a clear knee (Figure 8) meaning the connectivity remains stable for T [0.80, 0.85] at roughly node, then drops sharply for 2 0.87. Setting 0.85 therefore places us at the knee preserving strong cross- tross-template connectivity (useful for transfer) without flooding the graph with low-quality edges.
Mean Semantic Degree
40
20
4 semantic neighbors
10
0.86
Tsem
Figure 8: Average number of semantically similar nodes for different thresholds.
First-Step Weights. We quantify the first-step trade- off by plotting, as functions of a, (i) the probability that the selected start lies at true step-O node and (ii) the average query-node similarity of the chosen start (Figure 9). On the full graph, these curves smooth and monotonie, increases, semantic alignment consistently improves while
16

---

Published as a conference paper at ICLR 2026
Average Query Simllaris
1.0
0.0
0.8
0.6
0.6
P(Start
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
Figure 9: First-step selection trade-off on the Thought Graph. Left Probability that the chosen start is a at step-0 node. Right Average query-node similarity of the chosen start.
RoT TI jer 0.9% tamplates)
665 RoT
3,344 templates)
+3.4
+17.00
50
Average Accuracy
40
+0.7
0.0 Qwen3-1.78
Qwen3-4B
Qwen3-88
Figure 10: Template scalability analysis of RoT+TI. Accuracy is reported using smaller thought graph with 0.9k templates (solid bars) versus the full 3.34k templates (hatched bars). Increasing the number templates consistently improves performance, with larger models (e.g., Qwen3-8B) showing the greatest gains.
the bias toward legitimate starts reliably decreases. Notably, beyond o=0.8 the marginal gains in similarity taper, whereas the loss in step-0 probability becomes more pronounced. We therefore set a=0.8, which delivers high-alignment starting point without discarding the structural prior begin at valid template starts, and we observe that this choice is stable with tight uncertainty bands under full- graph sampling.
D ADDITIONAL MAIN RESULTS
"OOT" stands for "Out of Tokens," indicating that the model unable generate the answer within the maximum number of tokens permitted for its response.
D.1 ACCURACY AND EFFICIENCY RESULTS
D.1.1 TEMPLATE GRAPH SCALABILITY ANALYSIS
To examine the scalability of our framework, we conducted an ablation where the thought graph was constructed using only 0.9k templates randomly sampled from our dataset, compared the full 3. 3.34k-template graph used in the main experiments. Figure 10 reports the accuracy of RoT+T1 under both settings across Qwen3-1.77 1.71. 4B, and 8B models. The results indicate that accuracy consistently increases the number of templates grows, with particularly pronounced improvements at larger model sizes. For example, Qwen3-8B improves by 17.0 percentage points when moving from 0.9k to 3.34k templates. These results demonstrate that RoT scales favorably with graph size, suggesting that platforms with access to abundant user reasoning traces can expect significant gains from larger template repositories.
17

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison across different LLMs and baseline methods
Method
AIME 2025
AIME 2024
AIME 2023
AMC 2023
Acc (%)
Tokens
Acc (%)
Tokens
Acc (%)
Tokens
Acc (%)
Tokens
Qwen3-0.6B
CoT
6.67
10004,93
13533.03
13.33
10421.53
42.5
6447.14
RAG
13.33
10523.86
7.41
12533.04
10
10849.5
40
7037.12
BoT
6.67
38792.17
35882
13
30258.34
25
21943.69
COT-SC
13.33
44520.4
11.11
4755% 93
16.67
42095.83
40
26191.12
10933.53
7388.
RoT TI
6.67
5277.87
4318.7
3058.8
2289.92 15
Qwen3-1.7B
11525.89
6403.09
RAG
26.67
13280.8
33.33
12230.67
33.33
11456.15
80
6842.54
13.33
33657.47
25.93
41947.96
23.33
45898.9
62.5
50371.8
COT-SC
47804.8
51.85
30307.18
33.33
45657.83
80
25292.25
RoT
12791.73
40.74
11436.88
33.33
12577.16
72.5
8478.95
RoT TI
8568.47
33.33
9173.96
16.67
8292.7
57.5
5955.47
Qwen3-4B
COT
53.33
11488.33
55.56
10076.15
63.33
11366.2
90
5550.17
RAG
11670.26
62.96
10335.7
53.33
11829.9
90
6240.79
BoT
26.67
50194,53
14.81
43174.37
36.4
49331.67
80
35440,58
CoT.SC
44736.6
RoT
60
11344.6
66.67
10621.33
46.67
13277.83
87.5
6133.01
9421
62.96
46.67
10003.3
90
5052.9
Qwen3-88
CoT
60
12770.6
62.96
11232.55
50
12812.63
90
6468.07
12718.53
87.5
BoT
33.33
59970.86
18.52
49993,15
50
56428.9
85
40784.82
COT-SC
43571.8
77.78
41127.48
63.33
49529.87
92.5
20963.55
RoT
12831.67
66.67
11561.92
53.33
13341.63
90
6522.53
RoT + TI
53.33
11279.07
59.26
10930.03
50
10698.99
87.5
5780.75
Qwen3-14BB
CCT
73.33
9451.27
77.78
9685.22
60
12096.93
97.5
5669.67
RAG
86.67
9910.86
66.67
10981.38
73.33
11750.4
95
$277.07
BoT
OOT
OOT
OOT
OOT
COT-SC
66.67
39272.87
74.07
36955.67 67
63.33
46998.97
97.5
17032.82
RoT TI
66.67
9767.33
62.96
11201.03
63.33
10623.03
82.5
5191.57
18

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison across different LLMs and baseline methods.
Method
AIME 2023
AIME 2024
AIME 2025
AMC 2023
In Tok
Out Tok
Cost ($)
Tok
Out Tok
Cost (S)
In Tok
Out Tok
Cost (S)
In Tok
Out Tok
Cost ($)
Qwen3-0.6B
CeT
285.6
9719.33
50.18
279.59
13253.44
$0.45
335,4
10086.13
50.38
260.57
6186.57
S0.31
RAG
399.33
10124.53
50. 19
386
12147.04
$0.41
428 6
10420.9
50.40
371.27
6665
$0.34
BoT
8234.67
30557.5
6530.7
2935133
$1.02
6342.87
23915.47
$0.92
5604.96
16338.73
$0.85
CoT-SC
906.13
43614.27
$0.83
900
46658.92
$1.59
1105.73
40990.1
$1.55
806.5
25184.62
$1.28
11678.85
RoT TT
345.8
4932.07
$0.09
343.59
3975.1
$0.14
372.9
2685.9
$0.10
286.02
2003.9
$0.10
Qwen3-1.7B
12826.27
59
335
260.57
6142.55
RAG
399.33
12881.47
50.24
386
11844,67
$0.40
428
11027.53
$0.42
371.27
6471.27
$0.33
BoT
17598.55
16058.92
$0.33
15296.48
26651.48
$0.95
18988.8
26910.1
$1.08
17918.00
32453.73
$1.71
CGT-SC
919.13
46885.67
$0.89
598.74
29708
$1.01
1142.5
44515.33
$1.69
806.5
24485.75
$1.24
RoT
275.8
12515.93
50.24
273.44
11163.44
50.38
304.83
12272.33
50.46
216.03
8262.92
$0.42
RoT TI
345.8
8222.67
$0.16
343.59
8830.37
$0.30
372.9
7919.8
$0.30
286.02
5669.45
$0.29
Qwen3-41
CGT
285.6
11202.73
50.21
279.59
9796.56
$0.33
335
11030.8
$0.42
260.57
5289.6
S0.27
RAG
399.33
11270.93
50.21
386
9949.7
$0.34
428
11401.3
50.43
371.27
5869.52
$0.30
BoT
25746.73
24447.8
$0.50
22015.56
21158.80
$0.79
22563.66
26768
$1.09
21432.03
14008.55
$0.80
RsT
275.8
11358.8
50.22
273 /4
10347.89
$0.35
304.83
12973
$0.49
216.03
5916.99
$0.30
II
9075.2
50.17
9138
$0.31
372.9
9630.4
S0.37
286.02
4766.88
$0.24
Qwen3-88
CdT
285.6
12485
$0.39
279.59
10952.96
$0.62
335.4
12477.23
$0.79
260.57
6207.5
$0.52
11897.48
12299.93
BoT
33
27089.73
$0.94
25573.15
24420
$1.51
29494,1
26934.8
$1.86
26854.97
13929.85 85
$1.36
CoT-SC
920.67
42651.13
$1.35
899.78
40227.7
$2.29
1142.5
48387.37
$3.05
806.5
20157.05
$1.70
RoT
275.8
12555.87
50.40
273.44
11288.48
$0.64
304.83
13036.8
$0.82
216.03
6306.5
$0.53
RoT II
345.8
10933.27
50.35
343.59
10586.44
$0.60
372.9
10326.03
S0.65
286.02
$494.73
$0.46
Qwen3-14B
CcT
285.6
9165.67
50.58
279.59
9405.63
$1.07
335.4
11761.53
$1.49
260.57
5409.1
$0.91
RAG
277.13
9633.73
50.61
271.19
10710.19
$1.22
326.8
11423.6
$1.44
252.55
5024.52
$0.85
BoT
00T
OOT
0OT
OOT
OOT
0OT
0OT
COT-SC
920.67
38352.2
$2.42
891.41
36064426
$4.10
1098.57
45900.4
$5.79
769.55
16263.3
$2.74
9911.2
$1.53
216.03
RoT
345.8
9421.53
$0.60
343.59
10857.44
$1.23
372.9
10250.13
$1.30
286.02
4905.55
$0.83
19

---

Published as a conference paper at ICLR 2026
D.2 TOKEN USAGE AND COST IMPLICATIONS
The cost shows the total amount to run the entire experiment.
D.3 INFERENCE API COSTS
Table 6: Inference API Costs for Qwen3 models Alibaba Cloud.
Model
Input Price (Million Tokens)
Output Price (Million Tokens)
Qwen3-0. 6b
$0.1
$1.26
Qwen3-1.7b
$0.11
$1.26
Qwen3-44 44
$0.11
$1.26
Qwen3-8b
$0.18
$2.10
Owen3-14b
$0.35
$4.20
Qwen3-322
$0.70
$8.40
D.4 LATENCY RESULTS
Table 7: Performance comparison across different LLMs and baseline methods
Method
AIME 2023 Latency (8)
AIME 2024 Latency (6)
AIME 2025 Latency (6)
AMC 2023
Latency (6)
Qwen3-0.6B
CaT
86.7
171.01
116.85
68.05
RAG
94,75
157.17
127.03
79.68
BoT
298
337.85
270
174.42
CoT-SC
509.71
611.43
$42.92
343.21
120.11
141.02
129.02
79.73
RoT T1
32.93
29.58
17.07
12.7
Qwen3-1.7B
150.75
RAG
165.17
165
156.58
81.16
BoT
197.6
345.94
369.73
438.35
CC
627.12
401.14
672.93
337.36
RoT
154.66
148.85
176.77
110.95
RoT TI
79.28
112.52
98.87
64.05
Qwen3-4B
CUT
182.6
166.24
195.11
78.58
186.69
172.58
206.86
Bol'
455.45
410.22
560.21
268.87
692.94
RoT
184.52
177.89
245.36
90.53
RoT RoT
135.33
158.86
163.43
73.8
Qwen3-88
CUT
281.04
245.56
285.25
126.8
244.59
BoT
741.66
556.07
694.16
324.38
CaT-SC
910.9
900.96
1109.18
413.16
RoT
273.8
255.75
303.62
129.82
RoT+ 1I
246.3
240.14
228.72
113.79
Qwen3-14B
CrT
241.62
252.21
322.19
142.71
RAG
255.87
291.2
311.75
131.91
BoT
OOT
0OT
CuTSC
1015.18
969.05
1255.63
420.81
263.15
293.53
331.5
155.12
RoT +
251.24
295.89
278.88
129.89
20

---

Published as a conference paper at ICLR 2026
Table 8: Retrieval time (seconds) across datasets and Qwen3 models.
Model
Qwen3-0.6B
Qwen3-1.7B
Qwen3-14B 14BB
Qwen3-4B
Qwen3-8B
AIME 2023
0.0435
0.0434
0.0385
0.0375
0.0403
AIME 2024
0.0402
0.0406
0.0356
0.0338
0.0331
AIME 2025
0.0505
0.0342
0.0294
0.0300
0.0423
AMC 2023
0.0411
0.0376
0.0437
0.0356
0.0344
Average
0.0438
0.0389
0.0368
0.0342
0.0375
D.5 PATH SWITCHING RESULTS
As shown in Table 9, RoT+TI promotes stable, single-path reasoning, cutting back oscillations that inflate outputs under COT. As summarized in the table, all Qwen3 models benefit, with path- switch reductions spanning 14.2-81.8%. The effect persists larger scales, suggesting that template grounding complements scale rather than substituting for it. RoT+TI assists the model in guiding its
Table 9: Average path switches per output (CoT vs RoT TT)
Method
Avg Switches
A vs CoT (%)
Qwen3-0.6B
CoT
29.116
RoT +'I
5.295
-818
Qwen3-1.7B
CoT
24.045
RoT TI
12,732
-47.1
Qwen3-4B
CoT
26.821
RoT +'11
23,027
-14.2
Qwen3-8B
CaT
19.732
RoT TI
15.179
2331
Qwen3-14R
CoT
20.839
RoT
16.045
-23.0
reasoning, enabling it to arrive at the answer by adhering to the template. illustrative example can be observed in the blocks below, which depict the output from RoT+TI, RoT, and CoT, respectively. Through RoT+TI, the model successfully solved the questions using only 920 tokens, whereas RoT and CoT generated significantly more than 6000 tokens to solve the same question. This disparity arises due to the multiple path switching that occurs during the model's 's problem-solving process, as indicated by the (.......) in the output. RoT retrieves the appropriate template for the problem, but the model fails to adhere to the template even when it is provided in the prompt. In contrast, RoT+TI enables the model to solve the problem without path switching.
21

---

Published as a conference paper at ICLR 2026
RoT+TI:
Input and Output for AIME 2024 Q2 No. Out Tokens: 920
You are given a template to solve the problem. Use the given steps if applicable otherwise use them to guide your reasoning and present the solution steps logically to solve this problem:
Input:
Problem: There exist real numbers and y, both greater than I, such that log, (v*)
need to follow the given steps to guide me in reasoning solve the problem. will follow each step and modify the steps to make them match the problem and then solve the problem accordingly. The template is. Step I: Take logarithms (base equal to one variable) both sides to lincarize exponents. Example: For x y", take log, to get
log, (a**) = 10. Find zy. <lim_ fiin__nl><<im start>asistant
logg Strictly following the given steps for guidance, will now solve the problem starting from the step 1. Using step Output: Take logarithms (base equal one variable) of both sides linearize exponents. Let me apply this to the given equation. Given: log, (y²) = (1 = 10.
Step 1: Take logarithms (base equal one variable) of both sides to linearize exponents. For the first equation, log (3²" = 10, can rewrite this using logarithm properties. Recall that (ag)(a") clog, a. So: log, (v²) log, y 10. Similarly, for the second equation
(2g,(())) = 10, apply the same property: log, (z4 = 4y log,, = 10. Now, I have two equations:
1. x log, 10
2. 4y log, x = 10
Let me denote log, y = a. Then log, x J since log, a loga b = 1. Substituting into the
equations:
1. 1. a = 10 + 10
2. 4y 1 10
+ 4y 10 Substituting 10 into the second equation: 4y 10. Simplify:
a
4y 10. Multiply both sides by 10: 4wy 100 Divide both sides by xy 25. So, the value of xy is 25.
10
But wait, the problem states that and y are both greater than 1. Let me check this is consistent. From zy 25, since both and are greater than 1, this is possible. For example, x 5. = 5. Both are greater than I. So the solution valid. Therefore, the answer is 25</think>.
To solve the problem, we are given two logarithmic equations:
log, (v²) = log,, 4v) = 10
Step 1: Use logarithmic properties to simplify the expressions. We apply the logarithmic identity log, c clog, a to each equation. First equation: log, (yr" = x log, 10 Second equation: log, (1*") = 4y log, r = 10
Step 2: Introduce substitution to simplify. Let = log, y- Then by the change of base formula, log,, 1, since log, a log, b 1. Now substitute into the equations: First equation becomes:
10
10
x
Second equation becomes:
1 =
4y
4y
10
10
Substitute a into the second equation:
4y
4xy
I 10
10
10
Multiply both sides by 10:
4xy
100
xy
25
Final Answer:
25
22

---

Published as a conference paper at ICLR 2026
RoT:
Input and Output for AIME 2024 Q2 No. Out Tokens: 6746
"You are given a template to solve the problem. Use the given steps if applicable otherwise use them to guide your reasoning and present the solution steps logically to solve this problem: Step Take logarithms (base equal to one variable) of both sides to linearize exponents. Example: For x yf, take log, get y z log y. Problem: There exist real numbers z and y. both greater than 1, such that log, (y²) = log, (x**) = 10. Find zy." Output: <think>Okay let's try solve this problem. So, we have two equations here because both logarithms are equal to 10. The problem says that log base x of y equals 10, and log base y of 2 also equals 10. And need to find the product xy, where both and y are greater than 1.First, need to recall logarithm properties.......
Input:
Hmm. Now, need to relate these equations. Let me think about how to express log, (v)
and logy(2) (2)........
Let me recall that (x) can be written using change of base formula: logg (x) In(x)//n(y)) Alternatively, since we have expressions for x and in terms of a, maybe we can use the definition of logarithm.........
Let me see if I can take logarithms of both sides to make it casier. Taking natural logarithm on both sides......... Alternatively, since (v²) = 10, which is the same y = _ because logx(A) = B - A = RB So 15" _0 Similarly, (ogy(x"v) = 10 implies that x y 10 So now we have two equations:1) y" '0 2) x n So maybe we can manipulate these equations. Let me take the first equation........ I should have noticed that earlier instead of going through all the logarithmic manipulations and numerical solving. **Final *nswer** 25 </think>
To solve the problem: We given the following equations involving logarithms:
log, (v²) log, (z*²) 10
with the constraint that x > 1 and > 1, and we are to find the value of ry.
Step 1: Apply Logarithmic Identities using the logarithmic identity: (ag)(a") = c clogo We rewrite both equations: log,((1' log, (v) 10 2. log,, (x 4y log,, (x) 10
Step 2:
Use Change of Base Formula We know that: (ag
log. So, log, (1)
Tog,(z] Let's define: log,(x) (x)
log_ (3) =
Now substitute into the first equation:
x = 10 - x 10a
Substitute into the second equation:
10
4y - 10 = y
4a
2a
Step 3: Use the Definition of Logarithms From the definition of logarithms, a log, (x) implies: ya = x Substitute the expressions
for and y: (25) 10a This equation is transcendental and difficult to solve algebraically, but we can proceed to find ry without solving for x and y explicitly.
Step 4: Compute zy From earlier, we found: x 10a, 50a
y =
5. 2a
So, (10a) (E) =
25
Conclusion
Thus, regardless of the specific values of and y, as long as they satisfy the original loga- rithmic equations, the product zy is always:
25
23

---

Published as a conference paper at ICLR 2026
Co Input and Output for AIME 2024 Q2 No. Out Tokens: 6022
Input:
"Solve the following math problem efficiently and clearly. Present the solution steps logically.
For complex problems (3 steps or more): Use this step-by-step format:
Step I: [Concise description] [Brief explanation and calculations]
Step 2:
[Concise description] [Brief explanation and calculations]
Regardless of the approach, always conclude with: Therefore, the final answer is: answer
Where [answer] is just the final numerical answer that solves the problem. Ensure the num- ber clearly identifiable within the box.
Problem: There exist real numbers I and both greater than I, such that log, (v²)
logy (214) 10. Find zy.
Let's solve this step-by-step"
Output:<think' Okay, let's try to solve this problem. So, we have two logarithmic equa- tions: log base x of yd equals 10, and log base y of 4 also equals 10. And we need find the value of xy, where x and y are both greater than First, remember that logarithms have some properties that might help here.........
Wait, so that's s straightforward? So, the answer is 25? Let me check if that makes sense. Let me verify. xy 25, then let's see the equations hold. Suppose xy = 25. Let me pick some numbers. Let me assume........
But then why when tried x=5, y=5, it didn't work? Because maybe and don't have to be integers. Let think..........
Wait, 25 is 625, divided by 4 is 156.25. So, 156.25 (25/2) So, log, (25/2) = logz (2)/2) 2(10g2(25) log, (2) 2log (25) 1]. (2g) is log, (5 log, (5). So. 2 log, (5) - 1] 4 logg2 (5) - 2. But (5) is approximately 2.3219, so 4*2.3219
9.2876 2 7.2876. Which is not 10. So, that's not 10. Therefore, something is wrong here. But according to the algebra........
Let me let x = 5, then y=5. But then, check the first equation: y" = 5$ 3125, 20 5'0 9765625. Not equal. So, not equal. Alternatively, let take sqrt(25) 5, but same as above. Wait, maybe and are not integers. Let me think........
Therefore, the equations are consistent, and the previous contradiction was due to an error in substitution. Therefore, the solution is indeed xy=25. Therefore, the answer is 25, So, even though finding the exact values of and y complicated, the algebra shows that xy=25 is the correct answer. Therefore, the final answer is 25 </think> We are to find the value of zy. given that both and are real numbers greater than 1.
Step 1: Use the logarithmic identity We apply the logarithmic identity: log, (a*) (a)
This gives 1. log,((""))
(v")
log, (v) 102. log, (x 4y log,, (2) 10 Let2019s
denote: log, (v) In Inx
10 I
from the first equation.- log, (x) = In 5. from the
Ia Zy
second equation.
Step 2: Use the reciprocal relationship From the first equation:
10 = log,, (x) 10
log, (v)
x
From the second equation:
log, (x) =
2y
Setting the two expressions for log, (x) equal: 10
5.
2y
Step 3: Solve for zy Cross-multiplying:: 2xy = 50 + xy = 25
Final Answer
25
24


Reviews:
[{"review_id": "ifFC8Hsizd", "reviewer": "Reviewer_UvKG", "summary": "Authors propose a Retrieval-of-Thought (RoT), a “thought graph” that stores prior reasoning steps (thoughts) from solved problems. When a new query arrives, RoT first retrieves relevant thoughts from this graph based on semantic similarity and metadata, and then uses reward-guided traversal to assemble these into a problem-specific reasoning template. Templates are then used to guide inference. This allows the model to reuse and recombine prior reasoning patterns dynamically—reducing redundant exploration typical in Chain-of-Thought (CoT) reasoning.\nAuthors perform experiments on math reasoning benchmarks (AIME 2023–2025, AMC 2023) and Qwen3 models (0.6B–14B), and show that RoT reduces the amount of output tokens and inference latency, while maintaining accuracy.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel dynamic retrieval-based reasoning framework that reuses fine-grained reasoning steps via a thought graph. RoT's ability to assemble context-specific templates on the fly using reward-guided traversal is novel. It brings retrieval closer to compositional memory and human-like analogical reasoning.\n2. Results show consistent improvements across performed experiments", "weaknesses": "1. All experiments are conducted on mathematical reasoning tasks (AIME, AMC), which have highly structured, formulaic reasoning patterns and strong inter-problem overlap. This domain bias may overstate the general efficiency of template reuse, since the retrieval of semantically similar reasoning steps is easier in math than in open-domain or multimodal reasoning.\n2. The system currently relies on manual tagging of template types and knowledge domains (e.g., algebraic, geometric) to filter candidate nodes. This introduces a degree of human supervision and limits the method’s autonomy and scalability. Especially if we concider going beyond mathematical domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5X9FdsANAB", "reviewer": "Reviewer_ZE4Z", "summary": "This paper proposes Retrieval-of-Thought (RoT), a framework to improve the inference efficiency of Large Reasoning Models (LRMs) by reusing prior reasoning steps. RoT organizes individual reasoning steps (“thoughts”) into a thought graph with sequential edges (preserving template flow) and semantic edges (connecting analogous steps across templates). At inference, RoT retrieves query-relevant graph nodes, uses reward-guided traversal to assemble dynamic, problem-specific templates, and integrates these templates into the prompt (via `` tags for adherence). Experiments on mathematical reasoning benchmarks (AIME 2023–2025, AMC 2023) with Qwen3 models (0.6B–14B) show RoT reduces output tokens by up to 40%, latency by 82%, and cost by 59% while maintaining or improving accuracy—outperforming baselines like CoT, RAG, and BoT. Key contributions include: (1) a dynamic template construction paradigm (vs. static templates in prior work), (2) the thought graph with reward-guided traversal, and (3) empirical validation of efficiency gains across model scales.", "questions": "See the weakness part.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* RoT addresses a critical limitation of prior retrieval-based reasoning (e.g., BoT, RAG)—static templates—by introducing dynamic, query-adaptive template assembly via a structured thought graph. This mimics human “connecting the dots” (Gick & Holyoak, 1980) and enables flexible reuse of granular reasoning steps, a novel departure from fixed scaffolds.\n* The work is methodologically rigorous: it formalizes core components (templates, thought graph) with mathematical definitions, uses a curated template dataset (ReasonFlux) to avoid benchmark contamination, and conducts comprehensive evaluations (accuracy, tokens, latency, cost, path switching). Ablations (e.g., thought graph scalability, retrieval overhead) further validate robustness.\n* The paper is well-structured: Figure 1 clearly contrasts RoT with CoT, Algorithm 1 distills the inference workflow, and appendices (e.g., parameter selection, additional results) provide necessary details. Technical concepts (semantic edges, reward functions) are explained accessibly, balancing depth and readability.\n* RoT directly tackles the “efficiency-accuracy tradeoff” of LRMs—where longer reasoning traces improve accuracy but inflate latency/cost. By showing substantial efficiency gains (especially for small/medium models) without performance loss, it provides a scalable solution for real-world LRM deployment, where cost and latency are critical constraints.", "weaknesses": "* The evaluation is limited to mathematical reasoning (AIME/AMC). While the authors note RoT is domain-agnostic, testing on non-mathematical reasoning tasks (e.g., logic puzzles, code debugging, scientific problem-solving) would strengthen claims of generalizability—especially to domains with less standardized reasoning patterns.\n* The reward function parameters (e.g., α=0.8 for initial node selection, τ=0.85 for semantic edges) are justified via sensitivity analysis in Appendix C, but the analysis is limited to the used dataset/model. Testing how these parameters perform across different template sizes or reasoning domains would improve confidence in their general applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "orhZ806jIK", "reviewer": "Reviewer_VNeJ", "summary": "The paper introduces Retrieval-of-Thought (RoT), a framework that reuses previously generated reasoning steps as composable “thought” units to guide reasoning on new problems. By replacing text generation—which is typically slow and computationally intensive—with retrieval from a vector database, RoT improves inference efficiency while maintaining reasoning quality. \n\nThe key insight is that in the chain-of-thought process, individual reasoning segments can be represented as graph-like nodes, enabling retrieval and recomposition into new reasoning traces.\n\nExperimental results demonstrate that RoT+TI substantially reduces output tokens, per-sample cost, and end-to-end latency, while incurring only minimal accuracy degradation.", "questions": "1. Does RoT exhibit robustness against confounding or near-duplicate inputs? Specifically, when two problems share identical structures but differ only in numerical values, could the system retrieve reasoning blocks with correct logic but incorrect numerical content, thereby misleading the model’s final output?\n2. To what extent does the proposed Thought Graph Construction method generalize beyond mathematical reasoning tasks? Would the same approach remain effective for domains involving non-numerical or open-ended reasoning?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper proposes a retrieval-based chain-of-thought framework that efficiently reuses previously generated reasoning blocks, offering a novel and highly effective approach to improving inference efficiency.\n2. The work introduces a well-defined methodology for Thought Graph Construction and Retrieval, providing a conceptually elegant and technically sophisticated mechanism for organizing and reusing reasoning processes.", "weaknesses": "1. The experimental evaluation is limited to the Qwen3 family of models (0.6B, 1.7B, 4B, 8B, and 14B), all of which are relatively small and share the same architecture. This restricts the assessment of the proposed method’s scalability and generalizability across different model families.\n2. The evaluation relies solely on the AIME and AMC datasets, which together include only about 130 problems and primarily target high-difficulty mathematical reasoning. This narrow dataset scope limits the assessment of RoT’s generalization across varying difficulty levels and task domains.\n3. The paper does not evaluate RoT’s robustness to confounding or near-duplicate problems. For example, when two questions share similar structures but differ in numerical details, it remains unclear whether RoT might retrieve a reasoning block with correct logic but incorrect values, potentially misleading the model’s final prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)