Rethinking LLM Reasoning: From Explicit Trajectories to Latent Representations
Authors: Cong Jiang
Xiaofeng Zhang
Fangzhi Zhu
XiaoWei Chen
Junxiong Zhu
Zheng Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
efficient reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) have achieved impressive performance on complex tasks by generating human-like, step-by-step rationales, referred to as \textit{reasoning trajectory}, before arriving at final answers. However, the length of these reasoning trajectories often far exceeds that of the final answers, which incurs substantial inference costs even for relatively simple tasks. Advanced methods typically attempt to compress reasoning trajectory length through post-training, but they remain decoding-intensive and fail to inherently mitigate the efficiency challenge. In this work, we challenge the necessity of generating full reasoning trajectories and empirically demonstrate that LLMs can generate accurate answers using only fragmental reasoning paths, without relying on complete token-by-token sequences. To this end, we propose a novel \textbf{Latent Reasoning Tuning (LRT)} framework, which empowers LLMs to perform reasoning using implicit, compact, learnable representations instead of explicit textual trajectories. Technically, LRT replaces the costly autoregressive generation of reasoning steps with a single forward pass through a lightweight reasoning network, which generates latent vectors that encapsulate the necessary reasoning logic and condition the LLM to produce the final answer. Experiments on mathematical and out-of-domain benchmarks demonstrate that our LRT consistently outperforms relevant efficient reasoning methods. Moreover, by transforming explicit reasoning into latent reasoning, our approach surpasses the state-of-the-art Qwen3 hybrid reasoning framework. Code is available at \texttt{https://github.com/MobiusDai/LRT} .

Published as a conference paper at ICLR 2026
RETHINKING LLM REASONING: FROM EXPLICIT
TRAJECTORIES TO LATENT REPRESENTATIONS
Cong Jiang
Xiaofeng Zhang
Harbin Institute of Technology. Shenzhen jerryjiang gmaii com
Harbin Institute of Technology, Shenzhen
Fangzhi Zhu
Xiaowei Chen Independent Researcher
Junxiong Zhu
Independent Researcher
Independent Researcher
Zheng Zhang*
Harbin Institute of Technology. Shenzhen Shenzhen Loop Area Institute, Shenzhen
ABSTRACT
Large Language Models (LLMs) have achieved impressive performance on com- plex tasks by generating human-like, step-by-step rationales, referred soning trajectory, before arriving at final answers. However, the length of these reasoning trajectories often far exceeds that of the final answers, which incurs sub- stantial inference costs even for relatively simple tasks. Advanced methods typ- ically attempt compress reasoning trajectory length through post-training, but they remain lenge. In this work, challenge the necessity of generating full reasoning trajec- tories and empirically demonstrate that LLMs can generate accurate answers us- ing only fragmental reasoning paths, without relying on complete ooken-by-toke sequences. To this end, we propose novel Latent Reasoning Tuning (LRT) framework, which empowers LLMs to perform reasoning using implicit, compact, learnable representations instead of explicit textual trajectories. Technically, LRT replaces the costly autoregressive generation of reasoning steps with a single for- ward pass through lightweight reasoning network, which generates latent vectors that encapsulate the necessary reasoning logic and condition the L.MM to produce mathematical and out-of-domain benchmarks
rea-
and fail to inherently mitigate the efficiency chal-
the final answer. Experiments demonstrate that our LRT consistently outperforms relevant efficient reasoning methods. Moreover, by transforming explicit reasoning into latent reasoning, our approach surpasses the state-of of- oftte--f-theaaae is available https: / //it/ttiigt/ hightb...... com/ Mww.//oooiibss//......
the-art art Qwen3 hybrid reasoning framework. Code
INTRODUCTION
Recent advances in large language models (LLMs) have enabled slow-thinking reasoning mod- els (Min et al. 2024), including OpenAl ol (Jaech et al.) 2024). DeepSeek-RI (Guo et al. 2025) and Qwen QwQ (Team] (025). The output of these models typically consists of reasoning tra- jectory along with summarized answer, the latter serving as concise synthesis of the former. Through increased allocation of computational resources, these models have demonstrated signif- icantly enhanced capabilities in solving complex tasks. Such reasoning capabilities are acquired through supervised fine-tuning (SFT) and reinforcement learning. For instance, DeepSeek-RR RR em- ploys Group Relative Policy Optimization (GRPO) with rule-based reward signals following the SET phase, yielding models with superior reasoning performance.
Despite their impressive capabilities, reasoning LLMs often incur substantial computational over- head as they generate lengthy reasoning chains for backtracking and self-veriiioation even for simple
"Correspondence Zheng Zhang <daarennz21966 cmmil.com.>

---

Published as a conference paper at ICLR 2026
Reasoning Process
Question Tokens (2
T1 Po(-1X)
Fh ( PA-1)
Reasoning Tokens [N.PPo..,))]
Y1 Pa(1
Questioo
Ressoning Trajectory
Answer
(sompled stop by stop from
Yn Po(-1 X. R.R. Vm.1)
Reseaning Tokens (r..22....dd
Latent Reasoning Process
Answer Tokens Dyyl
Pg
In ... Ga(X)
Y1 - P&(1 X,R)
Large langiage Model
Question
Latent Reasoring Trajectory (computed directly via Gg)
Answer
Yu Po( X. Ya-1)
C. Reasoning
Figure I: Comparison between the schematic diagrams of the reasoning LLM generation process and the Latent Reasoning method generation process.
tasks. This phenomenon, often referred to as overthinking (Sui et al. 2025). leads to lengthy reason- ing chains that waste computation and increase inference latency, hindering real-time applications.
Mitigating the substantial inference costs of slow- thinkinn reasoning models, which are largely dom- inated by the auto-regressive generation of extended reasoning trajectories, has become critical research imperative. A prominent research line explores post-training approaches that explicitly compress reasoning (Luo et al., 2025a Hou al. 2025). Methods such as ShorterBetter (Yi al., 2025) construct dynamic rewards by selecting the shortest correct sample among multiple genera- tions, while LC-RI (Cheng al.) 2025) integrates collaboration-lennthh and compression terms in addition to accuracy rewards. These reinforcement learning based approaches encourage shorter responses; however, they remain fundamentally reasoning trajectories, and the imposed length reward may even constrain problem-solving for real hard problems. complementary line of work attempts to bypass reasoning entirely by substitut- ing trajectories with fixed prompts. For instance, NoThinking (Ma et al. (025)) prefills a fabricated thinking block to skip chain-of-thon oftiohof-thought generation, and Qwen3 (Yang et al. (2255) enforces direct answer emission via special control token. While such methods effectively eliminate reasoning tokens, their reliance on rigid prefilling introduces brittleness and can impair performance.
-tinkkiiiiin".. the models still traverse extended
Unlike existing methods that pursue efficiency primarily via fine-tuning prompt control, pro- pose latent reasoning tuning (LRT), a framework that fundamentally reimagines reasoning com- putation. As illustrated in Figure I we introduce additional lightweight component, termed the reasoning network, to facilitate model reasoning. This component converts explicit reasoning trajec- tories into fixed-length, implicit latent representations, thereby obviating the need for autoregressive sampling of individual reasoning steps. The core of LRT is training reasoning network G to gen- erates latent reasoning chains which support the reasoning model to generate the final answers. Our approach mitigates the limitations of the two aforementioned methods: on the one hand, it replaces explicit reasoning trajectories with latent representations that can be directly computed; on the other hand, since our latent representations are derived from reasoning network, they can be optimized to further enhance model performance, rather than relying on fixed representations for all inputs. Furthermore, its modular and non non-intruee non-intrusive design allows reasoning LLMs to be augmented with- out parameter modifications, thereby supporting seamless transitions between latent and explicit reasoning modes. Our primary contributions
follows:
We propose novel Latent Reasoning Tuning framework which enhances reasoning effi ciency by replacing the explicit, token-by-token generation of reasoning steps with a com- pact latent trajectory computed via an auxiliary network.
The design of our framework is grounded in key finding from our analysis of LLM rea- soning. We demonstrate that models maintain high accuracy even when conditioned on fragmented reasoning trajectories, establishing that fully explicit trajectory is not essential for correct inference. Building on this insight, our method further transforms the explicit trajectory into latent representation.
Experimental results demonstrate that LRT outperforms other efficient reasoning ap- proaches when forcing the model to reason efficiently and surpasses the performance of Qwen3's non-thinking mode, thereby validating the effectiveness of our framework.
2

---

Published as a conference paper at ICLR 2026
2 REASONING TRAJECTORY ANALYSIS
As analyzed in TokenSkip (Xia al. 2025). tokens in the reasoning trajectory contribute unequally; many serve primarily as transitional elements that maintain coherence. These tokens can therefore be omitted to compress the trajectory, and token importance can be quantified using perplexity or BERT-like language model. We further argue that, in slow-thinking models, the backtracking and self-verification behavior allows even certain important tokens or entire sub-steps to be partially compressed without loss of performance. To verify this hypothesis, we provide LLMs with rea- soning trajectories of varying completeness and measure how these incomplete trajectories affected final answer accuracy.
Settings. Let Pg be a reasoning LLM. In our implementation, we employ Deepseek-RI-Diig Qwen-7B. Given prompt X and its corresponding reasoning trajectory R. the model defines conditional distribution over answers Po R)). We can obtain the final answer by auto- regressively sampling from this distribution Y Pa(. RJ). To examine redundancy, construct incomplete trajectories by randomly omitting certain tokens or steps and then compare the performance of models conditioned on complete reasoning trajectories with those conditioned on incomplete variants. The skipping scheme is designed at two levels of granularity:
Token-level Skipping: For a skip rate p E (0, 1). construct (p) by independently deleting each token in R with probability (preserving the order of the remaining tokens).
Step-level Skipping: Segment into sentences/steps; for a skip rate p E [0, 1]. construct (g) by randomly deleting each step with probability p.
We generate answers conditioned on these incomplete trajectories: Pi(p) Po [X, (p))) and Y.(p) ~ Po [X, (p))).
Observation. Figure 2 presents sys- tematic comparison between models con- ditioned on complete reasoning trajecto- ries and their incomplete variants across five skipping ratios. When provided with complete trajectories, the model consumes an average of 3529.3 tokens and achieves a pass rate of 92.8%, outperforming all incomplete counterparts. Notably, the performance degradation from trajectory ablation remains minimal. As the skip rate increases, model performance demon- strates remarkable robustness: when 30% of tokens are randomly omitted, the pass rate decreases by fewer than 2 percentage points; 50% token-level skip rate, the model maintains a 90.60% pass rate while utilizing approximately half the original trajectory length.
Accuracy (%)
90
Step Original
92,80
91.85
20%
91.45
30%
91.53
90.35
50%
89.48
92.24
Level
20%
92.28
50%
60%
91.36
50%
90.60
Toker Nome (Dripra))
Twin Ntrra (Stup Lavel)
Tolkrr Nome (1ikm Lavel)
(Ongga))
Accorsay (3iup
Based on the above observations, conclude that: 1. Reasoning trajectories exhibit substantial redundancy. Consis- tent with our hypothesis, the model main- tains robust performance despite skipping 50% of tokens or steps, demonstrating that reasoning trajectories contain significantly more in- formation than required for correct answer inference. This suggests that current reasoning LLMs generate excessive intermediate representations. 2. Models demonstrate resilience to noisy or fragmental input. The model is able to exploit salient information even from highly degraded tra- jectories, despite their higher perplexity. This robustness indicates that reasoning LLMs possess strong information-filtrring capabilities and can identify critical reasoning components amid sub- stantial noise incompleteness.
Figure 2: Experimental results of Deepseek- R -Dstilll-- Qwen-7B on Math-500 and corresponding token con- sumption in reasoning trajectories.
3

---

Published as a conference paper at ICLR 2026
Slow-dhinking Reasoning
Reasoning
Decode
Decode
Decode
Decode
Prefil
Decode
Decode
Tt
Reasoning Trajectory
Latent Trajectory
Figure 3: Explicit Slow-thinking Reasoning vs. Latent Reasoning. Comparison of the decoding pro- cess. Our latent reasoning performs the reasoning steps in compact latent representations, avoiding costly intermediate text generation.
METHOD
In this section, we begin by analyzing the reasoning model alongside existing efficient reasoning approaches. We then introduce our proposed latent reasoning tuning framework, elaborating on its architectural components, training methodology, and inference procedure.
3.1 PRELIMINARY
For a reasoning LLM Po. the generation process typically involves producing intermediate reasoning content before arriving at the final answer to a given prompt. Formally, we denote the input prompt R = [rı,..., rk). and the final answer as Y
as X [zo...., ] the reasoning trajectory [yi...., ym where in general & >> m. The reasoning trajectory is generated autoregressively according to the conditional distribution Pa(- X). This process can be expressed as:
T1 Po(- I)),
12 Pa(: [X, .... Tk Po(. [X, rk-1)),
(1)
where [.-]] denotes the concatenation operation. The final answer is also sampled in this way, can be expressed Y ~ P- R.).
Previous research has focused on internaizzing the reasoning process by fine-tuning models to rectly predict the final answer Y without explicitly generating intermediate trajectories R. The optimization objective is to learn a model Pa such that Pa(- X) approximates the behavior of Po( [X, RJ), effectively bypassing explicit reasoning. However, discarding intermediate reason- ing steps often leads to suboptimal reasoning quality and reduced adaptability. Moreover, these methods typically output only the single answer, without any summary of the underlying rationale. Another line of work employs reinforcement learning to encourage models to generate more con-
cise reasoning trajectories R. These approaches optimize Pa such that (- [X, Rt)) aligns with the original distribution while reducing redundancy in R. Nevertheless, even the shortened trajectories R remain considerably longer than the final answer Y, thereby limiting efficiency gains.
Moreover, both internalization-Be based and RI.-based approaches require retraining the model, which substantially hinders the ability to leverage long-form reasoning for more challenging tasks.
3.2 LATENT REASONING TUNING
Our framework bypasses the computationally expensive process of generating explicit reasoning traces (illustrated Figure Instead, utilizes compact latent representations, thereby eliminating the redundancy inherent in step-by-step reasoning.
Under greedy decoding, the generation of the reasoning trajectory becomes a deterministic process. We can therefore formalize this process (Equation [T as a function X θ - R, where 1(X,00 0). Consequently, the probability distribution for the final answer is given by:
PY [X, R)) (Y [X, (XX(X,0)))
(2)
4

---

Published as a conference paper at ICLR 2026
Algorithm Latent Reasoning Tuning Framework
Require: Base model Pa. reference model Pres training dataset D = {(X,, Yi)
Ensure:
Trained reasoning network Gg
1: Initialize: Reasoning network Gs with parameters Φ
2: Freeze: Base model parameters
3: Stage Supervised Fine-tuning
4: for batch (X,Y) Y) D do
5:
Ex - Embeddding
P. Get the embedding of input tokens Extract final hidden states > Generate latent reasoning Compute SFT loss
6:
Hx HiddenStates (Ex)
7:
2
8:
LSFT log PY [Ex;=])
9:
Update & using V CSFT
D Update reasoning network
10: Stage 2: Reinforcement Learning
for batch X D do
12:
Ex - Embedding (X)
> Get the embedding of input tokens Extract final hidden states D Generte latent reasoning
Hx HiddenStatesg (Ex)
14:
f- G
15:
YIA Sample(Pa( [Ex; 2]))
D Generate K candidate answers
Tisk f [ComputeRevard({; })
> Compute rewards
17:
&- mean(ri:*), t- std(r1:*)
A& < (rk - -r)/ar
Normalized advantages
19:
Ps: < ComputeRatio(P) Po. Pit. Fx)
20:
k=1 min (pa Ak, clip(pk, 1 -- c)Ab) Update e using V CGRPO
LGRPO + K Σ
b Clipped policy loss b Policy update
Inference:
23:
function GENERATEANSWERRXK.o Ex + Embeddding (Xtest)
24:
25:
Hx HiddenStatesg/ (Ex)
26:
Ztest + (Hx)
27:
Y preed t- Decode Po(:
28: return Ypred
The function h preserves the autoregressive structure of reasoning, ensuring that each token E R depends causally on its predecessors ret- However, our analysis Section demonstrates that complete step-by-step trajectories are not essential for achieving high performance. This finding indicates that LLMs primarily leverage salient trajectory components to derive final answers, sug- gesting that the strict autoregressive constraint is not prerequisite. Motivated by this insight, we propose circumventing the explicit generation process by introducing a dedicated reasoning ner- work, G X + Z, which directly maps inputs to compact latent representations of reasoning trajectory:
2 = (X).
(3)
The latent representation = is optimized to support the downstream prediction of the correct answer, thereby serving as compact surrogate for the explicit trajectory R.
To train the reasoning network G& generate effective latent representations, employ two- stage training paradigm. The first stage uses Supervised Fine- Tuning (SFT) to align the behavior of the reasoning network with the reasoning LLM. The second stage leverages reinforcement learning to further enhance its problem-solving capabilities. The two-stage training and inference process is presented in Algorithm]]
The primary objective of the SFT stage is to ensure that the latent trajectories produced by G, enable the reasoning model Po to replicate the answer of its original, explicit reasoning process. Formally,
5

---

Published as a conference paper at ICLR 2026
aim to make the conditional probability distribution Po |X. (X))) closely approximate the target distribution Po [X, h(X, ()). A common approach for aligning distributions is knowledge distillation, which would involve minimizing the KL-divergence between them. But this method requires generating logits for the target distribution, a computationally prohibitive step. We therefore adopt more direct and efficient SFT approach that circumvents this requirement. Our SFT dataset DD consists of triplets (X,, R., Y) extracted from the outputs of reasoning LLM, where R& denotes the reasoning trajectory and Y, the final answer. While the dataset contains both trajectories and answers, our training objective only leverages (X,, Y;). For each input X;, we first extract its final hidden state representation, Hx,, from the reasoning model Po. This state serves as a contextual embedding of the input X;. The reasoning network Ga then maps this representation to latent trajectory. We optimize the parameters & of the reasoning network by minimizing the negative log-likelihood, formally expressed as:
L(d) = - log fo(Y [xx, Gx(H)))).
(4)
While the first stage aligns the reasoning network with the reasoning model's behavior, it is inher- ently limited by the quality of the in the training data. To transcend this limitation and enhance the model's 's intrinsic problem-solving capabilities, we employ reinforcement learning in the sec- ond stage of training. In this stage, we refine Go by providing a direct reward signal based on the correctness of the final answer. This signal offers verifiable feedback for optimizing the latent rea- soning process. Unlike SFT, which promotes imitation, the RI. objective incentivizes the reasoning network to explore the latent space for more effective reasoning trajectories that consistently yicld correct outcomes.
4 EXPERIMENTS
4.1 SETTINGS
Models. We evaluate our method alongside baseline approaches DeepSeek-R--Distil--ouee 1.5B (Guo et al. 2025) and the Qwen3 series (Yang et al. (025). DeepSeek-RI -iiii-Ri-iiiiiiii------l- suitable model for evaluation it is specifically optimized for reasoning tasks, making it common target for efficiency improvements. The Qwen3 series features a native hybrid reasoning mode controlled via chat templates, where special tokens toggle between thinking and non non-thinking modes. Our method offers an alternative approach to hybrid reasoning by transforming explicit rea- soning models into latent reasoning models, achieving similar flexibility with improved efficiency. The reasoning network employs vocabulary of 256 learnable embeddings.
(Zhang et al. 2025 to operate over
Datasets. For model training, we utilize the OpenR 1MenR1-Math-220k dataset (Hugging Face 2025) for supervised fine-tuning and the SeepScaleR-Prevee--DDee learning, respectively. To provide a comprehensive evaluation, select five diverse reasoning benchmarks that encompass mathematical, logical, and scientific domains. Our assessment begins with GSM8K (Cobbe et al.] 2021). which tests multi-step arithmetic reasoning through linguisti- cally diverse grade-school word problems. For more advanced mathematical challenges, we utilize MATH-500 (Hendrycks et al. [222] Lightman et al. 20023. a competition-level subset of the MATH dataset, alongside the American Mathematics Competitions (AMC) dataset (MAA) 2023), which as- sesses creative mooblem-solvingg and insight beyond routine calculations by demanding the synthesis of non-obvious solution strategies. To further probe the generalization capabilities of our method on out-of-domain benchmarks, we incorporate the LSAT (Zhong al.) (002)) and GPQA (Rein al., 2024). The LSAT evaluates analytical deduction and reading comprehension through complex argu- mentative structures, targeting crucial abstract reasoning capabilities. Concurrently, GPQA gauges performance on expert-level scientific problems, presenting collection of graduate-leve questions in physics, chemistry, and biology specifically crafted to resist straightforward information retrieval and instead demand profound domain knowledge and intricate multi-step reasoning.
(Luo et al. 2025b) for reintorcement
Baselines. In addition to the base model DeepSeek-RR SerpsSee--R--isill----- RI--Disill--Wenn 1.5B, we compare our ap- proach with several efficient reasoning methods: NoThinking (Ma et al. (025), which bypasses the reasoning process through a simple prompt; ShorterBetter (Yi et al. 2025). which employs rein- forcement learning with dynamic reward signal guide the model toward more efficient reasoning:

---

Published as a conference paper at ICLR 2026
and LC-R1 (Cheng et al. 2025), which combines length and compression rewards encourage the model to retain only the most critical steps.
4.2 RESULTS AND DISCUSSIONS
Table Accuracy (%) different baselines and our method on in-domain and out-of-domain tasks.
In In-Domain Tasks
Out-of-Domall Tasks
Method
Budget
Average
AMC
MATH-500
GSM8K
LSAT
GPQA
Baseline
33.25
43.15
70.00
19.02
24.24
37.93
NoThhinking
37.75
58.35
73.24
18.15
23.74
42.25
ShorterBetter
512
33.87
55.11
60.78
19.05
26.23
39.01
LC-R1
35.75
48.00
74.26
18.59
24.24
40.17
Ours
38.00
60.65
77.16
19.57
29.17
44.91
Baseline
42.88
67.50
79.10
20.98
28.16
47.72
NoThinking
40.25
66.70
75.00
22.72
25.88
46.11
ShorterBetter
1024
36.31
55.76
60.78
18.32
28.38
39.91
LC-R1
44.87
68.00
78.98
20.22
30.56
48.53
Ours
42.50
68.50
78.95
22.39
30.55
48.58
Comparison with Other Efficient Reasoning Methods. Table presents a comprehensive com- parison between method and four baseline models. Under the 512-token budget., for in-domain tasks, our method improves baseline accuracy across three benchmarks from 33.25%, 43.15%, and 70.00% to 38.00% 60.65%. and 77.16%, respectively. Compared to the NoThinking's prompt strategy, our method achieves average improvement of 2.16%. Furthermore, our approach out- performs the RL-based efficient reasoning methods ShorterBetter and LC-RI by average margins of 8.68% and 5.93%, respectively. For out-of-domain tasks, our method also demonstrates superior performance. Compared the baseline model, it improves accuracy from 19.02% and 24.24% to 19.57% and 29. 17%, respectively. In addition, our method surpasses the NoThinking, ShorterBet- ter, and LC-RI approaches by average margins 3,43%, 1.73%, and 2.96%, respectively. These results highlight the effectiveness of our method under limited token budgets. When more tokens are available, our approach continues to outperform other baselines in terms average accuracy.
Table 2: Performance comparison of Qwen3 non-thine non-thinking mode and our method on Qwen3-1.7B and Qwen3-4B.
In-Domain Tasks
Out-of-Domain Tasks
Model
Average
AMC
MATH-500
GSMSK
GPQA
LSAT
base @
44.50
66.05
66.79
30.81
26.52
46.93
ours @
44.50
60.90
77.01
32.07
27.61
48.42
Qwen3-1.7B
base@4 4
50.00
77.80
83.85
56.57
44.78
62.60
ours @4
51.00
77.40
89.61
62.12
53.91
66.81
base @
47.25
70.70
75.08
44.82
32.50
54.07
ours @
46.25
72.60
88.51
39.27
28.59
55.04
Qwen3-4B
base @4 4
54.00
80.00
88.10
64.65
42.17
65.78
ours @4
54.00
84.80
95.07
67.17
56.96
71.60
Comparison with the Qwen3 Series Models. The Qwen3 series integrates thinking and non- thinking modes within single model, with the mode determined by the chat template. Since our method transforms reasoning models into latent reasoning models, it offers altemative approach for achieving hybrid reasoning. To evaluate this capability, we converted the thinking mode to latent reasoning and compared its performance against the corresponding non-thing non-thinking mode. We report results for Qwen3-1.7B and Qwen3-4B models enhanced with our method, evaluated against their non-thinking counterparts across five benchmarks. Performance is measured using the pass @k

---

Published as a conference paper at ICLR 2026
metric: for each problem, k samples generated, and the problem is considered solved if at least one sample produces the correct answer.
As shown in Table 2 our Latent Reasoning Tuning method improves average accuracy across five benchmarks for both Qwen3-1.7B and Qwen3-4B models, increasing pass@1 from 46.93% and 54.07% 48. 48.42% and 55.04%, respectively. The improvements in pass @4 tial, rising from 62.60% and 65.78% to 66.81% % and 71.60%, respectively. Notably, while pass @ performance occasionally matches slightly underperforms the non-thinking mode, pass@ 4 con- sistently surpasses it, indicating that our method generates more diverse solution paths. These results demonstrate the effectiveness our approach enabling hybrid reasoning capabilities.
substan-
4.3 ABLATION ANALYSIS
This section examines the results of our latent reasoning tuning methods with respect to the number of latent reasoning tokens and the training strategies employed.
Table 3: Accuracy (%) of the latent reasoning method with varying numbers of latent tokens.
In- In-Domain Tasks
Out-of- Tasks
Tokens
Average
AMC
MATH-500
GSMSK
GPQA
LSAT
64
36.25
55.15
69.43
26.26
25.54
42.53
128
41.50
58.90
73.67
29.04
22.07
45.04
256
44.50
60.90
77.01
32.07
27.61
48.42
512
41.50
61.45
76.88
28.91
25.87
46.92
Analysis of the Number of Reasoning Tokens. Initially, we evaluated the LRM's performance using a fixed number of latent reasoning tokens. To further investigate the impact of token quantity, we conducted experiments with the Qwen3-1.7B model, varying the number of reasoning tokens from 64 to 512. All models were trained under identical experimental settings, with performance evaluated across five benchmarks. As shown in Table 3 performance improves reasoning tokens increase up to n < 256, with accuracy rising from 42.53% to 45.04% and then to 48.42%. This finding aligns with the test-time scaling law, as performance improves with increasing number of reasoning tokens. However, further increasing the token count does not yield continued improve- ments. When using 512 reasoning tokens, average performance falls below that of the 256-token model, with superior results observed only on the MATH-500 benchmark. This suggests that larger training scales may be necessary fully leverage additional latent reasoning tokens.
Table 4: Accuracy (%) of the latent reasoning method under different training methods.
In-Domain Tasks
Out-of-Domain Tasks
Training Method
Average
AMC
MATH-500
GSM8K
GPQA
LSAT
SFT SFT + RL
37.00
54.65
63.64
28.66
22.50
41.29
44.50
60.90
77.01
32.07
27.61
48.42
Analysis of the training methods. The reasoning network employs a two-stage training process. The first stage uses the reasoning dataset for supervised fine-tuning, followed by reinforcement learning to optimize the model and enhance its problem-solving capabilities. To evaluate the effec- tiveness of the two-stage approach, we compare the model trained solely with supervised fine-tuning to the model trained with both stages. As shown in Table 4 the two-stage training improves accu- racy by 6.45%. 7.5%, and 13.37% on MATH-500, AMC, and GSM8K benchmarks, respectively. For out-of-domaal tasks, observe considerable improvements well, with average gain of 4.26%. These results demonstrate that the two-stage training strategy plays a vital role in enhancing problem-solving capabilities.
8

---

Published as a conference paper at ICLR 2026
5
RELATED WORK
5.1 CHAIN-OF-THOUGIT REASONING
The reasoning capabilities of Large Language Models (LLMs) were significantly advanced by Chain-of- Thhhuuhh (CoT) (Wei et al. sequence of intermediate steps to deconstruct complex problems before deriving final answer. Building on this, a new paradigm of slow-thinking systems like OpenAl's ol (Jaech et al.) 2024), DeepSeek-R1 (Guo al. performance on challenging tasks et al. 2025). These models employ extensive test-time com- putation, generating vast reasoning trajectories that are subsequently summarized into a concise answer. This process is typically optimized via Reinforcement Learning with Verifiable Rewards (RLVR) (Wen et al. 2025). where the model is rewarded based on the correctness of the final out- come. While highly effective, this slow-thinking paradigm is hampered by critical inefficiency: overthinking (Sui et al. 2025). Models often generate disproportionately verbose reasoning chains, replete with redundant steps and superfluous calculations, particularly for simpler problems.
2022 Chu et al. 2023 prompting, a technique that elicits
2025), and Qwen-QwQ (Team 2025) has emerged to further elevate
5.2 EFFICIENT REASONING
To mitigate the overthinking issue, prior work has explored several strategies to enhance the effi- ciency of reasoning LLMs (Feng et al. 2025). Some approaches focus on inference-time prompting. which guides models to generate more concise reasoning steps (Xu et al. 2025 Aytes et al. 2025)
or even bypass reasoning entirely by forcing a direct answer (Ma et al. (0025). Other methods seek to instill this efficiency more directly by fine-tuning models compressed reasoning chains. For in- stance, TokenSkip (Xia et al. 2025) and C3oT (Kang et al. (025) utilize trajectories containing only keywords, while PAUSE Token (Goyal et al. 2023) replaces entire reasoning chains with special "pause tokens." ICoT-SI (Deng et al. (024) internalizes reasoning chains through staged training on step-skipped datasets, enabling models to perform reasoning steps internlly rather than explicitly. A particularly prominent strategy employs reinforcement learning to explicitly penalize verbosity, typically by incorporating a length-based penalty into the reward function. O1-Pruner (Luo al.) 2025a), for example, introduces a harggtnihrrmiizinggg reward, while other works apply penalties for exceeding token budgets (Aggarwal Welleck 2025] Hou et al., 2025) or use dynamic penalties based answer correctness Yeo al. 2025). While effective, these methods still operate plicit, token-based reasoning steps. Distinct from methods that shorten explicit reasoning, another line of work explores latent reasoning (Chen et al. 2025). These approaches (Hao et al. 2024 [Saun
shi et al.] 2025| Wu et al.] (2025] Geiping et al.] [025]] Ruan et al.] (025) circumvent the generation of textual CoT steps by performing reasoning in latent, continuous space. This computation is executed by iteratively refining the model's hidden states without decoding them into text at each step. Our approach diverges from these work by not training latent reasoner from scratch. Instead, we adapt pre-trained, explicit reasoning LLM, empowering it to leverage the latent representations for computation without generating intermediate text.
CONCLUSION
In this work, we investigate the overthinking problem by conditioning the answers of reasoning LLMs on fragmental reasoning trajectories. Our analysis shows that these models can exploit salient information even when trajectories are highly degraded and exhibit high perplexity. We then model the reasoning trajectory as function of the input. Building on these insights, we introduce the Latent Reasoning Tuning (LRT) framework, which uses an auxiliary reasoning network to model the trajectory and encode it as compact latent representation. With two-stage training, LRT effectively guides models toward correct reasoning. Comprehensive experiments across multiple benchmarks validate its effectiveness. Notably, since method does not modify LLM parameters, it enables flexible switching between latent and explicit reasoning modes, offering a practical alternative to hybrid reasoning systems.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research is partially supported by National Natural Science
Foundation of China
(Grant No. 62372132) and Shenzhen Science and Technology Program (Grant No.
RCYX202210081928500777
REFERENCES
Pranjal Aggarwal and Sean Welleck. L1: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.0466 2025.
Simon A Aytes, Jinheon Back, and Sung Ju Hwang. Sketch-of of thought: Efficient Ilm reasoning with adaptive cognitive-inspired sketching. arXiv preprint arXiv:2503.05179, 2025.
Xinghao Chen, Anhao Zhao, Heming Xia, Xuan Lu, Hanlin Wang, Yanjun Chen, Wei Zhang, Jian Wang, Wenjie Li, and Xiaoyu Shen. Reasoning beyond language: A comprehensive survey on latent chain-of-thought reasoning. arXiv preprint arXiv:2505. 16782, 2025.
Zhengxiang Cheng, Dongping Chen, Mingyang Fu, and Tianyi Zhou. Optimizing length compres- sion in large reasoning models. arXiv preprint arXiv:2506. 14755, 2025.
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309.15402, 2023.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot implicit cot: Learning inter- nalize cot step by step. arXiv preprint arXiv:2405. 14838, 2024.
Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey. arXiv preprint arXiv:2504. 10903, 2025.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint drXiv:250..0517,, 2025.
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, arXiv preprint arXiv:2310.02226. 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12941 2025.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason continuous latent space. arXiv preprint arXiv:2412.06779 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
Bairu Hou, Yang Zhang, Jiabao Ji, Yujian Liu, Kaizhi Qian, Jacob Andreas, and Shiyu Chang. Thinkprune: Pruning long chain-of- thothutght of 1lms via reinforcement learning. arXiv preprint arXiv:2504.01296, 2025.
Hugging Face, Open rl: A fully open reproduction of deepseek-rl, January 2025. URL https ttt.. ng /opnerrer
10

---

Published as a conference paper at ICLR 2026
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.167100 2024.
Yu Kang. Xianghui Sun, Liangyu Chen, and Wei Zou. C3o:: Generating shorter chain-of -hhouught without compromising effectiveness. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 39, pp. 24312-24320, 2025.
Yunxin Li, Zhenyu L. Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: survey on large multimodal reasoning models. arXiv preprint arXi::2505.0422 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. Ol-pruner: Length-barmonizing fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv 2025a.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol ol-preveew with a 1.5b model by scaling rl, 2025b. Notion Blog.
Wenjie Ma, Jingxuan He, Charlic Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. arXiv preprint S.kiv:5504098555 2025.
MAA. American mathematics competitions. In American Mathematics Competitions, 2023.
Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng. Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv:241.009411 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.1939) 2025.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-prodt q&a bench- mark. In First Conference on Language Modeling, 2024.
Yangjun Ruan, Neil Band, Chris Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts. arXiv preprint urXiv:2503./8880 2025.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint 27X2:25022.77466 2025.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Na Zou, al. Stop overthinking: survey on efficient reasoning for large language models. arXiv preprint vrXiv22503.1611 2025.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https: http:///weelmmm...m htt.:iin....i....ooooo///o/ http://lll//ww-3323
qw3 32b/
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https::///// gig thub. com/ httm///ugggggaaa/ttata. 2020.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of thouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 -2-837, 2022.
11

---

Published as a conference paper at ICLR 2026
Xumeng Wen, Zihan Liu, Shun Zheng, Zhijian Xu, Shengyu Ye, Zhirong Wu, Xiao Liang, Yang Wang, Junjie Li, Ziming Miao, et al. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base 1lms. arXiv preprint arXiv:2506. 14245, 2025.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics.
Haoyi Wu, Zhihao Teng, and Kewei Tu. Parallel continuous chain-of- tf-hought with jacobi iteration. arXiv preprint arXiv:2506. 25Xi::2500./8882, 2025.
Heming Xia, Chak Tou Leong, Wenjie Wang, Yongqi Li, and Wenjie Li. Tokenskip: Controllable chain-of-thought compression in llms. arXiv preprint arXiv:2502.12067 2025.
Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint urXiv:2502.18600 2025.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388,, 2025.
Edward Yeo, Yuxuan Tong. Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain- of-thought reasoning 1lms. arXiv preprint arXiv:2502.03373, 2025.
Jingyang Yi, Jiazheng Wang, and Sida Li. Shorterbetter: Guiding reasoning models to find optimal inference length for efficient reasoning. arXiv preprint arXiv:2504.21370, 2025.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang. Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05/76 2025.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint 4rXhv:234.006344 2023.
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reason- ing by superposition: A theoretical perspective on chain of continuous thought. arXiv preprint 1rX1v:2505.12514, 2025.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
A LLM USAGE STATEMENT
In the preparation of this manuscript, LLMs (Large Language Models) were employed exclusively for text polishing and grammar checking, with the goal of improving readability. All technical ideas, methodological designs, mathematical formulations, and experimental results were conceived, implemented, and verified solely by the authors.
B EXPERIMENT DETAILS
Training Setup. We train our latent reasoning network eight NVIDIA A100 GPUs. The frame- work is implemented using the open-source TRL (von Werra et al. 2020) library. In the SFT stage, train the reasoning network for 3 epochs with batch size of 64 and learning rate of x 10-3, In the subsequent RL stage, we use batch size of 1024, generate 8 rollouts per question, and cap the maximum rollout length 2048 tokens. RL training is performed for 100 steps with learning rate of 1 x 10 5 and a KL. penalty coefficient of 2 x 10
Inference Setup. All inference experiments are conducted on a single NVIDIA A100 GPU to ensure fair comparison of efficiency. We set the generation temperature to 0.6 and top-p to 0.95. To evaluate performance specifically under efficient reasoning constraints, we adopt the budget- forcing implementation from S1 (Muennighoff et al.) 2025) and enforce the same token budget across all models. Specifically, for the DeepSeek-RT-De RT--istil 1024-token budgets, and for the Qwen3 series models, Disttill-Qwee--... model, we use 512- and 1024-token budget unless otherwise indicated in the tables. To ensure a strictly fair comparison of inference cost, all baselines and method evaluated using the same HuggingFace Transformers (Wolf et al., 2020) stack identical hardware. Consequently, the reported speedups reflect algorithmic improvements rather than differences in system-level optimizations or implementation details.
C ARCHITECTURE OF THE REASONING NETWORK
As described in Section the reasoning network G. is initialized from Qwen3-Embedding- 0.6B (Zhang et al., 2025). p. Here, we provide additional architectural details. In our framework, the input to Go consists of the hidden states of the base model rather than token embeddings; ac- cordingly, we remove the original embedding layer of G.. We introduce linear projection layer fin to map the base model's s hidden states into the input space of Go, and second projection layer sout to project the output Go back to the hidden dimension of the base model. The sequence length of the latent reasoning is determined by set of learnable vectors f2......... Åz]. which are optimized jointly during both the SFT and RL stages. Formally, the latent representations are produced as
2 foar (Go (H) @ (f..f2....,f....
(5)
where Hx denotes the hidden states of the base model for input X, and @ represents the Hadamard product with broadcasting. The reasoning network is not trained from scratch; instead, it is ini- tialized from the pre-trained Owen3-Embeddinng- -.... has been trained on large-scale multilingual and long-text corpora, provid- ing semantically rich representations and leading substantially more stable training.
-0.6B model. We adopt this initialization because
D COMPLEMENTARY EXPERIMENTS AND ANALYSIS
D.1 COMPARISON WITH LARGER BASE MODEL.
As shown in Table on the Qwen3-8B backbone, our method achieves an average score of 59.63%. outperforming the non-thinking baseline by 3.19% average. This demonstrates that approach is not limited to 1.7B or 4B models and can effectively enhance the reasoning capabilities of larger 8B-scale base models.
To further illustrate the performance characteristics of our method, we also compare it with the thinking mode of the Qwen3-8B model. As shown in Table the performance of Qwen3 (thinking
13

---

Published as a conference paper at ICLR 2026
Table Comparison Qwen3 and our method on Qwen3-8B, The terms non-thinking and thinking denote the standard and reasoning modes, respectively.
In-Domain Tasks
Out-of-Domain Tasks
Method
Budget
Average
AMC
MATH-500
GSM8K
GPQA
LSAT
non-thinking
46.25
77.05
73.88
46.21
38.80
56,44
1024
ours
50.25
78.20
91.02
44.43
34.26
59.63
1024
48.50
71.30
90.52
40.15
29.57
56.01
thinking
2048
51.25
83.50
90.54
48.23
39.67
62.64
4096
55.75
90.10
91.04
56.82
54.35
69.61
mode) steadily improves more token budget is allocated. When inference cost is not constraint, the thinking mode can achieve even higher accuracy. However, in scenarios where low latency and computational efficiency are essential, method provides superior alternative.
D.2 How REASONING TOKENS INTERACT WITH BASE-MODEL CAPACITY
Table 6:
Accuracy (%) of the latent reasoning method with varying numbers of latent tokens for larger base model (Qwen3-8B).
In-Domain Tasks
Out-of-Domain Tasks
Tokens
Average
AMC
MATH-500
GSMSK
GPQA
LSAT
256
50.25
78.20 78.50
91.02
44.43
34.26
59.63
512
50.75
92.49
44.95
33.15
59.97
there exists a "sweet spot" in the number of latent tokens. For the Qwen3- 1.7B base model, increasing the latent length from 256 to 512 results in a slight performance drop, indicating that 256 tokens already saturate the model's effective capacity. To further examine how model capacity interacts with latent-token length, we extend the analysis to larger base model. As shown in Table 6 the Qwen3-8B model continues to benefit from longer latent trajectories within this range: increasing the length from 256 512 improves performance marks and yields a small gain in the overall average. These results suggest that larger base models possess sufficient capacity to exploit richer latent information, enabling the same 0.6B reasoner to leverage longer latent sequences (e.g.. 512 tokens) and achieve higher performance. Consequently, the performance-lengle curve is expected to peak at a larger latent-token budget for larger models.
As shown Table 3
four of five bench-
D.3 COMPARISON OF INFERENCE EFFICIENCY
Table 7: Performance comparison of inference cost on the Qwen3-1.7B model. The terms non- thinking and thinking refer to the standard and reasoning modes, respectively. The symbol indicates that the computation accounts for the number of latent tokens.
Method Latency (sec/question) Throughput (tokens/sec)
Peak Memory (MB)
thinking non-thinking ours
71.09
40.53
7538
14.62
48.93
3946
11.79
51.31 (73.02))
6528
To quantify the efficiency gains of the latent representations, we measured the average inference latency, throughput and peak memory on 64 random MATH-500 problems using the Qwen3-1.7B base model. We compare method against the base model's thinking and non-thinking modes.
As shown in Table 7 our method achieves the lowest latency, even outperforming the non-thine non-thinking mode. This is because the reasoning network guides the base model to produce concise, direct
14

---

Published as a conference paper at ICLR 2026
answers, thereby reducing the number of decoding steps. In terms of throughput, method also delivers the highest effective throughput. Its standard token-level throughput is 51.31 tokens per second, and the effective throughput rises to 73.02 tokens per second when accounting for the 256 learned latent vectors processed parallel. As for memory usage, the peak memory of our method falls between the non-thinking and thinking modes. This overhead is primarily attributable to the one-time generation of the latent representations, which temporarily increases activation and KV- cache usage and contributes the higher throughput. After the latent representations are produced, decoding proceeds token-by-token as in the non-thinking mode, and memory consumption drops below this peak. These results confirm that our method delivers substantially higher information density per unit time while maintaining memory footprint that remains well below that of slow- thinking reasoning.
D.4 EMPIRICAL ANALYSIS OF LATENT REPRESENTATIONS
Table 8: Cosine similarity across latent representations of different benchmarks.
AMC
MATH-500
GSMSK
GPQA
LSAT
AMC
0.438
0.565
-0.173
0.104
-0.276
MATH-500
0.565
0.730
-0.223
0.141
-0.347
GSM8K
-0.173
-0,223
0,076
-0.051
0.070
GPQA
0.104
0.141
-0.051
0.149
-0.032
LSAT
-0.276
-0.347
0.070
-0.032
0.441
The latent representations not linguistically interpretable in the way Chain-of-Thought traces are, since they are not composed discrete tokens. While we have analyzed their functional role in Section 3 based the training objective, here further examine their empirical geometric struc- ture across different benchmarks. For each dataset (AMC, MATH-500, GSM8K, GPQA, LSAT), first average the latent representations over the sequence dimension obtain question-level latent vector. We then center these vectors by subtracting the global mean and compute the average pairwise cosine similarity between questions from any two benchmarks.
As shown in Table three consistent patterns emerge: domain clustering, semantic separation, and complexity stratification. Competition-style math datasets (AMC and MATH-500) exhibit the highest cross-dataset similarity as well as strong within-domaie similarity, suggesting that their la- tent representations are closely aligned despite covering different subdomains. LSAT exhibits pos- itive within-domain similarity (z 0.441) but strongly negative similarity to AMC and MATH-500 (-0.276 and -0.347, respectively), indicating that logic-style reasoning occupies a distinctly dif- ferent region of the latent space compared to olympiad-style mathematical reasoning. GSM8K and GPQA fall between these extremes: they show moderate self-similarity and relatively small cosine similarity with both competition math and LSAT, reflecting their hybrid reasoning characteristics.
Overall, these patterns indicate that the learned latent representations are organized by problem domain and difficulty, functioning compressed, task-specific instructions and reasoning cues that guide the base reasoning LLM.
D.5 STATISTICAL SIGNIFICANCE
Table 9: Mean + standard deviation of accuracy across benchmarks for DeepSeck-RI-Dissill--0 1.5B under the 512-token budget.
Benchmarks
AMC
MATH-500
GSM8K
GPQA
LSAT
Mean Accuracy
38.00
60.65
77.16
29.17
19.57
Standard Deviation (a)
+2239
+1.39
+0.78
+1.40
+1.1
the answers are generated stochastically rather than deterministically. To mitigate the resulting varaacce, we already sample multiple reasoning paths for each query and report the average accuracy over these stochastic decodings. Thus, the improvements report are not
As detailed in Section B
15

---

Published as a conference paper at ICLR 2026
artifacts of a single lucky sample, but reflect consistent performance gains. To further quantify the stability of our results and directly address the reviewer's ' concern regarding variability, we provide detailed statistical breakdown (Mean Standard Deviation) in Table 9 As shown, most datasets exhibit very low variance ( 0.8%-1.499 (144), with AMC showing slightly higher variance due to its substantially smaller test set. In all cases, the performance improvements of method exceed the corresponding standard deviations, indicating that the gains are robust rather than artifacts of stochastic sampling.
E DISCUSSION OF OTHER LATENT REASONING METHODS
Recent latent reasoning methods, including Coconut (Hao et 2024 and related work, share with our method the broad objective of reducing explicit chain-of of- generation by operating in latent space. The approaches. however, differ mechanism, reasoning horizon, and architectural design. Existing methods often rely on iterative refinement of recurrent hidden state, where the model repeatedly updates continuous latent representation before producing the next token. This design encourages local, stepwise form of latent computation (Zhu et al.) 2025). Our method fol- lows a different path. The reasoning network predicts an entire latent reasoning trajectory a single forward computation, producing sequence of latent vectors that represent the overall structure of the reasoning process rather than just the next refined state. This parallel formulation provides a larger expressive space and avoids entanglement with the base model's 's decoding loop.
The intended reasoning horizon also distinguishes the two approaches. Prior latent methods typically rely only small number of latent tokens, often fewer than ten, scale suited to short-form inference or simple deductive steps. Our method is aimed at long-form reasoning tasks, where the explicit chain-of- of-thougttt becomes substantially longer. Our method allows the reasoning process to be expressed in parallel and reduces the tendency of the model to generate unnecessarily long or repetitive chains, thereby mitigating the overthinking problem.
A further distinction lies in how reasoning is integrated into the model architecture. Prior latent rea- soning methods require retraining or substantial fine-tuning of the base LLM so that the model can internalize the latent computation. Our methods retains the base model entirely unchanged. The rea- soning ability resides in a lightweight auxiliary module that can be enabled disabled inference time. This modular structure preserves the base model's original capabilities and allows seamless switching between latent reasoning and explicit chain-of -f-thought generation without modifying or reloading weights.
Overall, these points outline the key similarities and differences between our method and prior latent reasoning methods, offering a clearer view of its role in the broader development of latent reasoning techniques.
16


Reviews:
[{"review_id": "21NJh9jVzO", "reviewer": "Reviewer_nNzk", "summary": "This paper introduces the Latent Reasoning Tuning framework, a novel approach to enhance the reasoning capabilities of Large Language Models (LLMs) without modifying their parameters. The method involves training a lightweight, external reasoning network (specifically, Qwen3-Embedding-0.6B) to generate a set of latent vectors (256 learned embeddings) that condition the frozen base LLM's output. The framework is trained in two stages: Supervised Fine-Tuning (SFT) on the OpenR1-Math-220k dataset, followed by Reinforcement Learning (RL) fine-tuning on the DeepScaleR-Preview-Dataset.\nThe authors evaluate their method on several math and general reasoning benchmarks, including MATH and GPQA. The results demonstrate that this approach consistently outperforms the base models (tested on 1.7B and 4B parameter models) as well as other efficient reasoning baselines.", "questions": "1. How interpretable are the reasoning vectors? \n2. Could you provide some insights on the scalability of LRT?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses the critical and practical problem of improving the inference efficiency of LLM reasoning.\n2. A key strength is the framework's efficiency. By only training a small reasoning network and keeping the base LLM's weights frozen, the method is computationally lightweight. This modular design also suggests high flexibility, as the reasoning network could potentially be paired with various pre-trained base models.\n3. The authors provide ablation studies to justify key design choices, such as the necessity of the two-stage training pipeline (SFT followed by RL).\n4. The method achieves consistent improvements over the selected baselines across multiple benchmarks, validating the effectiveness of the proposed latent reasoning approach.", "weaknesses": "1. One of the major drawback of the proposed method is the loss of interpretability of the reasoning traces. It would be great to provide some analysis on the learned reasoning vectors.\n2. Although results show consistent improvements on 1.7B and 4B models, it remains unclear how the reasoning network scale with base model sizes. e.g. would a 0.6B reasoning backbone still sufficient for much bigger LLMs? Table 3 shows that the performance degrade when the number of reasoning token increases from 256 to 512, which seems to indicate that there's a sweet spot for the number of tokens depending on the model capacities, but this needs further experiments with various model sizes to verify. \n3. The paper lacks an appendix and fails to provide crucial details about the hyperparameters used for the experiments. This omission significantly hinders the reproducibility of the work.\n4. Although the method avoids the cost of fine-tuning the base LLM, it introduces the training and inference overhead of an additional reasoning model. The paper would benefit from a more direct comparison of the trade-offs (e.g., total training flops, inference latency) against other latent-variable baselines that do involve fine-tuning the base LLM, which would provide a more complete picture of the method's efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XlJMRDGpdT", "reviewer": "Reviewer_dCpV", "summary": "This paper proposes a novel reasoning framework, latent reasoning tuning (LRT), which uses an auxiliary network to generate a sequence of latent representations in a single forward pass, and then concatenates the prompt and the latent representations to generate the final answer. Experimental results on various benchmarks show its consistent performance gain over baseline methods, showing its effectiveness.", "questions": "What is $P\\_{ref}$ in line 239 (since $P\\_{\\theta}$ is fixed during training) ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The idea of using latent representations for efficient reasoning makes sense.\n2. The method is clean and efficient for training.\n3. The performance on different benchmarks compared to baseline methods is good.\n4. It reduces the computational cost of generation compared to explicit CoT.", "weaknesses": "1. (minor) In Table 2, it would be beneficial to include the results for the thinking mode, which would make the table more comprehensive and the comparison more straightforward, allowing for a clearer view of the trade-off between computation and accuracy.\n\n2. It would make the paper stronger if the authors could try larger models (such as 7B models) for LRT to show the method is scalable for larger base models. This should be doable (at least for SFT only) since the LRT method only needs to train the reasoning network, which can be smaller than the base model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5VtnSguGyf", "reviewer": "Reviewer_RA4D", "summary": "This paper proposes Latent Reasoning Tuning (LRT), which replaces the explicit, token-by-token generation of reasoning trajectories in LLMs with compact latent representations. The authors show that reasoning LLMs can maintain high accuracy even when conditioned on fragmented reasoning paths, suggesting significant redundancy in explicit reasoning chains. LRT introduces a lightweight \"reasoning network\" G_phi that maps input questions to fixed-length latent trajectories, which then condition the base LLM to generate final answers.", "questions": "What is the exact architecture of the reasoning network G?\nWhat are the actual efficiency gains at inference time? \nAre the performance improvements statistically significant? Can you provide confidence intervals or significance tests?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The trajectory analysis provides compelling evidence that models are robust to token omission with minimal performance degradation.\nIt replaces O(k) autoregressive steps with a lightweight model with fixed reasoning size, which should provide efficiency gains.\nThe approach is modular, allowing switching between latent and explicit reasoning modes.\nThe results seem to show improvements over \"baseline efficient reasoning\" methods across several benchmarks.", "weaknesses": "Architecture of G is underspecified: I think there is no clear description of the reasoning network architecture. The paper mentions it uses \"Qwen3-Embedding-0.6B\" but it's unclear what the actual architecture is. The discussion says the latent reasoner is not trained from scratch, then it means it reuses some LLM parts? This was not clear to me when reading the paper.\n\nMissing efficiency analysis: One of the main motivations for compressing reasoning chains is computational efficiency, but the computational overhead of the latent reasoning is not discussed. No metrics are provided for inference time, FLOPs, or memory usage. Since the approach uses fixed reasoning length and lightweight modules, there should be efficiency gains, but these are not reported or analyzed. It could be a useful addition to the paper.\n\nNo statistical significance testing: All tables lack measures of variability (confidence intervals or standard deviations). This makes it unclear whether and when the improvements are statistically significant. This is important for rigorous science.\n\nConcurrent work: The paper mentions other latent reasoning work (Hao et al., 2024; Saunshi et al., 2025; Wu et al., 2025) in only one sentence of the related work section. It is important to better distinguish this work from other reasoning in latent space approaches. Even though most of them are fairly recent, the authors are aware of them and the discussion dismisses these related work too easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pTraAuNRtW", "reviewer": "Reviewer_gaFX", "summary": "This paper introduces Latent Reasoning Tuning (LRT), a framework that replaces explicit token-by-token reasoning with compact latent representations generated by an auxiliary network. It aims to improve reasoning efficiency by performing implicit reasoning without generating lengthy step-by-step rationales.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "+ The problem of overthinking and reasoning inefficiency is indeed practical and highly relevant for today’s LLMs, so it’s valuable to see it studied from this new latent-reasoning perspective.\n+ The writing is clear and easy to follow—the method and its intuition are well-explained, making the technical parts digestible even on a first read.", "weaknesses": "+ Even though reasoning efficiency is an important problem, modern inference engines (like vLLM or SGLang) already accelerate long-token generation with KV-caching and routing optimizations. The proposed module doesn’t directly integrate with these frameworks, so it might actually reduce efficiency when generating the same number of tokens—this needs to be considered for a fair comparison with baselines.\n+ Using only 512 tokens for distilled-R1 feels too artificial. For genuinely hard reasoning tasks (like AIME-style problems), original responses often exceed 10k tokens, so of course the performance drops drastically. It would make more sense to compare against short-response RL methods (like LC-RL, L1, or even RLVR with the same token cap) for a fairer setup.\n+ The 512-token setup feels too synthetic and not representative of real-world reasoning scenarios. Trying longer contexts—even if not as long as full 32k setups—would strengthen the empirical validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)