Learning to Reason over Continuous Tokens with Reinforcement Learning
Authors: Yiran Zhao
Yuhui Xu
Doyen Sahoo
Caiming Xiong
Junnan Li
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Hybrid Reasoning
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) have shown strong performance in complex reasoning tasks, especially when guided by Chain-of-Thought (CoT) prompting. However, conventional CoT reasoning in the discrete token space suffers from high computational and memory costs due to verbose intermediate steps. Recent work has explored latent reasoning in the embedding space to improve efficiency, but often at the cost of clarity and performance. In this work, we propose $\underline{Hy}$brid $\underline{Rea}$soning ($\texttt{HyRea}$), a unified framework that enables LLMs to dynamically switch between explicit (token-based) and latent (embedding-based) reasoning during inference. To train the model to make these decisions effectively, we introduce a two-stage training pipeline: (1) a supervised cold-start phase that introduces latent reasoning by replacing low-entropy CoT steps with embeddings, and (2) a reinforcement learning phase using Group Relative Policy Optimization (GRPO) to fine-tune the model’s reasoning strategy based on task-specific rewards. 
Experiments on mathematical reasoning benchmarks show that \texttt{HyRea} achieves significant reductions in token usage while maintaining or improving accuracy, offering an effective and scalable solution for efficient multi-step reasoning in LLMs.

Published as a conference paper at ICLR 2026
LEARNING TO REASON OVER CONTINUOUS TOKENS WITH REINFORCEMENT LEARNING
Yiran Zhao
Yuhui Xu
Doyen Sahoo
Caiming Xiong Junnan Li
Salesforce Al Research
ABSTRACT
Large Language Models (LLMs) have shown strong performance in complex reasoning tasks, especially when guided by Chain-of-Thought (CoT) prompting. However, conventional CoT reasoning in the discrete token space suffers from high computational and memory costs due to verbose intermediate steps. Recent work has explored latent reasoning in the embedding space to improve efficiency, but often at the cost of clarity and performance. In this work, we propose Hybrid Reasoning (HyRea), a unified framework that enables LLMs dynamically switch between explicit (token-based
and latent cembedding-based reasoning during inference. To train the model to make these decisions effectively, we introduce a two-stage training pipeline: (I) supervised cold-start phase that introduces latent reasoning by replacing low-entropy CoT steps with embeddings, and (2) reinforce- ment learning phase using Group Relative Policy Optimization (GRPO) to fine-tune the model's reasoning strategy based on task-specific rewards. Experiments on mathematical reasoning benchmarks show that RyRea achieves significant reduc- tions in token usage while maintaining or improving accuracy. offering an effective and scalable solution for efficient multi-step reasoning in LLMs. Our code is publicly available at hitttps:/ //ttt/bbgtttbb/tt/// com/ 2haoyiran924/Hyee.
INTRODUCTION
Large Language Models (LLMs) (OpenAl 2023: Grattafiori et al. 2024 Team et al. 2023 2024
Yang al. 2024a Comanici a 2025 OpenAl 2025) have shown impressive capabilities in
complex reasoning tasks (Wang et al. 2024 Hsiao et al. 2025 Shi et al. 2025 Qu et al.) 2025).
particularly when guided by Chain-of-Though (CoT) prompting Wei al.) 2022), which encourages
step-by-step intermediate reasoning. However, conventional CoT reasoning operates entirely in the discrete token space, leading inefficiencies in computation and memory due to verbose outputs. As long-context and cost-effective inference become increasingly important, especially in reinforcement learning (RI.) settings for math tasks (Liu al.) 2024), methods like Deepseek-R2 (Guo al. 2025) face high token costs and slow convergence. To tackle these challenges, recent approaches propose operating directly in the embedding space, bypassing tokenization altogether (Hao et al. 2024b Yue al. 2025 Zhang al. 2025). These latent reasoning methods offer substantial token compression and improved efficiency. However, reasoming over embeddings remains inherently difficult and can lead to degraded model performance compared to traditional ooken-based inference, certain tokens encode complex, nuanced information that cannot be fully preserved in compressed embeddings (Hao et al., 2024b Shen al. 2025). Moreover, current models lack the capability selectively determine w which tokens should be compacted and which should remain in their original form.
In this work, we propose novel Hybrid Reasoning (HyRea) framework that combines explicit (token-based) and latent (embedding-based) reasoning Explicit reasoning offers interpretability through step-by-step generation but is inefficient, while latent reasoning improves efficiency by operating in embedding space, often sacrificing clarity and performance. As shown in Figure [ī] HyRea enables L.Mss dynamically switch between these two modes during inference, selecting the most suitable reasoning strategy based on the context. This adaptive mechanism allows the model to maintain high accuracy while significantly reducing the number of generated tokens, yielding
Throughout this paper, use the term "embedding" to refer specifically the last-layer hidden state.

---

Published as a conference paper at ICLR 2026
[Answer]
Discrete Token
Embedding
Decoding Layer
Switch Token
[Question]
Feed Forword
Policy LLM
Advantage
Self-Attention
Feed Forward
12
Self Self-Attention
Embedding Layer
Fn
[Question]
Hybrid Reasoning
Rollout with Hybrid Reasoning
Figure I: HyRea framework overview. The model dynamically switches between explicit (token- based) and latent (embedding-based) reasoning during inference. Explicit reasoning provides in- terpretability through sep-by-step token generation, while latent reasoning operates in embedding space for greater efficiency. HyRea adaptively selects the optimal reasoning mode based on context, enabling a flexible and efficient hybrid approach.
better trade-off's between performance and efficiency. To support this hybrid reasoning capability, we introduce a two-stage training framework. In the supervised cold-start phase, the model learns to use latent reasoning by partially replacing intermediate CoT steps with continuous embeddings. To guide this replacement, we prioritize steps with low entropy, which are more deterministic and thus easier for the model to learn in latent space. This encourages the model to interpret and generate latent computations within broader reasoning trajectory. In the subsequent reinforcement learning phase, to further refine the model's 's decision Decision-mmaing process. Specifically, we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024), a RL technique that stabilizes learning by evaluating policies within sampled groups. This phase enables the model to learn when to invoke explicit versus latent reasoning based on downstream rewards, such as accuracy, format correctness, and efficiency. By integrating explicit and latent reasoning in unified and learnable framework, HyRea provides flexible and scalable approach to improve reasoning performance in LLMs, particularly in domains like mathematics where both precision and efficiency critical.
We conduct comprehensive experiments to evaluate the performance of HyRea across various models and tasks. Our compression method was evaluated on two widely used open-source models, Swen2.55-BBIInstrue and Swen2.5-33BBIInttie (Yang et al. 2024a), and we demonstrate that it can reduce length of the answer approximately 60% of its original length without any loss in mathematical reasoning capability. Specifically, on MATH-500 (Lightman et al.] (023), Minerva Math (Hendryeks al., 202T). AMC23 (AMC) 2023), and Olympiad Bench (He et al. 2024). HyRea achieves competitive accuracies while generating solutions with average of only 309, 419, 452, and 492 tokens under Owwn2.55-BB respectively. In contrast, Qwen2.5-7B-Ime -IIstruct trained using conventional reinforcement learning approach produces substantially longer outputs, averaging 698, 671, 892, and 854 tokens on the same benchmarks. Furthermore, we evaluate the model trained with Rea on general tasks beyond mathematics, including MMLU (Hendrycks al. (2020) and GPQA (Rein et al. 2024). The results demonstrate strong generalization ability and a significant reduction in output length.
INTEGRATE EXPLICIT AND LATENT REASONING
2.1 EXPLICIT REASONING
Explicit reasoning refers to the process by which a model generates reasoning outputs in an autore- gressive manner, decoding token time. Formally, given input query [x1 22, ** ,.))
2

---

Published as a conference paper at ICLR 2026
the model first maps each token to its corresponding embedding via the embedding layer, resulting in the sequence E = [e(x1), e(r2), ... e((z)))))) These embeddings are then processed by a series of Transformer blocks introduced by Vaswani et al. (2017):
He [1r 122... Ax] Transformer(E))
where h; denotes the hidden embeddings at the i-th position from the final layer of the Transformer. Furthermore, the hidden state ft is transformed into probability distribution over the vocabulary set through the language modeling head, denoted as LMhead- The next token $241 is then selected by taking the argmax over the resulting logits:
a++1 arg max (logits) arg max (T.Mheaa
V
The input is then updated to (2,, 02
Ir 1e+1), which is subsequently processed sequentially by
Equation I followed by Equation
2.2 LATENT REASONING
Instead of decoding token by token, Chain of Continuous Thought (Coconut) (Hao al., 2024b) proposes to bypass tokenization entirely by operating directly on embeddings. Specifically, during decoding, the model feeds the hidden output from the final layer back into the first layer as input. That is, rather than decoding the next token using Equation 2 the model proceeds follows:
Ho+1 [h1, h2. he, h+41] Transformer( he).
(3)
where E denotes the initial embedding sequence and h, is the hidden state time step t and || represents concatenation. Compared to explicit reasoning, operating directly on embeddings allows for a more compact representation during inference, potentially reducing token consumption. However, reasoning over embeddings remains inherently difficult and can lead to degraded model performance compared to traditional token-based inference, as certain tokens encode complex, nuanced information that cannot be fully preserved in compressed embeddings. This loss of semantic fidelity can be particularly detrimental in tasks that require precise symbolic manipulation, such as mathematical reasoning or code generation, where even small distortions in representation may lead to incorrect conclusions. Moreover, current models lack the capability to selectively determine which tokens should be compacted and which should remain in their original form. Compression is typically applied uniformly or based on fixed heuristics, without accounting for the varying informational value of different tokens within reasoning trace. This inability to adaptively balance compression and fidelity limits the effectiveness of latent reasoning and highlights the need for more principled, content-aware strategies for selective representation.
2.3 HYBRID REASONING
To improve the efficiency of the reasoning process while minimizing performance degradation, we aim for the model to flexibly switch between explicit and latent reasoning during inference. When decoding the next position, the model can autonomously decide whether to operate the embedding space or the token space ppate-thattio is, whether to perform latent reasoning or explicit reasoning. Formally, at position i, the model chooses to employ the explicit reasoning mode, it follows Equation P to generate the next token. Conversely, if the model opts for the latent reasoning mode, it applies Equation 3 inserting a <start-lad pentt-latennt ent token at the beginning and a kend-latents token at the end to mark the latent reasoning span. Formally, the ideal structure of hybrid reasoning can be represented as:
[Question] [Step,] wwt.kkttttttttttttt [1atent] <end-latent> [Step ... [Answer).
3 TRAINING METHOD
In this section, mainly introduce the training method enable the model conduct hybrid reasoning.
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Hybrid Reasoning Training (HyRea)
Input: Pretrained language model L.M,, CoT training data DSFT, RL training data DRL max latent steps S. group size G. clipping threshold E
Output: Hybrid reasoning model LCM
// Stage 1: Cold Start Supervised Fine-Tuning
2: for from 1 to S do
for each batch (g. a) in DsFt do
4: Identify reasoning steps with low entropy
5: Replace selected steps with |sstartlateent> latent <end-latent>
6: Compute loss LSFT = log LLM(( [latent])
7: Update LCM using gradient descent LSFT
8: end for
9: Incrementally introduce 10% new data into DsFT at each iteration
10: end for
11: // Stage 2: Rei deineeeet Learning with GRPO
12: for each query g in DRL do
13: Sample G' outputs {01, 02,..., .2. ~ LLMaua-ll
14: Evaluate rewards (ri..... (aceuracy, formatting, latent usage)
15: Compute GRPO loss by Equation 6
16: Update LCM using gradient descent on LGRPO
17: end for
3.1 COLD START FOR REPLACE LOw-ENTROPY STEPS
To enable the model to perform hybrid reasoning, we adopt cold-start approach inspired by Deepseek-R1 (Liu al. 2024 In this initial phase, the model is trained using supervised fine-tuning (SFT) on chain-of-thought (CoT) (Wei et al., 2022) reasoning data to learn preliminary switching capabilities. This serves foundational step toward developing more advanced hybrid reasoning abilitics. Specifically, the original CoT data is structured as a sequence:
= [Question] [Stop,
[Answer].
To introduce latent reasoning, we select steps that have low entropy and replace them with latent segment [Latent] number of latent tokens. This creates a hybrid sequence which [Latent] serves as placeholder for latent reasoning. Importantly, the entropy-based replacement strategy prevents the model from compacting tokens that encode complex or critical information that cannot be faithfully represented by latent embeddings. During training, the loss is computed only over the visible (non-latent) tokens, ensuring that the model focuses on learning the interpretable parts of the reasoning trace while implicitly handling the latent segments.
[1atent] kend-latents where denotes the
Loold start = - log LCM [Latent]),
where C.CM((-) denotes the language model's likelihood function. In addition, the number of steps replaced by [Tatent] increases progressively during training, starting from 0 and gradually reaching a predefined maximum threshold S.
3.2 REINFORCEMENT LEARNING
We mainly employ the Group Relative Policy Optimization (GRPO) proposed by Shao et al. 2024
LCRPO (f)
E. pq))((), ~T
(a 9) A,, clip Ta (o: 4)
To(a; g)
A: }
(6)
min
8, 1+ A;
T
Ta (0: 4)
where
A r mean (11, ... ,ro)))
(7)
std((ri,T2, rr)))
4

---

Published as a conference paper at ICLR 2026
and reward consists of both format reward and an accuracy reward, well as latent reward. Specifically, we use the latent reward to guide the model in generating the token [Latent]. Addi- tionally, during loss computation, the remaining token steps. The overall algorithm is further illustrated in Algorithm T Through reinforcement learning, this approach eliminates the need for manually constructing synthetic data to train the model's 's switching capability. Instead, the model learns to perform hybrid reasoning in self-supervised manner, guided by reward signals that reflect both correctness and efficiency.
exclude the [Latent] token and calculate the loss only over
EXPERIMENT
4.1 EVALUATION SETUP
Datasets We primarily utilize two large-scale mathematical reasoning datasets for cold start:
Question
NuminaMathhCof (LI al. 2024), which con- tains 860k diverse math problems ranging from high school exercises international olympiad- level questions formatted in CoT style, and MetaMathQA (Yu et al. 2023). comprising 390k coobumn-oouutin pairs enhanced through vari- ous data augmentation techniques to promote diverse and robust reasoning pathways. Specifi- cally, we split the answer by "/n' and '., and en- sure that each equation is kept as independent step. Figure 2 presents a concrete example of the processed dataset. Additionally, in the rein- forcement learning stage, we adopt Math-121 challenging math dataset derived from (Light-
How mony even numbers are greater than 202 and lass than 4052
Steps
1. "The smallest even rumber greater than 202 and less than 405 204",
2. "The largest even number greater than 202 and less than 405 404',
even
range,
snallest rumber the largest number and divide by than edd I becouse we include both endpoints",
4. "So, the number of even numbers this range $(404-
204/2-1:201/2+
5-1=101.55*
fraction of resrest whole number".
"Since can't
round
"Thenefore, the number even rumbers grester than 202 and leas
403 $1bexed
Figure An example data point.
man et al. (023).
Backbone Models We evaluate two widely-used open-source L.LMs as backbone models. Qwen2.5- IB-Instruct and Qwen2.5-32B-Inm
(Yang et al. 2024a).
Baselines. We employ several state-of-the-art efficient reasoning method with less tokens (i)
CoT (Wei al. 2022); (ii) Reinforcement Learning (Guo al. 2025). which optimizes reasoning strategies through tril--and-rrov by rewarding accurate outputs; (iii Chain of Continuous Thought (Coconut) (Hao al. 2024b), novel paradigm that replaces discrete tokens with latent "continuous thoughts" by feeding the last hidden state directly back into the model, allowing reasoning in an unconstrained latent space and enabling breadth-first exploration over multiple reasoning paths; (iv) Soft thinking (Zhang al.) 22025), a training-five free method that mimics human-like soft reasoning by generating probability-weighee ambiguity and enabling more abstract, flexible reasoning with significantly fewer tokens.
concept tokens a continuous concept space, capturing semantic
Benchmarks To assess mathematical reasoning proficiency, we employ four benchmarks: MATH- 500 (Lightman et al. 2023), a 500-question subset of the MATH benchmark curated for rigorous; Minerva Math (Hendrycks et al. 2021); Olympiad Bench (He et al., 2024) as well as competition- level benchmarks such as AMC22 (AMC) 2023). We evaluate the performance of the model on each dataset using pass@ @ 1. Additionally, we analyze the length of each generated answer by counting the total number of tokens, including both standard tokens and the [Latent] token. Furthermore, we assess the frequency of switches between latent and explicit reasoning within each response.
For the cold-start, we employed the LLaMA-Factory library ZZeng et al. 2024), widely adopted GitHub-hosted framework for efficient mrgel fine-tuning, carry out all training procedures. Experiments are conduced on eight 140GB NVIDIA H200 GPUs, with learning rate as 10 global batch size as 32. Furthermore, reduce memory consumption during training, we applied ZeRO Stage-2 optimization and gradient checkpointing, both provided
Implement Details
5

---

Published as a conference paper at ICLR 2026
Table Main results of HyRea across four math reasoning benchmarks. We report accuracy (Acc), average number of output tokens (#Tokens), and average number of latent-explicit switches (#Switch).
Math-500
Minerva Math
AMC23
Olympiad Bench
Methods
Acc #Tokens #Switch
Acc
#Tokens #Switch
Acc #Tokens #Switch
Acc #Tokens #Switch
SFT
73.6
546
26.1
619
0
36.1
845
29.3
723
0
Qwen2557B-it
SFT RL
84.2
698
26.8
671
0
48.2
892
40.0
854
0
Coconut
70.4
106
22.1
174
1
33.7
217
26.8
296
1
Soft Thinking
66.4
617
16.9
604
24.1
784
24.7
595
1
HyRea w/o RL
71.8
248
20.6
288
3.1
34.9
339
4.9
28.5
378
4.3
ByRea
83.6
387
27.2
425
3.6
48.2
4.7
39.6
583
3.8
SFT
83.6
595
37.1
644
0
55.4
803
49.0
866
0
Owen2.5-32B-I
SFT RL
85.2
588
39.7
608
0
61.4
905
49.5
899
0
Coconut
79.8
179
32.4
209
1
53.0
285
46.2
403
1
Soft Thinking
82.4
574
33.4
602
1
55.4
776
43.7
887
1
HyRea w/o RL
79.8
305
3.2
34.6
267
3.9
54.2
304
5.2
47.3
428
4.7
HyRea
84.4
369
4.1
38.6
381
3.7
57.8
498
4.9
48.9
563
5.3
by the DeepSpeed library. Note that in the cold-start stage, the loss on <start-latent- and <end-latent> are scaled by a factor of 4 to emphasize their importance and help the model learn when to switch. For reinforcement learning training, we use the EasyR1 (Zheng et al. 2025) framework built on verl (Sheng al., 2024), with specialized support for VLMMs. Experiments are conducted using eight 140GB NVIDIA H200 GPUs with global batch size of 128, rollout batch size 128, a rollout temperature of 1.0, a consistent learning rate x 10 and rollouts.
4.2
MAIN RESULT
HyRea achieves strong accuracy-efficiency trade-offs. As shown in Table our proposed hybrid reasoning framework HyRea consistently delivers strong performance across all four math reasoning benchmarks, demonstrating both high accuracy and efficient token usage. Under the Qwen2.5-7B- Instruct, HyRea achieves an accuracy of 83.6 on MATH-500, nearly matching the SFT+RL bascline (84.2), while using less than half the number of output tokens (387 vs. 698). Similar trends are observed across other benchmarks: on Minerva Math, HyRea slightly outperforms SFT+RL (27.2 vs. 26.8) with a substantial reduction tokens (425 vs. 671); and for AMC23 and Olympiad Bench, it matches or closely approaches the best-performing baselines in accuracy (48.2 and 39.6, respectively), while reducing token usage by nearly 50%. When scaled up to
HyRea continues to show competitive results, achieving 84.4 accuracy MATH-500 with only 369 tokens, compared to 85,2 accuracy and 588 tokens for SFT+RL. On Minerva Math and AMC23, HyRea reaches 38.6 and 57.8 accuracy, respectively, again using significantly fewer tokens than the SFT+RL counterpart. Notably, HyRea also exhibits meaningful latent-explicit switching behavior, averaging 3.6 5.3 mode transitions per sample depending on the dataset and model size, compared to zero switches for all baselines and only one for Coconut and Soft Thinking. This switching capacity reflects HyRea's ability to dynamically leverage both explicit and latent reasoning strategies. Importantly, HyRea outperforms token- efficient approaches like Coconut, which, despite producing shorter outputs (e.g., 106-179 tokens on MATH-500) 500), suffers from significant accuracy degradation across all tasks. The ablation study (HyRea w/o RL) further highlights the benefits of reinforcement learning, as performance notably drops when RI. removed. Overall, these results underscore HyRea's a's efficctiveness compact solutions via strategic reasoning mode transitions.
token-efficient
scalable solution for multi-step reasoning achieving high accuracy with
ccasoving-achievingg
Complementing this, Figure 3 shows that during training, the accuracy and latent rewards steadily improve and stabilize, while the format reward remains consistently high. This indicates that HyRea learns to balance correctness, structure, and reasoning mode usage effectively. Overall, these results underscore HyRea's effectiveness as a scalable solution for multi-step reasoning-aceee achieving high accuracy with compact solutions via strategic reasoning mode transitions.

---

Published as a conference paper at ICLR 2026
Format
Accuracy
Latent
0.35
135
0.0
0.85
1330
0.6
0.55
0.36
120
320
Figure 3: Reward over steps.
Question
Table 2: Comparison of entropy based replacement and
If rectangle has length of 12 meters and width of meters, what its area and perimeter?
random sampling replacement.
Output
Math
Minerva
AMC23
Olympiad #Token Acc
-start-latenb-leree teetthllotentxeen--oteett length width square sstort-lareet---ehetteeddd (lenghh- width) (12 8)
Acc #Token
Acc #Token Ace
#Token
SFT+RL
84.2
698
26.1
619
48.2
892
40.0
854
Random
83.4
309
26.5
419
49.4
452
39.6
492
Entropy
83.6
287
27.2
372
48.2
426
39.6
483
Figure 4: Concrete Example.
Concrete Examples Figure exemplifies HyRea's ability abstract away low-utility, generic reasoning steps into latent space, effectively bypassing verbose yet semantically redundant content. By retaining only semantically salient computations and final outputs in explicit token space, the model achieves more compact and efficient reasoning trace without compromising interpretability. This demonstrates HyRea's capacity to balance informativeness and conciseness through dynamic reasoning mode selection.
4.3 ABLATION ANALYSIS
Random Replacement Table compares two strategies for selecting intermediate reasoning steps to replace with latent embeddings: random sampling significantly reduce token usage compared to the CoT+RL baseline while achieving comparable or better accuracy. Notably, the entropy-basee method consistently produces shorter outputs across all benchmarks, with token counts of 287 on MATH-500, 372 on Minerva Math, 426 on AMC23, and 483 on Olympiad represeipnn the lowest token usage among all methods evaluated. In contrast, random replacement uses slightly more tokens (c.g., 309, 419, 452, and 492, respectively). suggesting that entropy- based selection produces more compressible reasoning paths. terms of accuracy, entropy-based replacement also demonstrates greater stability. On Minerva Math, it achieves an accuracy of 27.2 compared to 26.5 for random, and on MATH-500 and Olympiad Bench, it matches slightly outperforms random selection. Although both strategies come close to the accuracy, entropy-based replacement offers a more favorable trade-off between performance and efficiency. These results validate our design choice of using entropy principled criterion for identifying deterministic and compressible reasoning steps to encode in latent space.
entropy-based selection. Both strategies
CoT+RL performance
As shown in Figure entropy-based sampling significantly outperforms random sampling during the cold-start stage. It achieves faster performance gains, surpassing 80 points within 10 iterations, while sample-based selection improves more slowly and plateaus around 75. This demonstrates that uncertainty-awant sampling more effectively prioritizes informative examples early in training, accelerating convergence and yielding better final performance.
Replace Number of Latent Figure presents ablation study the number of latent replace- ments, showing how varying the parameter c affects both accuracy and output length. As c increases from 8, accuracy (purple line) drops sharply- from over 80% below 10% (0%-indicaingg that
7

---

Published as a conference paper at ICLR 2026
Entrepy
50
Hybrid Coconut (Sampla) Hybrid Cocanut (Entropy)
Figure 5: Cold-start stage: sarrppy-based vs. random sampling.
Figure 6: Concrete Example.
Table 3: Generalization ability HyRea trained model.
Accuracy (50
Lenggh
00-
100
MMLU
GPQA
30
Acc
#Token
Acc
#Token
SFT
70.2
84
23.7
837
SFT+RL
73.4
102
29.8
1083
latent) [latent] (end latent)
HyRea
68.6
53
27.4
685
Figure 7: Ablation of replace number of la- tent.
replacing more latent components substantially degrades model performance. Conversely, output length (blue line) increases significantly, particularly after e 4, suggesting that excessive replace- ment leads to verbosity or redundancy. This inverse relationship between e and performance suggests that the training stability of HyRea and hybrid reasoning deteriorates rapidly as increases.
FURTHER ANALYSIS
5.1 SWITCHING PATTERN
Figure 6 provides fine-graing view of HyRea's reasoning behavior via entropy decoding steps. We observe that latent reasoning steps (marked with red crosses) tend to appear in low-entropy regions, indicating high model confidence in selecting the latent mode. Notably, these latent spans often occur at the beginning or end of the reasoning trajectory, suggesting HyRea prefers to compress either the problem setup the final answer derivation when confident. Additionally, switching between modes is relatively infrequent, aligning with our earlier observation of around 3-5 switches per sample. However, each latent span typically contains multiple latent steps rather than isolated calls, indicating that HyRea strategically groups latent reasoning into longer, coherent segments. This behavior highlights HyRea' ability balance interpretability and efficiency by using explicit reasoning in uncertain mid-process steps, while leveraging latent modules for confident, compressed segments.
5.2 ASSESSING MODEL GENERALIZATION CAPABILITY
We evaluate the generalization ability the HyRea- trained model on two benchmark datasets: MMLU (Hendrycks al.) 2020) and GPQA (Rein et al. 2024). As shown in Table 3 HyRea compared against models trained with supervised fine- ton-tunnng (SFT) and reinforcement learning RSFT+RL). While SFT+RL achieves the highest accuracy (73.4 MMLU and 29.8 GPQA),
8

---

Published as a conference paper at ICLR 2026
generates substantially longer responses (102 and 1083 tokens, respectively). In contrast, HyRea produces more concise outputs (53 tokens on MMLU and 685 on GPQA) while maintaining competi- tive aceuracy (68.6 and 27.4). This demonstrates Hy ByRea's 's strong generalization ability to untrained domains, balancing performance and efficiency without task-specific optimization. Such generaliza- tion is crucial for deploying language models in diverse, real-world settings where robustness and adaptability are essential.
6 RELATED WORKS
Mhain-of-Thoughh Reasoning Chain-of-Thougie (CoT) prompting (Wei et al. 2022) improves model performance by guiding the generation of intermediate reasoning steps. Such behavior can be elicited directly through prompt engineering without additional training (Khot et al. 2022 Zhou
al.) 2022). To further enhance reasoning quality, recent works employ supervised fine-tuning and reinforcement learning to explicitly optimize multi-step reasoning (uue et al. 2023 Yu et al.
2023 Wang et al. 2023 Xiong et al. 2025). An important extension of CoTT involves integrating
with tree search algorithms (ao et al. [023] |Ho et al.] 2024a] Xie et al.) 2023 Liao et al. (025b),
allowing models to explore and evaluate multiple reasoning paths to achieve better performance on
complex tasks. These advances have been formalized into test-time inference scaling laws (Snell
et al. 2024; Wu et al., 2024). which provide theoretical grounding for the observed performance improvements. The success of advanced reasoning models such OpenAl's 01 (OpenAl et al. (024) and DeepSeek's 's R1 (DeepSeek-Al Al et al. 2025) has further intensified interest in test-time search (TTS) techniques. However, the deliberative nature of these methods often leads to inefficiencies in token usage. As result, reasoning efficiency has become an increasingly important research
focus (Liu al. 2025 Xu al. 2025 Liao et al. 2025a Aggarwal & Welleck 2025).
Latent Reasoning Recent advances in LLM reasoning in latent space (Yang al. 2024b Biran et al. 2024) underscore the significance of hidden computations. To better leverage these latent dynamies, range of methods introduce special tokens to explicitly guide reasoning. For instance, <pause> tokens Goyal et al. (2023) are incorporated during both pretraining and downstream finetun- ing, yielding consistent improvements over standard training. Similarly, non-semantic filler tokens (e.g., "...' Pfau et al.] (2244) have been shown to enhance performance certain reasoning tasks. Several works explore alternatives to explicit chain-of-thouglee (CoT) prompting. Yu et al. (2024) proposes implicit CoT, where models internalize intermediate reasoning steps distilled from explicit CoT supervision. Wang et al. (2023) introduces a hierarchical reasoning structure guided by plan- ning tokens, which demarcate latent reasoning stages. More recently, HHoo et al. (2024b) proposes COCONUT, replacing discrete CoT tokens with continuous latent embeddings, leading to improved reasoning across multiple tasks. Zhu et al. (2025) provides theoretical justification for the superiority of continuous CoT's over their discrete counterparts. In contrast to these approaches, our work focuses on selectively replacing low-entropy tokens with continuous embeddings, allowing the model to dynamically switch between discrete and continuous reasoning modes. We formulate this switching behavior as reinforcement learning problem, enabling the model learn when and how to invoke latent computation for improved reasoning quality.
7 CONCLUSION
We presented HyRea, a hybrid reasoning framework that allows large language models to dynamically alternate between explicit token-based reasoning and latent embedding-basse based reasoning. aiming to balance interpretability, accuracy, and efficiency. To train this capability, we proposed a two-stage approach that begins with entropy-guided supervised fine-tuning, where low-entropy reasoning steps are selectively replaced with latent representations, followed by reinforcement learning using Group Relative Policy Optimization to refine the model's reasoning strategy based on task-specific rewards. Extensive experiments on challenging mathematical benchmarks demonstrate that HyRea achieves substantial reductions in output length while maintaining or improving accuracy compared to strong baselines. Furthermore, HyRea exhibits strong generalization ability mon-mathematical tasks, showing its potential as a versatile and scalable solution for efficient multi-step reasoning in large language models.

---

Published as a conference paper at ICLR 2026
REFERENCES
Mathematical association of america. american mathematics competitions (ame). 2023.
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint vrXiv:2503.46697 2025.
Eden Biran, Daniela Gottesman, Sohee Yang. Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406. /2775, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjee Ji. Erhang Li, Fangyun Lin, Fucong Dai. Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. I.. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen. Shanghao Lu, Shangyan Zhou. Shanhuang Chen, Shengfeng Ye. Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, Wang, Wangding Zeng, Wanjia Zhao Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nic, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He. Yunfan Xiong. Yuxiang Luo. Yuxiang You, Yuxuan Liu, Yuyang Zhou. Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang. Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu WU, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability 1lms via reinforcement learning, 2025.
URL htttt::///attttaa otg/ abs / 2501. 12948
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprint arXiv:2310.02226, 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 2rXiv:2501.129455 2025.
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Llm reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. arXiv preprint 4404 05221, 2024a.
10

---

Published as a conference paper at ICLR 2026
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769,
202444 2024b.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- bilingual multimodal scientific problems. In ACL (1). 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Vincent Hsiao, Morgan Fine-Morris, Mark Roberts, Leslie N Smith, and Laura M. Hiatt. A critical assessment of LLMs for solving multi-step problems: Preliminary results. In AAAI 2025 Workshop LM4Plan, 2025. URL
raootMMI
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint ar arXiv:22/0.02406, 2022.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong. Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. [https://uugiin
co/
aimo http://www...tttaaoo..oaaaaaaaaa
2024.
Baohao Liao, Hanze Dong, Yuhui Xu, Doyen Sahoo, Christof Monz, Junnan Li, and Caiming Xiong. Fractured tfain-of-thought reasoning. arXiv preprint arXiv:2505.12991 2025a.
Baohao Liao, Yuhui Xu, Hanze Dong, Junnan Li, Christof Monz, Silvio Savarese, Doyen Sahoo, and Caiming Xiong. Reward-guicke speculative decoding for efficient 1lm reasoning. arXiv preprint arXiv:2501.19324 19324. 2025b.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 2024.
Yue Liu, Jiaying Wu, Yufei He, Hongcheng Gao, Hongyu Chen, Baolong Bi, Ruihan Gong, Jiaheng Zhang, Zhiqi Huang, and Bryan Hooi. Efficient inference for large reasoning models: survey, arXiv preprint arXiv:2503. 23077, 2025.
OpenAl. Gpt-4 technical report, 2023.
OpenAL. Gpt-5 system card. Ectps://cdn.oot hatpa//ricccennttt--sttttteeeeee pd
August 2025. Accessed: 2025-08-13.
OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair. Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman. Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts,
11

---

Published as a conference paper at ICLR 2026
Daniel Kappler. Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li. Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace. Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao. Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Igge Akkaya, Ilya Kostrikov, llya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng. Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang. Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowd- hury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay. Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Floundou, Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong. Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, and Zhuohan Li. Openai ol system card, 2024. URL // arxiv............ org. abs / 24 12442114 22. 16720
Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXivv2404.15758 2024.
Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-rong Wen. Tool learning with large language models: a survey. Frontiers of Computer Science. 19(8). January 2025. ISSN 2095 2236. doi: 10. 10.100//ss110 1704-024-40678- doi orq/10 111.1100000/000222211
URL
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman, Gpqa: A graduate-level google-proof q&a benchmark. In First Conference Language Modeling, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.033.2 2024.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing 2025.
into continuous space via self-distillation. arXiv preprint arXiv:2502.21074,
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 2409. /9256, 2024.
Zhengliang Shi, Shen Gao, Lingyong Yan, Yue Feng, Xiuyi Chen, Zhumin Chen, Dawei Yin, Suzan Verberne, and Zhaochun Ren. Tool learning in the wild: Empowering language models as automatic tool agents, 2025. URL https::////........ //wrx/vv.vvvv///////a/ org/ abs 2405. 6533
12

---

Published as a conference paper at ICLR 2026
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters, 2024. URL httpttttt///rttttt org. abs/
1408800312
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0m 08299, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Chaojie Wang, Yanchen Deng, Zhiyi Lyu, Liang Zeng, Jujie He, Shuicheng Yan, and Bo An. Q Improving multi-step reasoning for Ilms with deliberative planning, 2024. URL https //axxiv http://.rggbbbb//24....
Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessan- dro Sordoni. Guiding language model reasoning with planning tokens. arXiv preprint arXiv:2310.05707, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 2124872-44877, 2022.
Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2024. URL
wttp:/gg/aabba//ae/b 00724
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information P Processing Systems, 36:41618---- 41650, 2023.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang. Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504. / 1343, 2025.
Yuhui Xu, Hanze Dong. Lei Wang. Doyen Sahoo, Junnan Li, and Caiming Xiong. Scalable chain of thoughts via elastic reasoning. arXiv preprint arXiv:2505.055: 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 21X1v:2412.15115, 2024a.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:2402. 16837, 2024b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 1188801-88222 2023.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.1222 2023.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407. 00023, 2024.
13

---

Published as a conference paper at ICLR 2026
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023.
Zhenrui Yue, Bowen Jin, Huimin Zeng, Honglei Zhuang, Zhen Qin, Jinsung Yoon, Lanyu Shang, Jiawei Han, and Dong Wang. Hybrid latent reasoning via reinforcement learning. arXiv preprint arXiv:2505.18454, 2025.
Zhen Zhang, Xuehai He, Weixiang Yan, Shen, Chenyang Zhao, Shuohang Wang. Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of Ilms in continuous concept space. arXiv preprint arXiv:2505.1577 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics.
URL http::/atxii..oo//
abs 2203 13372
Yaowei Zheng. Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality training framework. https://github.co com/
hi miyouga/EasyR]
2025.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint vrXiv:2255.00625 2022.
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by superposition: A theoretical perspective on chain of continuous thought. arXiv preprint 1rXiv:2505.12514. 2025.
14

---

Published as a conference paper at ICLR 2026
LLM USAGE
We used LLMs as general-purpose writing and debugging assistants. Specifically, LLMs were employed to help polish the writing (e.g., improving sentence clarity, grammar, and flow) and occasionally to assist with debugging minor implementation issues (c.g.. identifying syntax errors or suggesting code refactoring). However, all core ideas, research questions. methodological designs, codebase implementations, experiments, and analyses were entirely conceived, developed, and conducted by the authors. No part of the intellectual contribution, experimental framework, or scientific reasoning was generated by an LLM.
LIMITATION
While RyRea demonstrates strong performance across mathematical reasoning tasks and generalizes well to other domains, several limitations remain. First, the effectiveness of latent reasoning depends on the quality of entropy-based step selection; inaccurate entropy estimates may lead to the com- pression semantically important steps, resulting in performance degradation. Second, the current approach assumes access to well-structured Chain-of-Tie Thouugh data, which may not be readily avail- able in all domains. Third, although reinforcement learning improves the model's switching behavior, it introduces additional training complexity and computational cost. Lastly, while HyReacv reduces output length, it does not yet support interpretability of latent steps, which could hinder transparency in high-stakes applications. Future work may explore more adaptive selection mechanisms, improved interpretability of latent reasoning, and extensions to tasks with less structured reasoning formats.
15


Reviews:
[{"review_id": "hWu7hBLAIz", "reviewer": "Reviewer_wezr", "summary": "The authors propose HyRea that lets LLMs adaptively alternate between explicit token-level reasoning and latent embedding-level reasoning, with special tokens marking the start and end of the latent span. The training has two stages: \n\n1. Cold-Start Supervised Fine-Tuning (SFT):\nLow-entropy CoT steps are replaced with latent spans to introduce embedding-based reasoning while preserving interpretability.\n\n2. Reinforcement Learning (RL) with Group Relative Policy Optimization (GRPO):\nFine-tunes the model to learn optimal switching between reasoning modes, guided by task-specific rewards for accuracy, format, and efficiency.\n\nExperiment results on math benchmarks, MMLU and GPQA show the performance of HyRea is in between vanilla SFT and SFT + RL, with significantly shorted generation.", "questions": "See weeknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a elegant solution of alternating latent and explicit reasoning with special tokens + cold start SFT + RL.\n\n2. The performance of HyRea on math benchmark is close to vanilla SFT+RL, while the generation length is significantly shorter.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. There is only one latent CoT with short generation baseline (COCONUT), which was out last year, and is a weak baseline. The soft thinking baseline is training-free and does not shorten the generation, which does not align with the settings in the paper. It's a bit hard to calibrate the contribution of HyRea without suitable baselines. I'd suggest the authors to include more recent, stronger baselines that match their settings (training latent CoT to shorten generation). e.g. CODI (https://arxiv.org/abs/2502.21074). There are also quite a number of papers that produce mixed CoT chains. e.g. Token Assorted (https://arxiv.org/abs/2502.03275). RL with latent tokens: HRPO (https://arxiv.org/abs/2505.18454).\n2. Please include a detailed description of the reward functions (accuracy, format, latent) in the main text.\n3. If I understand correctly, the math-trained HyRea can perform reasonably well out-of-domain on MMLU and GPQA according to Table 3. Is SFT and SFT+RL also trained on math and tested OOD? It would be better to include other latent reasoning baselines to better calibrate the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DHK0u2VHG1", "reviewer": "Reviewer_Nae9", "summary": "This paper focuses on making reasoning with LLMs efficient by utilizing both discrete and continuous chain-of-thought (CoT) tokens. Towards this, the paper proposes Hybrid Reasoning (HyRea) that enables the LLM to adaptively switch between discrete CoT and continuous CoT during generation. HyRea consists of two phase training involving 1) a supervised cold-start phase and 2) an RL phase. By evaluating HyRea on various standard benchmarks, the authors show that HyRea exhibits a favorable trade-off between the model quality/accuracy and efficiency in terms of thought token lengths. Furthermore, the authors also demonstrate the good out-of-domain generalization beyond mathematical tasks for HyRea.", "questions": "- Could the authors provide examples similar to Figure 4 but with longer CoTs? Do the HyRea produce CoTs remain interpretable?\n- In Line 365, why are authors referring to CoT + RL as **upper bound** in accuracy? According to Table 2, this upper bounds is often exceeded by Random and/or Entropy.\n- While Figure 5 shows superiority of Entropy for cold-start stage, looking at Table 2 and Figure 5, Random appears to be more amenable to RL as the gap between Random and Entropy after RL is quite small?\n- In Line 409, the authors say \"The inverse relationship highlights a trade-off...\". According to Figure 7, smaller c is better for both accuracy and thought length. What trade-off are the authors referring to?\n- In Section 3.1, the authors may want to highlight how their method differs from Coconut.\n- In Line 40, the authors state ``However, reasoning over embeddings remains inherently\ndifficult and can lead to degraded model performance compared to traditional token-based inference,\nas certain tokens encode complex, nuanced information that cannot be fully preserved in compressed\nembeddings. `` Could the authors provide a citation or evidence for this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper studies a timely problem of making the LLM reasoning more efficient.\n- The proposed HyRea method is simple and effective, showcasing a good trade-off between reasoning performance and thought token lengths across multiple benchmarks. \n- The paper presents a detailed ablation study of various design choices.\n- The paper explores the generalization ability of HyRea trained models beyond mathematical tasks.", "weaknesses": "- Given that HyRea is aimed at achieve a good performance vs. token length trade-off, which sometimes results into worse performance than SFT + RL with discrete CoT (see Table 1 & Table 3), it is important to consider agains baselines that focus on achieving similar trade-off with just discrete CoT. If one encourage shorter thought lengths in SFT + RL via some form of length regularization or training on compressed thought sequences, would the resulting performance vs. thought length trade-off be worse than HyRea?\n- In Section 4.3, difference between **Random** and **Entropy** is not very significant. Authors should compare results from multiple runs to make a stronger case for **Entropy**. \n- In L221-223, the authors don't take [Latent] tokens into account while defining loss. What is the justification for this? Did the authors perform ablation for this design choice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mg8Sr7C5ca", "reviewer": "Reviewer_CEcx", "summary": "The paper presents HyRea, which is a hybrid reasoning framework for LLMs that learns to switch during inference between explicit, token‑based chain‑of‑thought and latent-space computation, marked by special start-end tokens. It is trained via an entropy‑guided cold‑start SFT that replaces low‑entropy CoT steps with latent spans while supervising only visible tokens, and GRPO‑based RL with rewards for accuracy, formatting, and latent use. They evaluate on math benchmarks MATH‑500, Minerva Math, AMC23, OlympiadBench with Qwen2.5‑7B/32B models, where HyRea achieves comparable accuracy with substantially fewer output tokens and exhibits a small number of learned mode switches per solution. It also yields shorter outputs with competitive performance on MMLU and GPQA, suggesting a practical path to more efficient multi‑step reasoning without sacrificing quality.", "questions": "- How is per‑step entropy measured—token‑level entropy aggregated per step? What thresholding is used during cold start? Also, how is the threshold on the number latent tokens $S$ is selected, does it depend on the dataset/task or length of the output? There's no discussion of this in the paper.\n\n- It's not mentioned in the paper what explicit reward function, clipping, or group size ($G$) is used during RL training. More importantly, once a trajectory includes latent tokens through hidden states that are tied to the current model weights, doesn’t that violate the on‑policy constraint of GRPO?\n\n- It's not stated in the paper what inference parameters are used across methods for the discrete tokens. Also, Section 2.1 is written based on argmax without sampling, which I believe is not the most general description of \"explicit reasoning\".\n\n- The paper does not properly discuss the COCONUT paper's setting. How are you reproducing the COCONUT paper on math benchmarks? Because what COCONUT actually does is curriculum-based learning by replacing the discrete tokens one-by-one with latent tokens, and at the end, it does full latent inference. So, are you following this on some SFT dataset? If you're not doing this, then the comparison with COCONUT is not fair. Also, how are you extracting the answer from COCONUT/Soft-Thinking trajectories? Please clarify these.\n\n- The paper also needs to discuss how they are reproducing Soft-Thinking paper's setting (as there are also some hyperparameters to tune, such as entropy in cold-stop during inference). Importantly, if COCONUT and Soft-Thinking are used without training, it's unfair to compare against them in Table 2, as they don't see any training data. This might be the reason why they are falling behind in the benchmarks.\n\n- The paper is missing connection to existing works on reinforcement learning training using continuous tokens, some example of which are \"Hybrid Latent Reasoning via Reinforcement Learning\" (https://arxiv.org/abs/2505.18454) and \"Continuous Chain of Thought Enables Parallel Exploration and Reasoning\" (https://arxiv.org/abs/2505.23648). Please discuss how the proposed method compares to these previous works, and add them as baselines to the tables if necessary.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "- The paper provides a hybrid policy through cold-start training with entropy‑based step replacement, and selectable number of latent tokens with switch tokens is novel on top of existing papers.\n\n- Good ablation studies to show the importance of different components of the method, along with a  behavioral analysis of switching patterns.\n\n- The results are competitive with fewer tokens on multiple math benchmarks.", "weaknesses": "- The paper has many under-specified details on the experiments, especially regarding the RL training and the fairness/properness of the baselines compared against, and these are not even specified in the appendix. This makes the work unconvincing in its current form. Also, there are some unclear or incorrect terminologies used in the paper. I'm willing to change my rating if I get satisfactory clarifications on experimental details.\n\n- The paper states that latent reasoning operates directly on embeddings, but explicit decoding already operates on and outputs token embeddings. So, I believe this terminology through using the term \"embedding\" is not clear for describing HyRea, which uses the hidden state $h_t$ as input.\n\n- > “reward consists of both a format reward and an accuracy reward, as well as a latent reward. Specifically, we use the latent reward to guide the model in generating the token `[Latent]`. Additionally, during loss computation, we exclude the `[Latent]` token and calculate the loss only over the remaining token steps.”\n  \n  The paper is vague in its description of the reward function and there are other details missing regarding the RL training.\n\n- The token counts in 4.2 don’t match Table 1. Table 1 reports 387 tokens for HyRea‑7B on MATH‑500 (not 287), 425 on Minerva‑7B (not 372), and 369 for HyRea‑32B on MATH‑500 (not 269).\n\n- The experimental results are only for Qwen2.5 7B and 32B models, and there are gaps in model sizes, especially smaller ones, and there is no discussion of how results would change with these sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dLqqWA1oxU", "reviewer": "Reviewer_DWUt", "summary": "The authors propose aHybrid Reasoning framework that combines explicit (token-based) and latent (embedding-based) reasoning (i.e. interpretability vs efficiency) to tackle the inefficiency of standard CoT based reasoning. Training is a done as a two-step procedure. \n\n1. Supervised Cold-Start phase : \nthe model starts to replace intermediate CoT steps (the ones with low entropy) with continuous embeddings. Specifically, for multiple rounds, such low entropy steps are identified, and replaced with ` [Latent] := <start-latent> c × [latent] <end-latent>`. The training loss is computed only on the explicit (non-latent) tokens. \n\n2. RL phase: \nThe authors use GRPO to further train the model, with the standard accuracy and formatting reward. They additionally add a latent reward to encourage the model to think in latent space. \n\nThe training procedure is conducted on two mathematical reasoning datasets for the first phase, a third one added for the RL phase. The Qwen 2 family of models (7B-32B) is used. Evaluation is conducted on standard Math benchmarks. \n\nResults show that the proposed approach strikes a better performance / efficiency tradeoff compared to the standard SFT and SFT+RL baselines. The authors moreover show that the entropy-based heuristic to identify which steps should be replaced in the first training phase performs well (better than random replacement), already striking a better tradeoff than standard SFT+RL. The authors additionally provide an analysis of different components of their method, as well as the transferability to other domains.", "questions": "1. How exactly is the latent reward computed ? the ratio of CoT steps that are latent ?\n2. Can the model do a reasoning step partially in latent space ?\n3. I am not sure I understood Figure 7. If $c$ is the number of latent tokens in each latent step, shouldn't having more latent tokens increase performance ? Or should we interpret this figure as, for a given CoT step that has e.g. 10 explicit tokens, we keep `10-c` as not latent and `c` as latent ?\n4. In Figure 6, how are you measuring the entropy ? Are you only measuring it for the `<start_latent>` token ? or are you also measuring it for intermediate latent tokens ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The authors are solving an important problem; the ability to retain the performance of CoT reasoning while making it more efficient has the potential for high impact in the field. \n2. The ability for the model to learn what (and what not) to compress is interesting.", "weaknesses": "1. A discussion on the training cost (esp. the added number of forward passes and the additional training time required) is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)