Hybrid Latent Reasoning via Reinforcement Learning
Authors: Zhenrui Yue
Bowen Jin
Huimin Zeng
Honglei Zhuang
Zhen Qin
Jinsung Yoon
Lanyu Shang
Jiawei Han
Dong Wang
Venue: NeurIPS 2025 poster ()
Keywords: Latent reasoning
hybrid reasoning
reinforcement learning
Primary area: deep_learning

Abstract: Recent advances in large language models (LLMs) have introduced latent reasoning as a promising alternative to autoregressive reasoning. By performing internal computation with hidden states from previous steps, latent reasoning benefit from more informative features rather than sampling a discrete chain-of-thought (CoT) path. Yet latent reasoning approaches are often incompatible with LLMs, as their continuous paradigm conflicts with the discrete nature of autoregressive generation. Moreover, these methods rely on CoT traces for training and thus fail to exploit the inherent reasoning patterns of LLMs. In this work, we explore latent reasoning by leveraging the intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we introduce hybrid reasoning policy optimization (HRPO), an RL-based hybrid latent reasoning approach that (1) integrates prior hidden states into sampled tokens with a learnable gating mechanism, and (2) initializes training with predominantly token embeddings while progressively incorporating more hidden features. This design maintains LLMs' generative capabilities and incentivizes hybrid reasoning using both discrete and continuous representations. In addition, the hybrid HRPO introduces stochasticity into latent reasoning via token sampling, thereby enabling RL-based optimization without requiring CoT trajectories. Extensive evaluations across diverse benchmarks show that HRPO outperforms prior methods in both knowledge- and reasoning-intensive tasks. Furthermore, HRPO-trained LLMs remain interpretable and exhibit intriguing behaviors like cross-lingual patterns and shorter completion lengths, highlighting the potential of our RL-based approach and offer insights for future work in latent reasoning.

Hybrid Latent Reasoning via Reinforcement Learning
Zhenrui Yue', Booen Jin', Huimin Zeng', Honglei Zhuang", Zhen , Yoon?,
Lanyu Shang'. Jiawei Han', Dong Wang' Uniiversity of Illinois Urbana-Chane Chaaaaaan,, "Google, "LMU
(zhenrui3, "ehherj huiminz3 1shang3, hanj, dwang24)@illinont edu,
{hlz, zhenqin ioseeniiiiiisuunoooon.. Seggunggyooooooooooooo com, shyyushhan@@mm.. edu
Abstract
Recent advances in large language models (LLMs) have introduced latent reasoning as promising alternative to autoregressive reasoning. By performing internal computation with hidden states from previous steps, latent reasoning benefit from more informative features rather than sampling discrete chain-or of of-thought (CoT) path. Yet latent reasoning approaches are often incompatible with L.Mss, as their continuous paradigm conflicts with the discrete nature of autoregressive generation. Moreover, these methods rely on CoT traces for training and thus fail exploit the inherent reasoning patterns of LLMs. In this work, explore latent reasoning by leveraging the intrinsic capabilities of LLMs via reinforcement learning (RL). To this end, we introduce hybrid reasoning policy optimization (HRPO), RL- based hybrid latent reasoning approach that (1) integrates prior hidden states into sampled tokens with learnable gating mechanism, and (2) initializes training with predominantly token embeddings while progressively incorporating more hidden features. This design maintains LLMs' generative capabilities and incentivizes addition, hybrid reasoning using both discrete and continuous representations. the hybrid HRPO introduces stochasticity into latent reasoning via token sam- pling, thereby enabling RL-based optimization without requiring CoT trajectories. Extensive evaluations across diverse benchmarks show that HRPO outperforms prior methods in both knowledge- and reasoning-intensive tasks. Furthermore, HRPO-ttained LLMs remain interpretable and exhibit intriguing behaviors like cross-lingual patterns and shorter completion lengths, highlighting the potential of our RL-based approach and offer insights for future work in latent reasoning.
Introduction
Latent reasoning has emerged as a compelling alternative to traditional autoregressive reasoning methods in large language models (LI.Ms) 8 35 39]. In contrast to the conventional chain-of-thougl
(CoT) [43 I7 [0]. which relies on the discrete decoding and sampling process, latent reasoning enables LLMs reason internally with continuous hidden representations from the previous steps. For instance, Coconut [I] achieves latent reasoning by utilizing the model's last hidden state "continuous thought', feeding back input embeddings to the next reasoning step, thereby matching the performance of CoT on reasoning-intenen intensive tasks. To show the difference between the autoregressive generation and latent reasoning, we compare both approaches in Figure [
Nevertheless, existing methods in latent reasoning utilize extensive CoT traces for training. That is, CoT trajectories required to learn informative latent representations. An example is CODI 35). which adopts self-distillation to train on discrete CoT tokens and transfers learnt features into continuous thoughts. Although recurrent latent reasoning removes the need for CoT data, it relies on training a multi-block LLM from scratch to reason internally [8]. Moreover, these methods employ tailored training paradigms for latent representation learning, incurring high training costs
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

<thinke
We
simply
edi
both
</thinks
<think
thtink
LM Head
LM Head
Decoder Layers
Decoder Layers
Embeddiing Layer
What 2+27
What 2+2?
Discrete Reasoning
Latent Reasoning
Figure 1: Comparison between discrete reasoning (left) and latent reasoning (right). Unlike the autoregressive sampling process in discrete reasoning, latent reasoning incorporates hidden represen- tations from previous steps to enhance reasoning performance (between <think> and </think>).
For example, Coconut requires multi-stage training CoT steps, which not only increases training compute but also delays the model's acquisition of complete reasoning chains [35]. Furthermore, we find that latent reasoning often incompatible with LLMs due the discrepancy between output hidden states and input embeddings (as we show Section 4.3) That is, feeding hidden states into the next decoding step degrades generation quality (e.g., repetition, incoherence), causing difficulties adapting LLMs for latent reasoning. Therefore, an ideal latent reasoning method should capitalize pretrained LLMs generalizability by seamlessly integrating continuous representations, preserving L.LMs' interpretability while mitigating CoT-dependent extensive training for broader applicability.
and overlooking the inherent reasoning capabilities of LLMs |
34
To this end, introduce hybrid reasoning policy optimization (HRPO), a novel hybrid latent reasoning optimization framework based on reinforcement learning (RL). HRPO unifies policy learning with latent reasoning. thereby utilizing the LLMs' intrinsic reasoning patterns without relying on CoT trajectories. To preserve the generative capabilities while encouraging the model to reason the continuous space, HRPO introduces gating mechanism to gradually incorporate hidden state representations from previous steps into sampled token embeddings. The gating mechanism initially configured in a way that the inputs come predominantly from the sampled tokens. As training progresses, the gate learns to incorporate richer, more informative features from previous hidden states for improved internal reasoning. Since the sampling operation introduces stochasticity, HRPO rollouts can be performed like standard RL methods, with hybrid outputs (tokens and latent representations) stored in the rollout buffer for policy updates. For optimization, HRPO leverages simple outcome-based reward and employs the hybrid rollout buffer to calculate log probabilities, enabling policy gradient updates that adaptively integrate both token-level and latent representations. By bridging discrete and continuous reasoning, HRPO provides scalable and training- -efiiieennni solution that unlocks latent reasoning in existing LLMs. As a result, HRPO enhances the adaptability of latent reasoning and leads to superior performance on both knowledge- and reasoning intensive tasks. We highlight our contributions in the following"
sessoning-intensivee
We introduce HRPO, the first reinforcement learning- based approach for hybrid reasoning, empowering LLMs to autonomously develop latent reasoning capabilities.
We design gating mechanism to preserve LLMs' generative abilities, which starts by prioritizing sampled token embeddings and, through RL-driven updates, progressively incorporates the continuous representations.
By leveraging the LLMs' inherent reasoning patterns through HRPO, mitigate the need for chain- ofain-of--thuree of-thhught annotations and expensive multi-stage training, offering an efficient and scalable alternative to existing latent reasoning methods.
To show the efficacy of the proposed hybrid latent reasoning, evaluate multiple knowl- edge and reasoning benchmarks and show that it outperforms existing models and latent reasoning baselines, demonstrating consistent performance gains across diverse scenarios.
OOur implementation is available https:/gituub.oom///eeeeeee

---

In addition, we provide insights into RL-based training of latent reasoning models and present intriguing reasoning patterns emerging from HRPO.
2 Related Work
2.1 Latent Reasoning
Early research in latent reasoning focuses analyzing the latent space computation within trans- former models [2 477. For example, Biran et al. [2] study multi-hop reasoning and show that 'back-patch' features from later layers can improve performance on challenging queries. Alternativeyy, latent representations can be used to construct informative features as in-context demonstrations to enhance few-shot performance at test-time 45 52.. In particular, Xu et al. 45] exploit latent skills tasks. Different from this line of work, hidden
to select in-context examples for reasoning-intensiie reasoning is also proposed to improve generative capabilities by incorporating latent variables into language modeling 8 20]. For instance, Geiping et al. { propose a depth-recurrence language model that injects latent variables and iteratively processes them to derive the final states used for decoding. Similarly, special tokens (e.g. <pause>) are inserted to allocate extra test-time compute for internal reasoning, leading to improvements across diverse scenarios O 29] Pfau al. [29) argue that filler tokens act as intermediate reasoning steps in multi-token computations, yielding measurable performance gains on parallelizable problems. Furthermore, implicit reasoning methods transform explicit, token-level reasoning trajectories into internal reasoning to enhance efficiency or accuracy 66 7 For instance, CODI [35 employs a self-distillation framework to align explicit and implicit reasoning tokens for improved performance. Concurrent to our work, hidden reasoning approaches
36) leverage previous output hidden states as next input embeddings, enabling compact yet informative internal reasoning. Nonetheless, the majority of existing methods require processed traces and extensive training. In contrast, we focus on hybrid latent reasoning through reinforcement learning to exploit the inherent generation capabilities of LLMs.
2.2 Reinforcement Learning
Reinforcement learning (RL) is a paradigm where agent interacts with environment, receives feedback, and learns to make decisions that maximize cumulative rewards over time
(37]. Recently, RL has been introduced improve language models by learning from implicit human feedback (RLHF) (8)). Such fine-tuning typically employs policy gradient algorithms and their variants like REINFORCE (38) To reduce variance, actor-critic methods like A2C (26) are proposed to compute a learnt baseline and leverage advantage estimates for better training dynamics. Similarly, proximal policy optimization (PPO) (32 introduces a clipped surrogate objective bound policy updates, thereby achieving training stability and robustness to hyperparameter choices. Parallel to these approaches, direct preference optimization (DPO) [3 is introduced to directly optimize language models using pairwise human preference comparisons. DPO's simpler variant such as SimPO 25 further mitigates the need of reference models. Despite DPO's efficiency, online RL methods remain preferred for their consistent superior performance [44.. Recently, reinforce leave- one-out (RLOO) D proposes REINFORCE-styl RL that generates multiple responses and utilizes the mean reward of the other responses baseline. Similarly, group relative policy optimization (GRPO) (33] and REINFORCE+ [66 compute baselines from group-level or batch-level reward scores across candidate completions, and thus reduce memory overhead while maintaining accuracy and stability for complex tasks. In this work, we design a novel online RL-driven approach to incentivize hybrid latent reasoning by progressively incorporating hidden states into LLM inputs, thereby providing richer representations for improved reasoning performance.
3 Methodology
3.1 Hybrid Reasoning with Gating
We first describe our notation and settings for hybrid latent reasoning. For input query x [2i, [T] and its corresponding token embeddings E = [er, (e.,e,,.., we describe the
raw hidden states from the LLM output at step / with h, namely:
A lh,, .e. he] (ra))former(E))
(I)
3

---

<thinko
We
simply
add
both.
<ithink
Rollout
Hybrid
Advantages
Rollouts
Query
Head
Policy Model
Standardized
Rewards
Decoder Layers
2. Policy Update
Palicy
Geting
Model
Policy
Embeddiing Layer
Loss
Model"
Ref.
Model
What is 2-2?
Hybrid Reasoning
HRPO Training
Figure Hybrid reasoning with gating (left) and hybrid reasoning policy optimization (right). During rollouts, the reasoning trajectory is generated hybridly with both discrete tokens and latent features, and for policy update, we compute the HRPO loss using the hybrid rollout buffer to update the model.
in which Trans ormer denotes the transformer model (i.e., decoder layers), A represents the final- layer hidden states produced by the Transformer. With the LM head (Head), the next output token f can be sampled from the output distribution over the vocabulary via:
E++1 ~ sof coftmax(Heedd((
(2)
However, hidden states often lie outside the model's token embedding manifold, which degrades generation quality when fed directly. To avoid this, we project he back into the embedding space ensure the inputs conform to the model's learned distribution. Specifically, we use the output probabilities P++1 to compute a weighted interpolation over the vocabulary:
h+41 WT Pe+1
Head
with PL+1 = softmax
(3)
||p+1ll
which 7 is the temperature and We denotes the embedding matrix the LLM. In other words, we compute the next input embedding weighted sum of all token embeddings, with weights given by Pt+11 addition, Pt+1 normalized to preserve the scale and variance the output vector. This sampling-free mapping ensures differentiability and aligns the projected embedding with the model's native input space, thus leading to improved training dynamics (see Section 4.3
While interpolated embeddings preserve semantic continuity, directly feeding h4+1 as the next token input removes stochasticity and injects noise from irrelevant tokens, causing degraded generation within RL rollouts. As such, we design hybrid approach for latent reasoning by gradually imposing hidden state representations into the sampled token embeddings with gating mechanism. Drawing on gated recurrence models 27). we formulate the gating mechanism
= σ (W, + ba), i, a (Wa + bb,,,
Q: exp(-c- sotttsoottttsa)ttoo.. @r),
Et-1 { at o Er+l1 v1- & (ie h441 t think, I & think,
ex41 is the resulting hybrid input for the next step, ē denotes the embedding of the sampled discrete token dt+1, whereas h441 is the projected hidden states as in Equation ( The gates r, and & leverages sigmoid function ar to control the blending, scales ē,+1, is fixed scaling constant, and A is learnable vector. Note that hybrid reasoning only applies during the reasoning phase (i.e.. t think), while the final answer is still generated via standard autoregressive decoding, as we show Figure (left). By initializing a: + (see Section A). the inputs first draw predominantly from the sampled token embeddings. thereby effectively preserving the LLM's generative capabilities. As the training progresses, the value range a, converges to an optimum range and thus incorporates informative features from both hidden representations and sampled tokens.

---

Overall, our hybrid reasoning approach projects hidden states into the embedding space via weighted interpolation. Moreover, the sampling steps preserve stochasticity for effective reinforcement learning. We employ a plug-and-play gating mechanism that initially prioritizes sampled token embeddings while gradually integrating latent signals, providing richer inputs for subsequent reasoning.
3.2 Hybrid Reasoning Policy Optimization (HRPO)
Rather than relying on strong supervision, we optimize the policy model via hybrid rollouts using reinforcement learning (RL), fully harnessing LLMs' native reasoning capabilities. Inspired by recent RL advances such group relative policy optimization (GRPO) [33], we introduce hybrid reasoning policy optimization (HRPO), an efficient RL-driven framework that enable LLMs to fuse discrete tokens with continuous representations for hybrid reasoning.
As illustrated in Figure 2 (right), the proposed HRPO optimizes the policy (parameterized by () to maximize the expected reward for input æ drawn from dataset DD and the sampled hybrid outputs (discrete tokens) and H (hidden representations):
max
(12)))
(5)
where r is a simple outcome-based reward function and a denotes the ground truth answer (i.e., it outputs for correct prediction in y and 0 otherwise). The rewards are computed solely on the discrete tokens within the answer span. To obtain an unbiased, low-variance advantage for hybrid latent reasoning, generate 9 hybrid rollouts per input query and compute the advantages by standardizing the rewards within the group (i.e., for the i-th response, the advantage is calculated by Consequently, the policy gradients can be estimated with:
A, ne-neen((p ([r (2uooal))))ll)
V JoJuno (0) = Eg (()...))()))))----- mo0-111))
Ivul Vo log To 31,00 A1, - [VoDKI(TollTed
1
(6)
9
M4
where Neel denotes the reference model and KL-divergence acts as a regularizer, controlled by hyperparameter B. This objective follows simple REINFORCE-Sty formulation, fusing discrete token inputs with continuous hidden representations across the reasoning span via the introduced gating mechanism. The hybrid trajectories that yield higher returns are assigned larger advantage estimates, encouraging policy updates to increase the log probabilities of their subsequent reasoning tokens. For the KL divergence term, we compute log probabilities using solely token IDs for Trof as we find it more effective preserving training stability. Different from PPO / GRPO objectives, we omit the likelihood ratio and directly use raw log probabilities in Equation (6) because ratio clipping is rarely encountered under our conservative learning schedule. Furthermore, since the hidden representations for a single gradient update; attempting to reuse ev-nven with importance sampling-viokee -violates the on-policy constraints. As such, our HRPO implementation remains lightweight, strictly on-policy and could be seamlessly combined with further RL. optimizations.
directly tied to the parameters 0, each trajectory should only be used
In summary, the proposed HRPO framework unifies hybrid latent reasoning under simple RL objec- tive that fully leverages LLMs' intrinsic reasoning capabilities. During rollouts, the decoding process progressively fuses discrete and continuous representations through learnable gate, preserving coherence while exploiting hidden states. For policy updates, HRPO derives advantages directly from outcome rewards and performs policy gradient steps with KI. regularization. As a result, HRPO incentivizes LLMs to dynamically integrate sampled tokens with latent representations, delivering stable and efficient on- on-policy hybrid reasoning training without separate value function.
4 Experiments
We evaluate HRPO on both knowledge- and reasoning-intensive tasks: (1) open-domain & multi-hop knowledge-intensive question answering (Knowledge); and (2) science, technology. engineering or mathematics (STEM) benchmarks. The experimental results are reported as follows.
5

---

Table 1: Evaluation performance of various larger LLMs and trained models on open-domane and multi-hop QA benchmarks. The table reports exact match scores based on top-3 retrieved documents on five datasets: NQ. TriviaQA, HotpotQA, 2WikiMQA and Bamboogle. The upper block reports results for several RAG baselines using the larger Qwen 2.5 7B L.LM, while the lower two blocks evaluate smaller Qwen models (1.5B and 3B) trained with different strategies.
NQ
TriviaQA
HotpotQA
2WikiMQA
Bamboogle
Average
Qwen2.5-7B-Instruuc
QA
0.134
0.408
0.183
0.250
0.120
0.219
CoT
0.048
0.185
0.092
0.111
0.232
0.134
IRCoT
0.224
0.478
0.133
0.149
0.224
0.242
Search-ol
0.151
0.443
0.187
0.176
0.296
0.251
RAG
0.349
0.585
0.299
0.235
0.208
0.335
Qwen2.5-1.5B-Inse Instructt
SFT
0,094
0.193
0.129
0.210
0.024
0.130
RAG
0.288
0.477
0.228
0.203
0.072
0.254
PPO
0.327
0.527
0.256
0.242
0.184
0.307
GRPO
0.293
0.480
0,202
0.213
0.120
0,261
HRPO (Ours)
0.364
0.553
0.273
0.276
0.216
0.337
Qwen2.5-3B-Instee
SFT
0.249
0.292
0.186
0.248
0.112
0.217
RAG
0.348
0.544
0.255
0.226
0.080
0.291
PPO
0.356
0.563
0,304
0.293
0.240
0.351
GRPO
0.381
0.570
0.308
0.303
0.272
0.367
HRPO (Ours)
0.378
0.593
0.316
0.318
0.296
0.380
4.1
Evaluation Knowledge Benchmarks
We first evaluate HRPO five open-domain and multi-hop question answering (QA) datasets: Natural Questions (NQ), TriviaQA, HotpotQA, 2WikiMultiHoPAA (2WikiMQA) and Bamboogle [14] For each query,
19
21
30
48
the E5 embedding model 42 to retrieve the top-3 Wikipedia
documents as context (details presented in Section A Following [8] we merge the NQ and HotpotQA training sets train HRPO models, and evaluate
each dataset's evaluation split.
The exact match results of HRPO and baselines (including supervised fine-tuning (SFT), retrieval augmented generation (RAG) 22 and RL-based PPO [32] and GRPO (33) for the 1.5B and 3B Qwen2.5 Instruct models [46] are presented in Table I We also include comparisons to several QA and RAG baselines using the larger Owen2.5-7B-Im Instructt backbone, including: direct inference
(QA), chain-of-thouge (CoT) 43 interleaving retrieval with CoT (IRCoT) [I)) Search-ol
23) and
RAG 22
For each block in Table we mark the best performance bold for clarity.
Across all knowledge benchmarks, HRPO delivers the strongest exact match (EM) scores with smaller Qwen models and rivals the much larger 7B baselines. In particular, observe: (1) HRPO reaches 0.380 EM with Qwen2.5-3B, outperforning the strongest 7B RAG baseline by 4.5%. Similarly, HRPO with the smaller 1.5B backbone scores average of 0.337, achieving consistent gains and surpassing PPO by 3.0%. (2) HRPO consistently outperforms other RL-based methods. For example, HRPO with both the 1.5B and 3B backbones surpasses the strongest RL baseline by 3.0% and 1.3% respectively; the only dataset both models perform similarly is NQ. (3) Interestingly. GRPO underperforms PPO by 4.6% on the 1.5B backbone but outperforms it by 1.6% on the 3B model, likely a consequence of sparser rewards and limited sampled trajectories with smaller model. (4) RL- based methods perform on par with the pest-performing RAG baseline, with HRPO delivering the largest performance gains particularly on terse, incomplete queries (NQ) and multi-hop questions (2WikiMQA)- while yielding modest improvements on one-hop datasets like TriviaQA. Overall, these results demonstrate that combining retrieval augmentation with hybrid latent reasoning yields state-of -fate-of-the-art knowledge performance under computation constraints, establishing HRPO as competitive alternative to both RL-based learning methods and larger retrieval augmented LLMs.

---

Table Evaluation performance of various larger LLMs and trained models on STEM benchmarks. The table presents accuracy scores on five datasets: GSM8k, MATH, MATH500, MMLU-ST and ARC-C. The upper block reports results for several few-shot baseline L.Mss 2 7B, while the lower two blocks evaluate smaller Qwen models (1.5B and 3B) trained with different strategies.
GSM8k MATH MATH500 MMLU-ST
ARC-C
Average
Larger LLMs (Size 2 7B)
DeepSeekMath-7B
0.642
0.362
0.346
0.565
0,678
0.519
Gemma-2-9B
0.707
0.377
0.364
0.651
0.682
0.556
Qwen2.5-7B
0.854
0.498
0.464
0.723
0.637
0.635
MAmmoTH227B
0.684
0.367
0.396
0.624
0.817
0.578
MAmmoTH2-88
0.704
0.358
0.732
0.642
0.822
0.652
Qwen2.5-1.5B-Insttee
SFT
0.560
0.300
0.302
0.403
0.602
0.433
Distilled CoT
0.706
0.503
PPO
0.694
0.507
0.518
0.566
0.715
0.600
GRPO
0.711
0.502
0.524
0.562
0.737
0.607
HRPO (Ours)
0.720
0.518
0.536
0.569
0.742
0.617
Qwen2.5-3B-Inse
SFT Distilled CoT
0.670
0.348
0.360
0.454
0.474
0.461
0.799
0.575
PPO
0.819
0.597
0.604
0.582
0.811
0.682
GRPO
0.834
0.602
0.604
0.601
0.814
0.691
HRPO (Ours)
0.845
0.613
0.630
0.590
0.820
0.700
4.2
Evaluation on STEM Benchmarks
We also evaluate the performance of the proposed HRPO the reasoning-intensiv STEM datasets: GSM8k, MATH, MATH500, MMLU-STEM (MMLU-ST) and ARC-Challenge (ARC-C) 4 24]
12 3. Table 2 reports the performance of HRPO alongside fine-tuned baselines (SFT, SFT with distilled CoT from QwQ [40]) and RL baselines (PPO 32 and GRPO 33) on the Qwen 2.5 1.5B
and 3B Instruct models [46] In addition, select several larger LLMs (2 7B in size) using few-shot
CoT for comparison [46] 33 49). For GSM8k, we train on the training split, and for MATH and MATH500, train the MATH training split. For MMLU-ST and ARC-C,
train on the merged auxiliary MMLU and ARC-C training sets. Distilled CoT is only available for GSM8k and MATH due to dataset size constraints. We also highlight the best scores in each block in bold.
Across the five STEM benchmarks, HRPO delivers the strongest results with compact Qwen back- bones and could match the performance of much larger LLMs. Our key observations are: (1) SFT underperforms compared to distilled CoT and RL-based methods, suggesting the efficacy of RL with verifiable rewards vrasoning-intensive tasks. (2) With the 3B backbone. HRPO achieves an average accuracy 0.700, matching the best 7B baseline on four of the datasets. Even the 1.5B HRPO averages at 0.617, outperforming the 7B leader MATH by 2.0%. (3) At 1.5B, HRPO improves on the strongest altermative GRPO with notable boosts on MATH and MATH500 (1.6% and 1.2%), whereas the average gain narrows 3B, implying that HRPO is smaller models. (4) HRPO registers the highest accuracies recorded for sub-7B models on MATH (0.613) and MATH500 (0.630), demonstrating the value of RL-hased hybrid reasoning challenging benchmarks. Taken together, these results show that hybrid latent reasoning unlocks the power of much larger LLMs in compact backbones, proving the effectiveness of the proposed HRPO.
beneficial for
4.3 Analysis of HRPO
Different Strategies for Latent Reasoning. We compare different strategies to compute latent representations. Specifically, we use three methods to integrate hidden states into RL and train the 1.5B Qwen model on the MATH dataset. These variants are: (1) hidden states, which use the final layer hidden states the next input; (2) interpolation, which employs interpolated embeddings
7

---

as defined in Equation (3): and (3) HRPO, our hybrid latent reasoning Equation 4 We vi- sualize the exponential moving average (EMA) of rewards along with the GRPO baseline in Figure [] Due to the mismatch between hid- den states and embeddings, using hidden states degrades generation and yields nonsensical roll- outs with zero reward. Although interpolation performs similar to HRPO for the first few hun- dred steps, the rewards eventually collapse and only slowly recover, likely because interpola- tion introduces excessive noise. We also provide direct comparison between HRPO and latent reasoning methods in Section B Overall, our approach achieves superior training dynamies with faster convergence while maintaining stability comparable to GRPO, highlighting the efficacy of our hybrid design choice in HRPO.
Interpolation
Hidden States
GRPC
0.5
200
6.23
2.4
Step
Figure 3: Reward on MATH for Qwen-2.5-1.5B using different latent reasoning strategies.
0.95
(mn 0.99
0.95
0.99
Frii 0.96
Learning Rate
0.98
GRPO
Hidden Ratio Learning
0.20
Completion Length
100
120
0.10
300
1000
1.50
1502
1.00
2.0
500
2000
3.50
25000
3.50
Step
Step
Figure 4: Hidden ratio with varying mmin in exp(-c: softplus(A)) and learning rate. We visualize the hidden ratio and completion length for training runs with Tmin from [0.95, 0.98, 0.99].
Ratio of Latent Representations. We track how the balance between discrete tokens and continuous latent representations shifts as LLMs learn reason hybridly. Here, train Qwen 1.5B the knowledge task and visualize both the mean hidden ratios (i.e., VI- az) and completion lengths (along with GRPO) in Figure 4 Across all runs, the hidden ratio increases steadily, even as the learning rate tapers off toward the end of training under a cosine schedule. In addition, completion lengths increase during the initial phase and later decline across all methods, with the drops most significant HRPO. Furthermore, setting Tmin = 0.95 leads to an interesting behavior where completion lengths substantially decrease an effect not seen in the other variants This may be because the hidden representations effectively capture historical context, thereby shortening completions while maintaining or even improving performance (see Table As such, hybrid latent reasoning could be particularly effective when leveraging contextual information for reasoning.
Table 3: Impact of A-initialization HRPO's performance across knowledge and STEM tasks.
Knowledge
Init Range
NQ
TriviaQA
HotpotQA
2WikiMQA
Bamboogle
Average
[0.95 0.999]
0.364
0.553
0,273
0.264
0.184
0.328
[0.98 0.999]
0.336
0,553
0.263
0.276
0.216
0.329
[0.99 0.999]
0.336
0.534
0.258
0.275
0.216
0.324
STEM
Init Range
GSM8k
MATH
MATH500
MMLU-ST
ARC-C
Average
[0.95 0.999]
0.705
0,516
0.536
0.569
0.735
0.612
[0.98 0.999]
0.703
0,509
0.532
0.563
0.732
0.608
[0.99 0.999]
0.720
0.518
0.526
0.567
0.742
0.614
Tmin is used to initialize A such that exp( softplus(A)) is drawn uniformly from [rmin 0.999).
8

---

0.7
0.5
0.9
0.5
T- 0.9
200
Completion Length
Reward
0.4
250-
200
0.2
202
400
600
220
1000
1200
1900
202
410
se0
.20
1000
1300
1000
Step
Step
Figure 5: Sensitivity analysis for temperature T in Equation 3).
We visualize the reward and
completion length for training runs with different temperature selected from [0.3, 0.5, 0.7, 0.9].
Initialization of A for Gating, Beyond hidden ratio, we examine how the initialization of A-which control the balance between latent features and token embeddings- affects HRPO performance. Specifically, we initialize exp(-c- sof (p] from [Tuin 0.999) and report the results on Qwen 1.5B in Table where lowering "'mn yields a higher initial hidden ratio. For the knowledge domain, performance improves as m'in decreases: the best average performance occurs at m'in 0.98, and most individual datasets peak at min = 0.95. In contrast, the STEM benchmarks display a bimodal trend: performance rises when mmin is either lower or higher, but drops for the intermediate range [0.98, 0.999]. This pattern implies that the model profits from emphasizing either explicit token trajectories latent representations, whereas a mid-level mix is sub-optimal. In summary, our results show that knowledge tasks benefit from lower mmin- whereas optimal performance for STEM tasks arises from leaning toward either explicit token trajectories latent representations.
Sensitivity of on Hybrid Reasoning. We further investigate the impact of temperature T on HRPO: lower J values reduce noise but overemphasize top tokens, whereas larger spreads probability mass across more tokens. We explore т E {0.3, 0.5, 0.7, 0.9) and present the rewards and completion lengths of the 1.5B Qwen model on MMLU in Figure 5 The left panel indicates that = 0.3 and T == 0.5 converge faster and reach the highest reward plateau, outperforming higher values (r 2 0.7) and showing the benefits of smaller T. Interestingly, the right panel reveals that both smaller and larger 7 values shorten completion lengths, while t = 0.5 and r = 0.7 maintain longer generations. This may be because lower 7 sharpens token distribution, yielding confident latent vector that lets HRPO finish quickly. In contrast, higher 7 flattens the distribution and enhances informativeness, prompting the policy to extract answers in shorter rollouts. Overall, we find HRPO to be robust across varuing T selections, only completion length varies noticeably. Further analysis is in Section B
MMLU Quary: This is story that happaned in Jepan. In order to the house, a Jeparese form open the wells. Japanson.... What can we learn from the free lowed? Choices:
A teach us usceever to abandon our lovediones. y
B. teachiis give more help D. encourages to live even konger!
ones.
C. It talla take
the stuck Bzard.
Hybrid Rollout)
for Chinase
To solve this problem we need to undersiand the meaning of the story and the story
First, underatand the etory: from the panspettive a loand.
mone, but walting: from atory. The story
It wants to carney Tist. 39 Bay WXX ik- BLy or
its partner another toard appeared and brought It and shows that loves The foard
MEW togg g mat M- RU Thah 7 tetts 28 sh 20 a 37-00 St.
26 de
1- 2 EERE W 2.
11/1
Nost what do resed tel us that thy kzord can love like this, even It small, It can love like a real lMnin. This makess think about how we can become more seffess and loving. 3. Thei, we neld to sestyre each option on to oon which best the interntonn the story. it teds us to not veetied partner tells us 3o sympotiizn with the trapped lzaad. This does not it the interation of the story berouuee the little lowed has symparthyy just waitedd and took care of its portner D. encoumges us live longer. This does not th: the Intertion of the story,
1it % the 11 11-0-00 to 22
29 18110
B0R996E 1 &
abandon people If This fhts into riionnn the story the litll
par
not
in becesswe the linen the did
help,
and took care
anddysio. rifiitr
the choice
abansoo
bitla keard cld not longes was just maiting and caring
mate.
Figure 6: Example cross-lingual reasoning (English-Chin Chinese) and its translation for HRPO.
Hybrid Latent Reasoning Patterns. Finally, we highlight several intriguing reasoning patterns that emerge from HRPO. First, the hybrid outputs show readable trajectories by interpreting the tokens even without any CoT supervision. Second, HRPO exhibits cross- lingull patterns in some completions, fluidly integrating tokens from different languages, suggesting that latent representations can generalize across linguistic boundaries (see Figure 6) Moreover, the hybrid reasoning process often delivers compact yet accurate responses to simple or factual queries, where the model requires fewer decoding steps thanks to the richer context encoded in the hidden representations. These

---

emergent patterns indicate that hybrid latent reasoning can improve both interpretability and efficiency over existing latent reasoning approaches. Further qualitative examples can be found in Section C
5 Conclusion
In this work, we propose hybrid reasoning policy optimization (HRPO), a novel reinforcement learning (RL) framework that unifies discrete token sampling with continuous latent representations through a learnable gating mechanism. By gradually incorporating hidden features into sampled token embeddings, HRPO incentivizes L.MMs to refine their reasoning strategies hybridly. Extensive evaluations on knowledge and STEM benchmarks demonstrate that HRPO outperforms both SFT and RL baselines, achieving consistent gains across diverse scenarios. Moreover, our analysis reveals that HRPO not only ensures stable hybrid latent reasoning but also triggers intriguing reasoning patterns, showing its potential in reasoning-intenss for RL-based continuous space learning. While promising, we recognize that HRPO introduces additional computation overhead, the on-policy design may reduce large-scals training efficiency, and its continuous representations can be less transparent. Therefore, future work will aim to address these limitations by exploring simpler designs, off-policy extensions and advanced latent reasoning techniques to improve both the interpretability and efficiency of HRPO.
settings and providing insights
6 Acknowledgments
This research is supported in part by the National Science Foundation under Grant No. CNS-2427070, IIS US-2331069, IIS- 1202488, IIS 210-23,, 2N3-2131622. The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.
References
[1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402. 14740, 2024.
[2] Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406. 12775, 2024.
[3] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05A4 05457,, 2018.
[4] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
[5] Soham De, Samuel I. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402. 19427, 2024.
[6] Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint 23Ni::3311.01600 2023.
[7] Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14838 2024.
[8] Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartold- son, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint 2rXiv:2502.05171, 2025.
10

---

[9] Sachin Goyal, Ziwei Ji. Ankit Singh Rawat. Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprini arXiv:2310.02226, 2023.
[10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in IIms via reinforcement learning. arXiv preprint arXiv:2501 2025.
[11] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
[12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[13] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0888 038744 2021.
[14] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609-6625, 2020.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint trXiv:2/06.09668 2021.
[16] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262. 2025.
[17] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint MxXi::2412.16720, 2024.
[18] Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training Ilms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
[19] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611 2017.
[20] Deqian Kong, Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig, Zhangzhang Si, Chuan Li, Jianwen Xie, Sirui Xie, al. Scalable language models with posterior inference of latent thought vectors. arXiv preprint arXiv:2502.0156; 2025.
[21] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
[22] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459- 9474, 2020.
[23] Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366 2025.
11

---

[24] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever. and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[25] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
[26] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928- 1937. PmLR, 2016.
[27] Antonio Orvieto, Samuel Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670-26698. PMLR, 2023.
[28] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 11227730-27744- 2022.
[29] Jacob Pfau, William Merrill, and Samuel R Bowman. Let's think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404. 15758, 2024.
[30] Ofrr Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022.
[31] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-537777 2023.
[32] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.000. 06347, 2017.
[33] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint wxxxx:24422.033000 2024.
[34] Xuan Shen, Yizhou Wang, Xiangxi Shi, Yanzhi Wang, Pu Zhao, and Jiuxiang Gu. Efficient reasoning with hidden thinking. arXiv preprint arXiv:2501. .0.0./9/00 /9201, 2025.
[35] Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du and Yulan He. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074 2025.
[36] DiJia Su, Hanlin Zhu, Yingehen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint arXiv:2502.03225. 2025.
[37] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[38] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[39] Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuandong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts. arXiv preprint arXiv:2502.088. 085544 2025.
[40] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm. github. i boo/blo///ae bwtg///g- 32b/
12

---

[41] Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-then -fttuuhgttthtt reasoning for knowledge-imtenivee multi-step questions. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 10014-10037, 2023.
[42] Liang Wang, Nan Yang, Xiaolong Huang. Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022.
[43] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[44] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study. arXiv preprint arXiv:2404. 10719, 2024.
[45] Zifan Xu, Haozhu Wang, Dmitriy Bespalov, Xian Wu, Peter Stone, and Yanjun Qi. Lars: Latent reasoning skills for chain-of-thought reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3624-3643, 2024.
[46] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 2rXiv:2412.15115 2024.
[47] Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint arXiv:240..16837, 2024.
[48] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2380, 2018.
[49] Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. Advances in Neural Information Processing Systems, 77:90629-90660 2024.
[50] Zhenrui Yue, Huimin Zeng, Yimeng Lu, Lanyu Shang, Yang Zhang, and Dong Wang. Evidence- driven retrieval augmented response generation for online misinformation. arXiv preprini arXiv:2403. 14952, 2024.
[51] Zhenrui Yue, Honglei Zhuang, Aijun Bai, Kai Hui, Rolf Jagerman, Hansi Zeng, Zhen Qin, Dong Wang, Xuanhui Wang. and Michael Bendersky. Inference scaling for long-context retrieval augmented generation. In The Thirteenth International Conference on Learning Representations, 2025.
[52] Yufan Zhuang, Chandan Singh, Liyuan Liu, Jingbo Shang, and Jianfeng Gao. Vector-icl: In-context learning with continuous vector representations. arXiv preprint arXiv:2410.05629, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Claims in abstract and introduction correctly reflect our contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are included in the paper and conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The experiment settings are provided in the paper and appendix, our code also publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We use publicly available models and datasets, our code is also publicly
available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpolliicggiddeeCoooooo /oaeeee/e/ooeubbbiiieoo ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We use publicly available models and datasets, our training and evaluation details are also properly reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: We provide the statistical significance in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiment details are properly described the paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We confirm that the paper aligns with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work (likely) has societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets are properly used and cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The method in this research adopts LLMs components and provides sufficient descriptions.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Implementation
For hybrid latent reasoning, our plug-and-pin component by design compatible with any LLM architecture. We initialize its linear layers with a uniform distribution from [-1/v|H 1/ HH]. where |H] denotes the hidden state dimension. The gating parameter A is selected such that the quantity a exp( - softplus(A) is drawn uniformly from [Fmin: 0.999], with the sealar constant fixed at 8 [S] Tuning Tiin adjusts the initial fraction of hidden states involved in hybrid reasoning: a larger value increases the proportion of sampled token embeddings and can be helpful for enhancing generation quality during the initial training phase. Similarly, the temperature hyperparameter r in Equation (3) can be tuned for optimal task performance, although HRPO remains robust across wide range of values. To efficiently train the LLMs with HRPO, we patch the models with optimized kernel implementations
@ and employ low-rank adaptation (LoRA) [[5]. The default choice of hyperparameters are reported Table 4 for HRPO experiments.
Table 4: Experiment hyperparameter settings.
Algorithm
HRPO
Epochs
1
Optimizer
Adami 8bit
Optimizer Momentum
B1, A2
0.9, 0.99
Weight Decay
0.1
Learning Rate
5e-6
Learning Rate (Linear in Equation 4
1e-4
Learning Rate (A in Equation (4)
1e-3
HRPO B
0.005
Max Gradient Norm
0.1
Gradient Accumulation Step
4
Group size in HRPO
4 / 8
Total Train Batch Size
32 / 64
LR Scheduler
Cosine with Warmup 0.1 BF16-mixed
Warmup Ratio
Precision (WA)
LoRA Modules
query, key, value, dense 32
LoRA A Rank
LoRA
64
The hyperparameters are selected empirically to balance efficiency and performance, and thanks to HRPO's lightweight design and additional optimizations, our framework can run on a single GPU across diverse tasks. Additionally, we apply larger weight-decay coefficient to (1) enhance HRPO training stability and (2) encourage the gating towards incorporating more latent representations (since smaller positive values increase the hidden ratio /1 - az). For simpler knowledge tasks and GSM8k, fix the HRPO group size 4, which already delivers strong performance. For more challenging benchmarks, namely MATH, MATH500, MMLU-ST and ARC-C, we instead generate hybrid completions for each query. As for prompt and completion lengths, select them empirically based on our observations, and the selected values are summarized in Table
Table 5: Experiment prompt / completion lengths.
Prompt/ / Completion Length for Knowledge Tasks
2048
512
Prompt/ / Completion Length for GSM8K
512 / 512
Prompt/ Completion Length for MATH MATH500
512 / 1024
Prompt / Completion Length for MMLU-ST & ARC-C
512 / 512
For both training and evaluation, we build each prompt by prepending system message that directs the LLM to perform step-by-step internal reasoning before generating final answer. The user query is then appended, and the entire input is formatted with the model chat template. Different from
21

---

prior work [00 [8], we adopt the minimalist delimiter # to separate the model's hybrid reasoning span from its final answer. This is because the delimiter tokenizes as a single unit, adding no length overhead while providing a clear signal to switch from hybrid latent reasoning to autoregressive answer generation. We also penalize repeated occurrences the delimiter within the completion (by assigning 0 reward regardless answer correctness) to prevent the model from early termination of hybrid reasoning. We illustrate full prompts for different type of tasks, showing the system message and example queries in Figure 7 Figure 8 and Figure 9 respectively.
Example Prompt for Knowledge Tasks
<mi sim_start/ >system
conversation between User and Assistant. The user asks a question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The final (reasoning process) #### (answer). im__nd/>> <lim_start/>use
is provided after the #### tag, i.e.,
Context (which may or may not be relevant):
Clyde River (New South :ales)::::00l ::::Clyde River (New South Wales) The... Barwon River (New South Wales) ::: : and Weir River (part of... Taponga ::ve:::::apor:::: River The Taponga River, an inland...
Question: What direction does the river that Austrolebias bellotti are found in flow?<| iimimmmnn end
imimstaart /tt.staart>ssssttnttt
Figure Example prompt for knowledge tasks, contexts are partially omitted due to space constraints.
Example Prompt for GSM8k / MATH MATH500
< </im_start/ />syste
conversation between User and Assistant. The user asks a question, and the assistant solves it. The assistant first thinks about the mind and then provides the user with the answer The final answer is provided after the #### tag, i.e., (reasoning process} #### (answer). <lim_end/> <|in_start >user
reasoning process
Natalia sold clips to 48 of her friends in April, and then she sold half many clips in May. How many clips did Natalia sell altogether in April and May?< him_mmm_]
<lim_stard, Passistant
Figure 8: Example prompt for GSM8k MATH MATH500 in HRPO.
For each question in our knowledge-intensive QA setup, we embed the query with E5 embedding model 42 The entire English Wikipedia 2020 dump is pre-encoded with the same model, after which we perform approximate nearest neighbor (ANN) search and select the three sechest-scoring documents. These top-3 passages are concatenated form the external context fed the LLM, as illustrated in Figure / In our evaluation, we generate tokens using greedy decoding and compute latent representations according to Equation (3). thereby ensuring the reproducibility our results. For outcome-based reward and evaluation settings knowledge tasks, report exact match scores on test splits following [50] 51 18 For mathematical (GSM8K, MATH and MATH500) and multiple-choice datasets (MMLU-ST and ARC-C), we follow 49 for prot-processiig and scoring.
22

---

Example Prompt for CIMLU-ST/ARC-
<lim_start /tatttt
A conversation between User and Assistant. The user asks a question, and the assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The final answer is provided after the #*## tag, 1.e., (reasoning process) #### (answer).. < <lim_end/> sin_start//user
Question: Two people are pushing car. One person is pushing with force 450 and other person pushing with force of 300 N. What information is needed to determine the net force applied to the car by the people?
Options:
A. the direction of the road
B. the direction of the forces
C. the weight of the two people
D. the weight of the automobile< im_end|
<lin_start/>assiin
Figure 9: Example prompt for MMLU-ST/ ARC-C in HRPO.
B Additional Results
Comparison to Latent Reasoning Methods. In addition to strong RL methods such as PPO and GRPO in our main experiments, we also benchmark the proposed HRPO against additional latent reasoning baselines. Specifically, we evaluate HRPO, Coconut and CODI on the GSM8K and MATH reasoning datasets, using the 1.5B Qwen backbone. For Coconut, we train with its augmented CoT data (no MATH split is available), whereas for CODI we adopt the original datasets' CoT trajectories. The results are reported Table 6 We observe: (1) HRPO achieves the best accuracy on both datasets, with 9.42% and 23.63% respective gains over the best performing latent reasoning baseline CODI. (2) Even compared to distilled CoT from a significantly larger model QwQ, HRPO still scores consistent improvements on both datasets, showing the effectiveness of our hybrid latent reasoning. (3) Coconut lags behind on GSM8k, indicating limitations of latent reasoning by compressing CoT tokens, whereas CODI improves substantially with CoT SFT but still trails Distilled CoT and HRPO. Overall, HRPO achieves the best performance against all baselines, demonstrating its consistent advantages over CoT distillation and prior latent reasoning methods.
Table 6: Performance comparison of HRPO against alternative latent reasoning methods and distilled
CoT baseline.
Coconut
CODI
Distilled CoT
HRPO
GSM8k
MATH
GSM8k
MATH
GSMBk
MATH
GSM8k
MATH
Accuracy
0.315
0.658
0.419
0.706
0.503
0.720
0.518
Sensitivity Analysis for A and In addition the results reported in Table 3 we further present the performance of various A initializations on the Qwen 3B model, as shown in Table 7 Our observations echo the same trends the 1.5B backbone: smaller initial Tiin consistently benefits both knowledge and STEM tasks. Moreover, performance peaks when rmin is selected either lower or higher, and drops slightly within the intermediate range of [0.98, 0.999]. We also examine the sensitivity of the 5 hyperparameter used to construct the interpolated embeddings and present the corresponding results both backbone models in Table g The training rewards and completion lengths for GSM8k, MATH and the knowledge tasks are shown in Figure 10 Figure [] and Figure 12 We note that choosing 7 in the range of 0.5 0.7 offers reliable balance of efficiency and accuracy, the performance often peaks around this interval for both backbone models. Overall, we find that
23

---

Table 7: Impact of A-initializattin on HRPO's performance for the Qwen 3B backbone.
Knowledge
Init Range
NQ
TriviaQA
HotpotQA
2WikiMQA
Bamboogle
Average
[0.95 0.999]
0.845
0.613
0.622
0.576
0.820
0.695
[0.98 0.999]
0,842
0,600
0.614
0,585
0.813
0.691
[0.99 0.999]
0.838
0.606
0.630
0.590
0.817
0.696
STEM
Init Range
GSM8K
MATH
MATH500
MMLU-ST
ARC-C
Average
10.95 - 0.9991
0.367
0.593
0.316
0,311
0.296
0.377
[0.98 0.999]
0.378
0.588
0.311
0.298
0.296
0.374
[0.99 0.999]
0.375
0.584
0.309
0.318
0.288
0.375
HRPO benefits from a smaller initial Tmin- which outperforms larger Vimin settings and highlights the value latent representations for complex reasoning. Moreover, HRPO is robust to the choice of т, where the performance scores remain stable with only minor fluctuations at the extremes.
Table 8: Impact of r on HRPO's performance for both backbone models.
Model
Qwen2.5-1.50
Qwen2.5-3B
0.3
0.5
0.7
0.9
0.3
0.5
0.7
0.9
GSM8K
0.717
0.720
0.705
0,694
0.842
0.841
0.845
0.833
MATH
0.518
0.516
0.507
0.514
0.597
0.606
0.613
0.599
MATH500
0.522
0.536
0,532
0.524
0.622
0.614
0,622
0.630
MMLUST
0.561
0.569
0.559
0.567
0.577
0.590
0.574
0.580
ARC-C
0.735
0.741
0.742
0.724
0.820
0.817
0.809
0.808
NQ
0,320
0.336
0,317
0.364
0.378
0.375
0.373
0.363
TQ
0.524
0.534
0.553
0.553
0.588
0.593
0.578
0.578
HotpotQA
0.263
0.260
0.252
0.273
0.311
0.316
0.309
0.306
2Wiki
0.276
0.272
0.264
0.244
0.318
0.311
0.297
0.293
Bamboogle
0.216
0.216
0.216
0.176
0.296
0.288
0.296
0.280
= 0.3
= 0.7
= 0.3
= 0.7
68
Completion Length
2000
Reward
0.6
0.4
223
P
8C0
Step
Figure 10: Reward and completion length for training runs with different temperature values GSM8k using the Qwen 1.5B backbone.
Additional Analysis for A Initialization. We further provide an expanded analysis of how varying Tmin in the initialization of A affects training dynamics with the larger Qwen 3B backbone. Figures Figure 13] Figure 14] Figure 15 and Figure 16 the reward and completion length curves for the knowledge tasks, GSM8k. MATH and MMLU-ST ARC-C respectively. Overall, our findings here echo the observations in Section 4.3) different rimn values exhibit similarly high training stability and preserve the LLM's generative capabilities, but selecting a smaller Tmin (i.e., larger initial hidden ratio) generally accelerates convergence and shortens generated completions. Nevertheless, these benefits are less pronounced for the backbone than for the 1.5B counterpart, which we attribute to the fewer update steps and trainable parameters in HRPO. In summary, our analysis shows

---

T 0.9
0.5
T 0.9
6.-
Completion Length
100
0.5
Reward
500
0.3
0.2
6.1
202
420
603
320
200
400
820
Step
Step
Figure 11: Reward and completion length for training runs with different temperature values on MATH using the Qwen 1.5B backbone.
0.7
0.5
T= 0.9
0.5
r= 0.9
65-
Reward
200-
0.3
Completion
750
1020
1250
1500
1750
250
1000
1500
1750
Step
Step
Figure 12: Reward and completion length for training runs with different temperature values on knowledge tasks using the Qwen 1.5B backbone.
that HRPO preserves stable training dynamics and effectively leverages LLMs' intrinsic reasoning patterns across Tmin values; moreover, choosing a smaller 7min further enhances convergence and yields shorter generated sequences, which be especially beneficial for smaller-scale LLMs.
Statistical Significance Analysis on the Improvements HRPO. In our main experiments, follow the standard practice of using greedy decoding for pass @ evaluation, ensuring our results are easy evaluate and reproducible. To evaluate the significance of the performance gains of HRPO, we conduct additional sampling-basee evaluations on the STEM tasks, which exhibit greater variance compared to greedy decoding. Averaged results are presented in Table with statistically significant outcomes (paired t-test, < 0.05) highlighted in bold. These results show that HRPO consistently outperforms PPO and GRPO across both backbones on all benchmark datasets. For the 1 1.5B backbone, t-tests confirm these gains are statistically significant in three out of five tasks. The improvements are even more pronounced with the 3B model, which achieves an average gain of +1.4% and shows statistical significance in four out of five comparisons. These findings demonstrate that our hybrid-RL framework, HRPO, not only delivers reliable performance gains over established baselines but also does so with high statistical confidence across the majority of STEM tasks.
Table 9: Significance test on HRPO's performance improvements.
Qwen2.5-1.5B
GSM8k
MATH
MATH500
MMLU-ST
ARC-C
PPO
0.701
0.505
0,511
0.551
0.716
GRPO
0.710
0.510
0.512
0.554
0.722
HRPO
0.712
0.515
0.517
0.565
0.731
Qwen2.5-3B
GSM8k
MATH
MATH500
MMLU-ST
ARC-C
PPO
0.825
0.597
0.600
0.574
0.802
GRPO
0.827
0.595
0.599
0.577
0.808
HRPO
0.838
0.606
0.609
0.585
0.815
25

---

Fm
0.95
Frrs
Froi
Frie.
Fnn
06
200
240
y
0.4
2.00
220
0.7
100
292
530
750
1003
1250
1992
1750
250
520
790
1002
1350
1503
1750
Step
Step
Figure 13: Reward and completion length for training runs with varying initial mmin on knowledge tasks using the Qwen 3B backbone.
0.95
Frid
0:98
0.95
0.99
0.0
2.50
0.5
200
0.4
150
402
200
Step
Step
Figure 14: Reward and completion length for training runs with varying initial rmin on GSM8k using the Qwen 3B backbone.
C Qualitative Analysis
To further highlight HRPO's reasoning patterns, we present additional qualitative examples. Each example provides the reasoning trace by decoding the sampled tokens from the hybrid reasoning process, and we include both successful and erroneous cases across different tasks in the following.
The correct examples are provided in Figure 17 Figure 18 Figure 19 Figure 20 Figure 21 where as
the mistakes are provided in Figure 22 Figure 23 Figure 24 Figure 25 Figure 26 we show the raw strings and omit the options contexts in the examples due to space constraints.
From these examples, we identify four reasoning patterns that can lead to correct answers: (1) Purely English reasoning with coherent trajectories (Figs. Figure 17 and Figure [8), a pattern commonly observed in LLM reasoning outputs. (2) Predominantly English reasoning punctuated by rare tokens (c.g., %en rather than (n), as shown in Figure 19 (3) Cross-lingual reasoning that interweaves multiple
languages (English and Chinese in Figure 20 (4) Reasoning with many uncommon tokens and
atypical steps, yet still arriving at the correct answer (Figure[1 21). These latter three patterns are rarely observed in standard reasoning LLMs but are more prevalent in HRPO trained models, demonstrating that HRPO can enhance reasoning by leveraging LLMs' intrinsic generative capabilities across different languages and token types, thereby delivering improvements across diverse scenarios.
As for reasoning errors, we also identify several common patterns: (1) Cross-lingual mistakes arising from limited parametric or contextual knowledge, in Figure 22 and Figure 23
(2) Correct answers
that violate the predefined format and thus receive a zero score (Figure 24p.
(3) Repetitive loops
that continue until the response hits the maximum completion length (Figure 25 (4) Cross-lingual
reasoning that is nonetheless truncated by the length limit (Figure 26). Overall, these patterns indicate that, while HRPO effectively integrates discrete and latent representations in its internal reasoning process, it may be further enhanced through refined output formatting (e.g., potentially with format reward), extended optimization schedules with conservative learning, increased model parameters, and longer context generation capabilities, pointing to promising directions for future research.
26

---

Fan 0.95
Imt 0.98
Fron = 0.99
Trtn 0.95
Fron 0.98
Fnn 0.99
700
0.
003
100
0.2
400
802
2003
600
Step
Step
Figure 15: Reward and completion length for training runs with varying initial miin on MATH using the Qwen 3B backbone.
3.2
225
0.0
200
173
150
122
100
0.2
1400
800
1200
1300
Step
Step
Figure 16: Reward and completion length for training runs with varying initial Mmin on MMLU-ST / ARC-C using the Qwen 3B backbone.
Query: Which film has the director who died first, The People Vs. Larry Flynt Manila In The Claws Of Light?
HRPO (Tokens in Hybrid Rollout): To determine which film has the director who died first, we need to identify the directors of both films and find their dates of death.IninTi People Larry Flynt:IDiirecto: Miloš FormaninMillš Forman died on August 19, 2012. in the Claws of Light:lnDirector: Lino BrockainLino Brocka died June 1, 1973.nninCopparin the dates:InMiloš Forman died on August 19, 2012InLino Brocka died on June 1, 1973InlnLino Brocka died first. IninTherffore, the film with the director who died first Manila in the Claws of IninThe answer is: Manila in the Claws of Light #### # Manila in the Claws of Light
Figure 17: Correct reasoning example in HRPO.
Query: Which religion a domed structure called a Stupa shrine?
HRPO (Tokens in Hybrid Rollout): To determine which religion a Stupa as a shrine, let's consider the provided context and the information we need to extract/nin- Stupa is described I'mound-like (such as šarīra typically the remains of Buddhist monks or nuns) that is used as a place of meditation. I"Int The context mentions that \"Religious buildings in the form of the Buddhist stupa, a dome shaped monument, started to be used India as commemorative monuments associated with storing sacred relics of the Buddha.!"n- Additionally. \"Shanti Stupa is Buddhist white-domed stupa (chorten) on a hilltop Chanspa, Leh district, Ladakh, in the north Indian state of Jammu and Kashmir!"InlnFrom these points, Stupa is a religious structure used by Buddhism, specifically for storing relics and as a shrine.uninfreasoningg process) We asked which religion uses domed structure called Stupa as a shrine. From the given context, it is clear that the Stupa is associated with Buddhism and is used shrine. (####) Buddhism
domed structure called
hemispherical structure containing relics
clearly that the
Figure 18: Correct reasoning example in HRPO.
27

---

Query: John fills 6 foot by 4 foot pool that is 5 feet deep. It cost V$.1 per cubic foot to fill. How much does cost to fill?
HRPO (Tokens in Hybrid Rollout): To find out how much costs to fill the pool, we need to first calculate the volume of the pool. The volume of rectangular prism (like a pool) is given by the formula%%n%nnVolum = Length x Width x Height%n%nnIn this case, the length is feet, the width is feet, and the height is 5 feet. So, Vnnouume 6 fhet% cost to fill the pool is given as $0.1 per cubic foot. Therefore, Wnnoosst = Volume Price per cubic foot%nCost 120 $0.1 $nSon,SSo, the cost to fill the pool $12. (reasoning process) #### (12)
120 cubic
Figure 19: Correct reasoning example in HRPO.
The of the proper divisors of 18 is 21. What is the sum of the proper divisors of 198?
Query:
HRPO (Tokens in Hybrid Rollout): To find the sum of the proper divisors of 198, we first need to identify all the proper divisors of 198. Proper divisors are the divisors of a number excluding the number itse क : 1, 2. 3, 6, 9. 11, 18, 22, 33, 66, 99, 1981n 198 $. # # inin EL proerr divisors 22 3, 6, 9, 11, 18, 22, 33, 66. 99Amin *
# i 3t #3 AD: : 1n1 + + + 3 6 9 + 11 + 18 + 22 33 66 + 99 270 jtt. 19889pooper
ai
(reasoning process) #### # (answer) 270
Figure 20: Correct reasoning example 4 in HRPO.
Query: What is (7 !) ??
To find the value of (=(((((-1))(11])%)) we need to apply the rules exponents. (Note: The exponent rule %%((-a)'(-1)- -a1(-))%% but )%nl am (leave the word in the blank) and the study of music will be called (reasons) to For this, the Rhythm Method is the only way teach music:9nnfreaoonigg process) #### understand the reasoning process behind the given problem, let's break down the components and matherntical operations iwvolve.%nn%nFinee (-19%,.,,% %n%nBy exponent rules, %%((a (-m)) [-n] = a'(-m %lot -n) a ())..%). Therefore. %%((("(-1))'(1] Rhythm Method is the only way to teach music. However, the blank in the Rhythm Method is left incomplete with a word, which leaves the option use any word that fits the context, such as 'method', 'system', 'approach', 'tool', 'technique', or 'strategy' :n%vnn these points, we can condlude::nn&nnrresoonnn process) #### %%(7%%)
HRPO (Tokens in Hybrid Rollout):
we are dealing with %%((7"
7'((-)) %lot -1) = 7'1 = 7%%). %n&nNext, we are told that the
Figure 21: Correct reasoning example 5 in HRPO.
Query: What party was Johan Henrik Deuntzer a part before the party that translates the "Radical Left?"?
HRPO (Tokens in Hybrid Rollout): Social Democratic League #### Social Democratic League (reasoning process) : a Johan Henrik Deuntzer before the party that translates the \"Radical Left.l" E. te tt A9 B3
Henrik Deuntzer 2 #1 A5 mm
FSocial Democratic League, it.. 1 (Johan Henrik Deuntzer) #### Johan Henrik Deuntzer
Figure 22: Mistaken reasoning example HRPO.
28

---

Query: When the Marlins are the home team, games are played at this park which was LEED certified as the greenest MLB park in what year?
HRPO (Tokens in Hybrid Rollout): 2012 (reasoning process)
ME 3 mm # 4 > MLB E A RE IN 49, 12 प. 1
Marlins Park #E dA 4 1S Fit 29 LEED die A A 5i
Marlins Park E 1
S fh 18 (answer) 2012 4, Marlins Park 1E MLB BER
# 2 LEED E AA
Figure 23: Mistaken reasoning example HRPO.
Query: Can you name the singer of the title track for '1971 Diamonds Are Forever'?
HRPO (Tokens in Hybrid Rollout): Thought:
1971 Diamonds -
Are Forever X f th #8
89 E
E 3r (2 -e 3nn $ 2 \'The Spy Who Loved Mel" #1 3 # # I'On Her Majesty's Secret Servicel " ta + 40 $ # \iDiamonds Are Foreverl" el (a
# I"Goldfinger' B
$ l'Thunderballl
#B
\"For Your Eyes Onlyl i 4 1n44
A H '"Moonraker'
(R All th
Vnnnn a popular Irish rock band from Northern Ireland, so it's possible that one of the singers from Westlife could have performed the song as well. However, the question asks for the singer of the title track for the 1971 film \"Diamonds Are Forevert". and the answer is the one mentioned in the background information. Therefore, the singer of the title track for '1971 Diamonds Are Forever' is Shirley Bassey.
35 25 (Silvia Braithwaite) #### Silvia BraithwaiielnnnWWestiif is
Figure 24: Mistaken reasoning example 3 in HRPO.
Query: What the largest prime factor of 1391?
HRPO (Tokens Hybrid Rollout): To find the largest rank for the first emperor, placing parentheses. The largest rank of the integral is 1391. Let's first factorize 1391 into its prime factors. 9e1to9 is an odd number, so is not divisible by 2. The sum of the digits of 1391 (1 3 + 9 1) 14, which is not divisible by 3, so 1391 is not divisible by 3. It is not divisible by either, but is divisible by 7: 1391 7 = 198.3571. The product of the digits 1391 is 1 x 9 27, which is divisible by 3, so 1391 is divisible by 3. Therefore, 1391 3 x 463. %n%nNNw we have the prime factorization of 1391 as 1391 = 3 x 463. Next. we need to determine either or 463 is prime number. The numbers 3 and 463 are both integers. A series of operations involving integers. Therefore is divisible by 3, so is not prime.
3 and 463 are not prime, but the number 463 is prime number, where 463 is prime number since it has RE
positive integer divisors other than 1 and itself. at E Al
RE
#
A A
Figure 25: Mistaken reasoning example 4 in HRPO.
29

---

Query: There exist nonzero integers $a$ and $b$ such that the quadraticlnI\((x b) (bx-
a)^2 = xlIJhas one integer root and one non-integer root. Find the non-integer root.
HRPO (Tokens in Hybrid Rollout): To find the non-integer root of the equation W((x b + (bx a)^2 xII.. let's first simplify and analyze the equation. IninLet W 1=
W) and W y_2 2 bx aw-a a I). The given equation transforms into VV y 1 + y_ 2 = x W0.IninFrom here, we need to explore the nature of W(XX), which is function of Wy 1W) and [y)_ InInGiven the
equation W(y y_ 1 y 2_2 = x W), we know that VXXl)) is a function of the combination of \yy 1W) and ( and since W y_ 1 W) and Wy 2 W) are linear in terms of Wall) and W(bl)) we need to analyze the roots.InnnTo find the roots, we need to find the values of W(xlI) that make the quadratic equation true. Let's solve for W(all) and W(bl)) in terms of MnxI)_InknIf we assume W (ax- + (bx a)²2 = x W), let's assume value for V(XI) and check if the roots fit the criteria. IninFor simplicity, let's check if there's value of W(XI) such that the roots are one integer and one non-integer. IninOne way to approach this is to notice that the sum of squares Wyy 1*2 M y_2"2 x W) will be satisfied if WXx W) itself is sum of two squares of integers. By trying out some values, we can find out the roots. Intn assume the non-integer root, we denote it by WnnV), and solve for it. nnnn example, if we assume W(a 1W) and W(b 2W), then
2
get:inl(((
2) + (2x- - 1 = xIl.ininExpanding this, we get Wxx²2 - 4x + 4 4x"2 4x + 1 = xl),, which simplifies to M(5x12 8x + 5 xiI) or W(5x*2 9x 5 Inn..ni roots of this quadratic equation can be found using the quadratic formula W(x = Wifra(-bb lppm lisqrt(b"2 4ac])22a)))miinHeree the non-integer root W(nll) is the root closest to but not equal to an integerininSo, the non-integer root Wnnl) in the equation, W(n = Wifrac(9 lppm llsqrt(81 100))10A)) which simplifies to Wnn Wffrac(9 \pmm Maartt/epjJ///AA//Hoeeeee, W Nsqrt(-19]V) is not real, so we conclude that there must exist a non-integer root W(nl)) that satisfies the given equation.nnnNNow let's calculate the value of Winl)) in the equation, W(nl) W(Nfrac(9 Iipm \lsqrt(81
100))10R)) which simplifies to Wn = Wifrac(9 Ippm 1isortt19011000..nnnnnneeet क W(ll) æn W(bl) f9 i32 3t
BI Wnll) W(lfrra(99
lpp llsqrt(81 - 100))10)V) 1t Wn Wfrrac(9 1ppm Visrt([19))
In an B9 W(nl)). inin ltt 31 38 H2 #R [E3 lt. H E * & T THE JE d #o 95 a He te HR (niV). se E #4 Ininfreasonabil.221111 E BA e de AE M AS de (4,
W(Nlsqrt(-19)) デt Fi 1L SE L
9
Win Wffra((9 llpm - & # #9 #h
In E # # Wnnl) -
- -
Wnll) \n
Wnll) 22 ED Win Wffrac(9 lpm llsqrt(81
100)) (n Wnnl)) # 18 Wnll) # #4.
ED Wnn = \Ifrac(9
Figure 26: Mistaken reasoning example 5 in HRPO.
30


Reviews:
[{"review_id": "CiE62va7en", "reviewer": "Reviewer_AfNc", "summary": "This paper introduces HRPO, a novel RL framework designed to enhance the reasoning abilities of LLM. The work primarily tackles key limitations of previous latent reasoning methods, like coconut, which demands extensive training resources and difficult to integrate with AR models. The core idea of this method is a learnable gating mechanism that dynamically blends the embeddings of discrete tokens with continuous hidden representations from previous steps. This enables the LLM to perform \"hybrid reasoning\". The model is trained with an on-policy RL framework using the simple rule-based reward. Extensive experiments on both knowledge-intensive and STEM datasets demonstrate the effectiveness of the method.", "questions": "1. `r_min` is firstly used on Line 267 without prior definitions.  Maybe I miss something. It is better to clarify the notation before using it. \n   \n2. It would be great if you can compare to coconut, despite that I know it is a bit of hard to well-reproduce their work.\n    \n3. More discussions (addition to Figure 4) about length reduction are appreciated. It is well-known that latent thinking is an efficient LLM reasoning method, which accelerates the reasoning process, both in faster speed and shorter length.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n1. Good motivation and technically sound methodology. The central idea of using on-policy RL to train a model to learn a hybrid reasoning strategy is both novel and significant. It provides an elegant solution to the problem of integrating continuous reasoning into discrete LLMs without needing explicit CoT trajectories for supervision.\n\n2. Comprehensive experiments and promising results.\n\n## Weakness\n1. Significance of improvements. While HRPO consistently outperforms GRPO, the margin of improvement is sometimes modest, particularly for the 3B model (e.g., 0.694 vs. 0.691 average on STEM, a 0.3% relative improvement; 0.378 vs. 0.367 average on Knowledge, a 1.1% improvement). While consistency across many datasets is a strong signal, a critic might argue the gains are incremental over a strong RL baseline.\n\n2. Lack of theoretical analysis for the gating mechanism. For example, the Equation(4), I am confusing that why `a_t` is in such a formulation, why the weight between discrete token and latent thinking is balanced by `a_t` and `\\sqrt{1-a_t^2}`, instead of `1 - a_t` or other formulations.", "was_revised": true, "final_justification": "After carefully considering the authors' rebuttal and the discussions among reviewers, I maintain my original assessment of this submission.", "rebuttal": "We thank you for the constructive review and are excited that you find our work “novel”, “significant”, “good motivation” and “technically sound”. In the following, we have included responses and additional experimental results to respond to the raised weaknesses and questions, we hope these could address your concerns regarding our manuscript:\n\n**W1. Significance of improvements.…**\n\n- We acknowledge your concerns w.r.t. the statistical significance of the HRPO results in comparison to baseline methods. This stems from (i) our focus on a simple hybrid RL implementation using REINFORCE with group-relative rewards [1–2], which does not include optimization techniques like per-token rewards or ratio clipping; and (ii) we follow the standard practice of using greedy decoding for pass@1 evaluation [3–5], ensuring our results are easy to evaluate and reproducible. To address your concerns, we conduct additional sampling-based evaluations on the STEM tasks, which are often more challenging and exhibit greater variance. We present the averaged results below, with statistical significance (paired t-test) highlighted in bold:\n\n- | 1.5B | GSM8k | MATH  | MATH500 | MMLU-ST | ARC-C |\n| ------ | ----- | ----- | ------- | ------- | ----- |\n| PPO    | 0.701 | 0.505 | 0.511   | 0.551   | 0.716 |\n| GRPO   | 0.710 | 0.510 | 0.512   | 0.554   | 0.722 |\n| HRPO   | 0.712 | **0.515** | 0.517   | **0.565**   | **0.731** |\n\n- | 3B | GSM8k | MATH  | MATH500 | MMLU-ST | ARC-C |\n| ------ | ----- | ----- | ------- | ------- | ----- |\n| PPO    | 0.825 | 0.597 | 0.600   | 0.574   | 0.802 |\n| GRPO   | 0.827 | 0.595 | 0.599   | 0.577   | 0.808 |\n| HRPO   | **0.838** | **0.606** | **0.609**   | **0.585**   | 0.815 |\n\n- Across both backbones, HRPO outperforms PPO and GRPO on all benchmark datasets. For the 1.5B backbone HRPO, paired t-tests show these gains reach statistical significance (p < 0.05) in three out of five tasks. With the 3B model the improvements are even larger (+1.4%) respectively, and four out of five comparisons are statistically significant. These results demonstrate that our simple hybrid-RL framework HRPO not only delivers consistent performance gains over established baselines, but does so with high statistical confidence across the majority of the STEM tasks.\n\n**W2. Lack of theoretical analysis for the gating mechanism…**\n\n- We thank the reviewer’s interest in our gating mechanism. In the following, we explain why we mix the discrete token embeddings and the projected latent embeddings via the coefficients in Eq. 4. Our gating component is designed following common practice in state-space models / recurrent models [6-7], and the mixing is designed to have the following properties: (i) norm‑preserving circular mixing: hybrid embedding always lies on the same hypersphere as the original embeddings. This preserves the overall magnitude (and thus the scale/variance) of the input; (ii) spherical linear interpolation (slerp) between the two vectors. This yields a smooth arc on the unit sphere, rather than the straight‑line “lerp” that one would get with linear interpolation. Empirically, slerp‑style mixing maintains more consistent embedding geometry and leads to more stable RL rollouts, as demonstrated in our preliminary experiments and existing literature [8]. In brief, this choice ensures norm‑preservation and a smooth, non‑linear interpolation between the two representations. We will add a short theoretical rationale and the discussion of our design choice in the revised manuscript for clarification.\n\n**Q1. r_min is firstly used on Line 267 without prior definitions…**\n\n- We thank the reviewer for catching that $r_{min}$​ first appears in Line 267 without a prior definition. $r_{min}$ denotes the lower bound of the gating component’s sampling range; that is, during the initialization of $\\Lambda$, we sample the gating coefficient $a_t$ uniformly from [$r_{min}$, 0.999]. With higher $r_{min}$​ values, this ensures that at initialization the model draws predominantly from token embeddings. To address this, we will introduce a clear definition of $r_{min}$ in our revised manuscript before the first occurrence of $r_{min}$.\n\n**Q2. It would be great if you can compare to coconut…**\n\n- Thank you for the suggestion, although HRPO represents the first hybrid latent reasoning framework via RL, we find it valuable to compare to other supervised latent reasoning methods such as Coconut (supervised CoT), CODI (supervised CoT + distillation) along with discrete-token CoT distilled from larger reasoning models [9-10]. We reported the full results in our technical appendix and provide these here for your reference. For GSM8k, Coconut achieves 0.315 accuracy, whereas HRPO reaches 0.720, a 40.5% improvement. Moreover, HRPO attains 0.518 accuracy on MATH, outperforming the strongest latent‑reasoning baseline by 10% (Coconut does not provide CoT split for MATH). These results show that, even though Coconut leverages hidden‑state feedback as “continuous thought,” it still lags significantly behind our RL‑based hybrid reasoning approach.\n\n- | Method         | GSM8k | MATH |\n| -------------- | ----- | ---- |\n| Coconut        | 0.315 | –    |\n| CODI           | 0.658 | 0.419|\n| Distilled CoT  | 0.706 | 0.503|\n| HRPO           | 0.720 | 0.518|\n\n**Q3. More discussions (addition to Figure 4) about length reduction…**\n\nWe agree that further discussions on HRPO’s length‑reduction benefits could strengthen the paper. In the paper, we show that as training progresses, all variants’ completion lengths first grow and then contract, but HRPO with a smaller initial hidden‑ratio bound exhibits the steepest drop. This effect is more pronounced on knowledge‑intensive tasks, where we retrieve relevant contexts to answer complex multi-hop queries. For example, with the 1.5B backbone on HotpotQA, 2Wiki, and Bamboogle, HRPO’s average output lengths are 11.46, 99.99, and 93.09 tokens, respectively, versus GRPO’s 60.03, 101.82, and 101.93, suggesting consistent length reduction with increased latent context information. This may be traced back to the fact that the hidden representations can capture rich context information and thus shorten completions by providing a concise answer. We’ve also expanded this discussion in the revised manuscript and hope this can address your concerns. \n\n\n[1] Sutton, Richard S., et al. \"Policy gradient methods for reinforcement learning with function approximation.\" Advances in neural information processing systems 12 (1999).\n\n[2] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).\n\n[3] Zeng, Weihao, et al. \"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\" arXiv preprint arXiv:2503.18892 (2025).\n\n[4] Zuo, Yuxin, et al. \"Ttrl: Test-time reinforcement learning.\" arXiv preprint arXiv:2504.16084 (2025).\n\n[5] Su, Yi, et al. \"Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domains.\" arXiv preprint arXiv:2503.23829 (2025).\n\n[6] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" International Conference on Machine Learning. PMLR, 2023.\n\n[7] De, Soham, et al. \"Griffin: Mixing gated linear recurrences with local attention for efficient language models.\" arXiv preprint arXiv:2402.19427 (2024).\n\n[8] Azuri, Idan, and Daphna Weinshall. \"Generative latent implicit conditional optimization when learning from small sample.\" 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021.\n\n[9] Hao, Shibo, et al. \"Training large language models to reason in a continuous latent space.\" arXiv preprint arXiv:2412.06769 (2024).\n\n[10] Shen, Zhenyi, et al. \"Codi: Compressing chain-of-thought into continuous space via self-distillation.\" arXiv preprint arXiv:2502.21074 (2025)."}, {"review_id": "wtEg6urQtc", "reviewer": "Reviewer_tnmr", "summary": "To tackle known limitations of latent reasoning in LM training (e.g., incompatibility of output and input embeddings), the authors propose a gating mechanisms that gradually incorporates hidden states from previous steps into sampled tokens that is trained with an RL algorithm (“hybrid reasoning policy optimization”, HRPO), which is derived from GRPO by omitting likelihood ratios, due to the rollout nature of having token and latent representation rollouts. The derivation of HRPO and the underlying gating mechanism uses a sampling free mapping to allow for differentiability and a learnable gating parameter vector.\n\nThe performance of HRPO is evaluated on knowledge benchmarks (NQ, TriviaQA, HotpotQA, 2WikiMQA, and Bamboogle) and STEM benchmarks (GSM8k, MATH, MATH500, MMLU-STEM, ARC-Challenge). As baselines, the authors compare to more capable models (7B parameters) with RAG and COT as well as SFT, RAG, PPO, and GRPO baselines for the smaller models also trained with HRPO (Qwen2.5 instruct 1.5B and 3B). \n\nWithout reporting statistical significance analysis or uncertainty estimates, the authors present results that indicate a strong performance of HRPO compared to the baselines. In addition, the authors study the impact of HRPO gating parameter initialization and sensitivity and qualitative reasoning studies.\n\nThe submission does not explicitly discuss limitations or has an impact statement.", "questions": "- In Figure 3, why do the GRPO and HRPO plots share almost all artifacts (they look like they are perfectly in synch for most jumps in either direction). \n- Is there an experiment that could test the hypothesis in lines 268-270? \n- Have you trained the same model for both tasks with HRPO? Showing that you can train HRPO for multipel tasks simultaneously would demonstrate that it can work without the gating mechanisms, e.g., introducing negative interference. \n- What are the limitations of your submission? \n- What are the potential societal impacts of HRPO? For example, can HRPO and latent reasoning training make evaluations of models for safety-critical applications harder as the models learnt to be less verbose about their reasonings (just an arbitrary example)?", "limitations": "No (see questions for feedback).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Reasons to accept:\n\n- The paper is well written and, to the best of my knowledge, original work (Clarity).\n\n- The authors provide good theoretical description and background for the introduction of HRPO and conduct a series of experiments to make an argument for all their claims (Quality).\n\n- The idea of using a gating mechanism is elegant and promising. If all claims are true, the submission has the potential to offer a new way of training models on their latent reasoning in an RL setting (Significance).\n\nReasons to reject:\n\n- The authors do not conduct any statistical significance analysis whatsoever and do not report statistical uncertainties of their results. Their results are promising and I would like to award a higher score, but due to good scientific practice, it is unclear whether the reported improvements over the baselines are statistically significant and whether the discussed claims in section 4.1 and 4.2 are valid. This concern can be fully addressed by reporting, e.g., 95% confidence intervals for all results in Table 1, Table 2, and either conducting time-series uncertainty estimates (e.g., Gaussian processes) or clearly stating that the results in the Figures 3, 4, and 5 are only statistical trends, not results. If the improvements in Table 1 and 2 are statistically significant, I will happily increase my score. \n- The authors train the gating mechanism/HRPO separately for each set of tasks, makign it ambiguous whether you can use it to train models across tasks. \n- The lack of a limitation section with a balanced discussion as well as the lack of an impact statement weakens the position of the submission.", "was_revised": true, "final_justification": "The authors were able to address my concerns and I believe the proposed HRPO has the potential to influence and guide future work in this field.", "rebuttal": "We appreciate the insightful and constructive review and are thrilled that our work is regarded as “elegant”, “promising”, “original” and “well written”. Below, we provide our responses along with additional results to respond the mentioned weaknesses and questions:\n\n**W1. The authors do not conduct any statistical significance analysis.…**\n\n- We acknowledge your concerns w.r.t. the statistical significance of our HRPO results in comparison to baseline methods. This stems from (i) our focus on a simple hybrid RL implementation using REINFORCE with group-relative rewards [1–2], which does not include optimization techniques like per-token rewards or ratio clipping; and (ii) we follow the standard practice of using greedy decoding for pass@1 evaluation [3–5], ensuring our results are easy to evaluate and reproducible. To address your concerns, we conduct additional sampling-based evaluations on the STEM tasks, which are often more challenging and exhibit greater variance. We present the averaged results below, with statistical significance (paired t-test) highlighted in bold:\n\n- | 1.5B | GSM8k | MATH  | MATH500 | MMLU-ST | ARC-C |\n| ------ | ----- | ----- | ------- | ------- | ----- |\n| PPO    | 0.701 | 0.505 | 0.511   | 0.551   | 0.716 |\n| GRPO   | 0.710 | 0.510 | 0.512   | 0.554   | 0.722 |\n| HRPO   | 0.712 | **0.515** | 0.517   | **0.565**   | **0.731** |\n\n- | 3B | GSM8k | MATH  | MATH500 | MMLU-ST | ARC-C |\n| ------ | ----- | ----- | ------- | ------- | ----- |\n| PPO    | 0.825 | 0.597 | 0.600   | 0.574   | 0.802 |\n| GRPO   | 0.827 | 0.595 | 0.599   | 0.577   | 0.808 |\n| HRPO   | **0.838** | **0.606** | **0.609**   | **0.585**   | 0.815 |\n\n- Across both backbones, HRPO outperforms PPO and GRPO on all benchmark datasets. For the 1.5B backbone HRPO, paired t-tests show these gains reach statistical significance (p < 0.05) in three out of five tasks. With the 3B model the improvements are even larger (+1.4%) respectively, and four out of five comparisons are statistically significant. These results demonstrate that our simple hybrid-RL framework HRPO not only delivers consistent performance gains over established baselines, but does so with high statistical confidence across the majority of the STEM tasks.\n\n**W2. The authors train the gating mechanism/HRPO separately for each set of tasks.…**\n\n- We thank the reviewer’s concerns regarding the multi-task capabilities of HRPO. Our experiments follow previous works in reinforcement learning with verifiable rewards and latent reasoning, with similar training and evaluation datasets / settings [6-7]. Additionally, HRPO focuses on the algorithmic design of RL-based latent reasoning, and thus remains task‑agnostic and seamlessly supports multi‑task reinforcement learning. For example, we train a single model on the combined MMLU auxiliary dataset and ARC training set, then evaluate it on both benchmarks in our experiments. Similarly, we post-train one model solely on the NQ + HotpotQA data mixture and assess it on multiple knowledge-intensive tasks. Experiments show HRPO reaches the highest average EM with Qwen2.5-3B, outperforming the strongest RAG baseline by 4.3%. Similarly, HRPO with the smaller 1.5B backbone scores an average of 0.337, achieving consistent gains and over previous RL methods like PPO by 3.0%. These results demonstrate that HRPO can be easily extended to multi‑task scenarios similar to baseline RL algorithms like PPO or GRPO. \n\n**W3. The lack of a limitation section with a balanced discussion..…**\n\n- We acknowledge that further discussions on the limitations and impact statement could strengthen the paper. Therefore, we outline our key limitations and societal impact considerations which will be included in the revised manuscript. We hope these could address your concerns w.r.t. the limitations / impacts of HRPO.\n\n- **Limitations:** HRPO could introduce a few limitations compared to existing RL algorithms: it incurs additional computation and hyperparameter overhead due to the added gating parameters and the need to tune $r_{min}$ for optimal performance. In addition, its on‑policy design may reduce large-scale training efficiency compared to off‑policy methods. Moreover, although the hybrid outputs remain interpretable, the reasoning process happens in the continuous states and can be less clear due to unique patterns of HRPO (e.g., cross-lingual reasoning). In future works, we will explore simpler design, off‑policy extensions and advanced latent reasoning techniques to improve the interpretability, efficiency of HRPO and enhance RL-based continuous‑state reasoning.\n\n- **Societal Impacts:** HRPO itself is a task‑agnostic, algorithmic advance, similar to existing methods like PPO or GRPO, and does not by itself encode new domain‑specific behaviors. Its primary societal impact is therefore primarily the use of latent representations for reasoning. The positive impacts include HRPO’s adaptability, performance and concise reasoning, which could reduce compute costs and latency in complex AI systems. One potential risk is that, although latent reasoning remains interpretable, it could make reasoning traces less transparent and thus bring challenges in safety‑critical domains. Overall, we see no high‑level societal risks beyond those already posed by model‑based RL methods, but encourage continued discussions on the design / impacts of HRPO to ensure safety, alignment and transparency when deploying it for post‑training LLMs.\n\n**Q1. Why do GRPO and HRPO plots share almost all artifacts in Figure 3?**\n\n- In our experiments, we use the *identical random seeds and settings* (data, training steps, group / batch sizes etc.) for baseline methods and our HRPO. Given their shared components (reward function, advantage estimation etc.) between GRPO and HRPO, it is expected that GRPO and HRPO training reward curves exhibit very similar trends. As for the rapid early reward increase in HRPO, besides the initial warmup steps, we attribute this to HRPO’s gating component, which effectively learns hybrid representations and therefore yields faster convergence and higher reward plateaus.\n\n**Q2. Is there an experiment that could test the hypothesis in lines 268-270?**\n\n- The hypothesis that higher hidden ratios capture rich context information and shorten completions, can be validated by correlating hidden ratio trajectories with completion length and performance across different $r_{min}$ initializations, as we presented in Figure 4. This effect is more pronounced on knowledge‑intensive tasks, where we retrieve relevant contexts to answer complex multi-hop queries. For example, with the 1.5B backbone on HotpotQA, 2Wiki, and Bamboogle, HRPO’s average output lengths are 11.46, 99.99, and 93.09 tokens, respectively, versus GRPO’s 60.03, 101.82, and 101.93, suggesting consistent length reduction with increased latent context information.\n\n**Q3. What are the limitations of your submission?**\n\n- Please refer to our response to weakness 3.\n\n**Q4. What are the potential societal impacts of HRPO?**\n\n- Please refer to our response to weakness 3.\n\n\n[1] Sutton, Richard S., et al. \"Policy gradient methods for reinforcement learning with function approximation.\" Advances in neural information processing systems 12 (1999).\n\n[2] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).\n\n[3] Zeng, Weihao, et al. \"Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild.\" arXiv preprint arXiv:2503.18892 (2025).\n\n[4] Zuo, Yuxin, et al. \"Ttrl: Test-time reinforcement learning.\" arXiv preprint arXiv:2504.16084 (2025).\n\n[5] Su, Yi, et al. \"Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across \n\n[6] Hao, Shibo, et al. \"Training large language models to reason in a continuous latent space.\" arXiv preprint arXiv:2412.06769 (2024).\n\n[7] Jin, Bowen, et al. \"Search-r1: Training llms to reason and leverage search engines with reinforcement learning.\" arXiv preprint arXiv:2503.09516 (2025)."}, {"review_id": "aErclkYHZt", "reviewer": "Reviewer_K5az", "summary": "The paper introduces Hybrid Reasoning Policy Optimization (HRPO), a novel reinforcement learning (RL)-based framework designed to enhance latent reasoning in large language models (LLMs). Unlike traditional autoregressive chain-of-thought (CoT) methods, HRPO integrates continuous hidden state representations with discrete token sampling, leveraging a gating mechanism to balance these representations. The approach aims to utilize LLMs' intrinsic reasoning capabilities without relying on CoT trajectories, thereby reducing training costs and improving adaptability. The authors claim HRPO outperforms existing methods on knowledge- and reasoning-intensive tasks, maintains interpretability, and exhibits unique behaviors such as cross-lingual patterns and shorter completion lengths.", "questions": "1) Given that computational efficiency is a critical concern for reasoning in large language models, why was there no comparative analysis of HRPO's efficiency against autoregressive CoT methods? Could the authors provide data on training time, inference latency, or computational resource requirements?\n2)  How is the gating mechanism optimized during RL training? Are there specific hyperparameters or loss functions used to control the balance between discrete and continuous representations?", "limitations": "See the Questions.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1) HRPO is the first approach to apply reinforcement learning to hybrid latent reasoning, combining discrete token sampling with continuous hidden state representations. This novel framework leverages stochasticity from token sampling to enable RL-based optimization, bypassing the need for CoT trajectories.\n2) By utilizing LLMs' inherent reasoning patterns, HRPO reduces reliance on extensive CoT annotations, which are computationally expensive and often unavailable. This makes the approach more practical for broader applications.\n3) HRPO-trained models remain interpretable and exhibit intriguing behaviors, such as cross-lingual reasoning patterns and shorter completion lengths, suggesting practical benefits beyond performance metrics.\n\n\nWeaknesses\n1) The paper does not provide a comparative analysis of computational efficiency, a critical factor for practical deployment. The hybrid approach, which combines explicit token sampling with latent reasoning via a gating mechanism, may incur higher computational costs compared to traditional CoT methods, yet this is not addressed.\n2) While the gating mechanism is a core component, the paper lacks details on how it is optimized during training. The process of learning to incorporate richer hidden state features is vaguely described, making it difficult to assess its robustness.\n3) The paper claims HRPO enhances adaptability but provides limited discussion on specific real-world applications where its hybrid reasoning approach offers clear advantages over existing methods, reducing its perceived practical value.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review, we’re delighted you find our work “novel”, “practical”, “exhibit intriguing behaviors” and “outperforms existing methods”. In the following, we present detailed responses and additional experimental results addressing the weaknesses you raised, and we hope these clarifications could resolve your concerns:\n\n**W1. The paper does not provide a comparative analysis of computational efficiency...**\n\n- While our primary focus was on demonstrating the effectiveness of HRPO in improving reasoning performance and reasoning efficiency, we agree that training efficiency is critical for real‑world deployment. As such, we would like to highlight that: (i) our gating mechanism adopts the straightforward recurrent‑modeling design [1-2], which adds minimal compute cost at inference (limited to decoding steps, while prefill processes just the prompt tokens). Thus in our experiments, we find no noticeable effect on inference speed across model sizes or generation lengths (see details in our response to Q1); (ii) compared to existing RL algorithms, HRPO only introduces negligible parameters in its gating (in millions compared to billions of LLM sizes), and thus the resulting model sizes, training speed as well as VRAM footprint remain virtually identical from GRPO; (iii) we have implemented parameter efficient / full fine‑tuning with optimized kernels, where the models are patched for efficient rollout and policy update, we also find that for post-training with 8‑bit AdamW could further reduce both memory and compute overhead. We hope these points clarify HRPO’s efficiency and its negligible compute overhead relative to standard RL-based reasoning.\n\n**W2. While the gating mechanism is a core component...**\n\n- We thank you for raising this point and explain the training process of HRPO in the following. Concretely, the gating and the LLM parameters are learned “end‑to‑end” under our RL objective (Eq. 6), which resembles the REINFORCE algorithm augmented with grouped rewards and standardized advantage estimation [3-4]. In each training iteration, the LLM first generates multiple rollouts for each of the input queries, where the rollouts are evaluated with the reward function to compute their respective advantages. Next, we estimate the policy gradients of HRPO as per Eq. 6, which are used to update the LLM and gating parameters with the optimizer (e.g., AdamW). Aside from the additional gate and our novel hybrid generation strategy, HRPO’s training loop follows the same model‑based RL workflow (e.g., PPO, GRPO) common in RLHF / RLVR. We will expand the descriptions of our training process with more details in our revision, and we hope this could clarify your concerns regarding the training of HRPO.\n\n**W3. The paper claims HRPO enhances adaptability but provides limited discussion on specific real-world applications...**\n\n- We appreciate your valuable comment and expand on practical scenarios where HRPO’s hybrid latent reasoning is uniquely beneficial. We believe HRPO could be particularly useful in the following scenarios: (i) low‑annotation domains: by eliminating reliance on human‑annotated CoT traces, HRPO is ideal for training LLMs in specialized domains where step‑by‑step rationales are scarce or costly. (ii) cost‑sensitive training / inference: the efficient training framework and shorter completion lengths reduce both training costs and inference latency in production settings. (iii) strong reasoning required: on STEM problems and complex multi-hop queries, HRPO’s latent representations encode intermediate reasoning effectively and remain interpretable, potentially improving user’s step‑by‑step feedback efficiency. We thank you for the suggestion and will include additional discussions on the real-world applications of HRPO in the final manuscript to emphasize these points.\n\n**Q1. Given that computational efficiency is a critical concern…**\n\n- In our experiments, we observe negligible difference of training and inference compute due to the minimal design of HRPO. As an illustration, we evaluate GRPO and HRPO on the 1.5B backbone (with 8bit AdamW & LoRA, more details are reported in the experiment settings in our technical appendix) using the GSM8k and MATH benchmarks, reporting training throughput, peak VRAM usage, and inference latency on their evaluation sets. Despite adding the gating mechanism, HRPO incurs only a minor compute and memory overhead while matching GRPO’s inference speed. These differences are negligible in practice, suggesting that HRPO achieves its reasoning improvements with comparable training and inference costs.\n\n- | Metric                          | GRPO    | HRPO    |\n|---------------------------------|---------|---------|\n| GSM8k Training Throughput       | 269.88  | 262.21  |\n| GSM8k Training VRAM (max)       | 11.082  | 11.536  |\n| GSM8k Inference Time            | 16:25   | 16:21   |\n| MATH Training Throughput        | 228.97  | 225.86  |\n| MATH Training VRAM (max)        | 14.154  | 15.222  |\n| MATH500 Inference Time          | 14:15   | 14:04   |\n\n**Q2. How is the gating mechanism optimized during RL training?…**\n\n- We thank you for the question. In HRPO, the gating parameters are randomly initialized and start by favoring the token embeddings to preserve the generative capabilities. In training, the gating weights are learned “end‑to‑end” alongside the LLM parameters via the same policy gradient objective used for the rest of HRPO as in Eq. 6. In other words, we generate rollouts to select high‑reward responses, then minimize their negative log‑probabilities (weighted by advantages) to compute policy gradients. These gradients update both the LLM and gating parameters, with the objective of finding an optimal policy that produces semantically rich hybrid representations for reasoning traces. As for the hyperparameter, we tune $r_{min}$ to control the balance between discrete and latent features (i.e., the lower bound of the gating component’s sampling range). With higher ​$r_{min}$, this ensures that the model draws predominantly from token embeddings, whereas lower $r_{min}$ encourages exploration in the hybrid space and the learning of informative latent representations. We also provide a detailed explanation of our training pipeline in our response to weakness 2 and hope these could clarify the learning of our gating mechanism, we remain available for discussion in case of any further concerns / questions. \n\n\n[1] Orvieto, Antonio, et al. \"Resurrecting recurrent neural networks for long sequences.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] De, Soham, et al. \"Griffin: Mixing gated linear recurrences with local attention for efficient language models.\" arXiv preprint arXiv:2402.19427 (2024).\n\n[3] Sutton, Richard S., et al. \"Policy gradient methods for reinforcement learning with function approximation.\" Advances in neural information processing systems 12 (1999).\n\n[4] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024)."}, {"review_id": "U6Y3fxhVjO", "reviewer": "Reviewer_f57F", "summary": "This paper introduces HRPO using RL to train LLMs for hybrid latent reasoning. The hidden state $h_t$ and token embedding $ê_{t+1}$ exist in different vector spaces and cannot be directly combined. The paper proposes projecting $h_t$ back into the token embedding space by using the predicted next-token probability distribution $p_{t+1}$ to compute a weighted average of the vocabulary embeddings $W_e$. After obtaining compatible inputs, HRPO dynamically blends them via a gating mechanism to produce the final next-step input $e_{t+1}$. Entirely RL-based (i.e., GRPO), HRPO requires no CoT annotations and learns through RL with simple outcome rewards. The latent space encodes information to enable HRPO to generate more concise reasoning steps than traditional CoT.", "questions": "1. What are training time and computational resource requirements?  \n2. The model performs best when relying heavily on either tokens or latent states but struggles in intermediate regimes. What explains this bimodal behavior? Could it reflect fundamentally distinct reasoning modes for different tasks?  \n3. Can HRPO generalize to unseen tasks without RL fine-tuning? How might it scale to larger models (70B+)? Would the optimal gating balance shift with model size?  \n5. Has the projected hidden state  been analyzed? For math tasks, do these vectors cluster near relevant numbers/operators in embedding space? Visualizations or clustering analyses could clarify latent space properties.", "limitations": "yes. But I did not find explicit limitations or broader impact section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1. Combines RL, gating mechanisms, and latent reasoning while addressing key limitations of existing latent reasoning methods (e.g., reliance on CoT data, incompatibility with LLMs).  \n2. Clear structure, with intuitive visualizations (Figures 2 and 3) highlighting the method core ideas and advantages.  \n3. Extensive experiments across benchmarks and ablation studies demonstrate superior performance.\n\nWeaknesses:  \n1. For large models, their native autoregressive reasoning might already suffice. It maybe making HRPO marginal benefits negligible.  \n2. HRPO primarily combines existing techniques: latent reasoning, gating mechanisms (from GRU/LSTM), and KL-penalized RL policy gradients (from GRPO).  \n3. HRPO claims interpretability, but cross-lingual examples suggest abstract, language-independent internal representations. How does this `surface interpretability` (token traces) relate to `intrinsic abstraction` (latent reasoning)? Does the actual reasoning occur in an opaque latent space?", "was_revised": true, "final_justification": "I thank the authors for providing the response. The rebuttal addresses most of the key concerns I initially raised. Based on the authors' response, I increase my rating for the paper.\n\nThe authors clarify the distinction between  `surface interpretability`  and `intrinsic abstraction`  This resolves the logical inconsistency, while also clearly defining the boundaries of the method's interpretability – the true reasoning process occurs within the abstract latent space. This point is now clear, and I consider this a reasonable trade-off.  The authors also commit to including a t-SNE/PCA visualization analysis of the latent space in the final version. And I suggest that the authors include the computational overhead in their paper.", "rebuttal": "We thank you for the insightful feedback and are excited that you find our work “addressing key limitations of existing latent reasoning” and “demonstrate superior performance”. Below, we respond to your questions and weaknesses, we hope these could address your concerns regarding our paper:\n\n**W1. For large models, their native autoregressive reasoning might already suffice...**\n\n- While post-trained LLMs do possess strong native autoregressive reasoning, HRPO still yields consistent & substantial gains by unlocking their hybrid representations. For example, compared with larger LLMs or further RL baselines, HRPO (with the 3B backbone) achieves the highest average EM of 0.378, outperforming a much larger Qwen2.5‑7B baseline at 0.335. Similar performance gains can be found in our STEM experiments, where HRPO demonstrates both improved performance and intriguing reasoning patterns. These results demonstrate that hybrid latent reasoning remains beneficial even when autoregressive reasoning is already strong, providing novel directions for future research in reinforcement learning and LLM post-training.\n\n**W2. HRPO primarily combines existing techniques...**\n\n- We appreciate the reviewer’s feedback on HRPO’s design and respectfully disagree with it. We argue that although HRPO builds on existing foundations in reinforcement learning, recurrence modeling and latent reasoning, we propose a first‑of‑its‑kind generation / training paradigm by learning an optimal combination of continuous and discrete representations. The proposed HRPO is not only novel, but also carefully designed for optimal performance and efficiency. In particular, we overcome several key limitations in our HRPO implementation: (1) projection step: maps hidden states back into the token embedding manifold in a differentiable way while preserving the model’s generative capabilities. (2) learnable gating (Eq. 4): anneals from predominantly token embeddings to richer hybrid inputs, enhancing the training dynamics and utilization of reward signals. (3) on‑policy learning: simple RL objective that fuses discrete and continuous trajectories, enabling accurate policy gradients without a separate value function. The novel hybrid design along with optimal implementation choices of HRPO yields a robust, resource‑efficient framework that consistently outperforms both pure autoregressive and latent‑only approaches across diverse benchmarks, demonstrating HRPO as a scalable, high‑performance paradigm for RL‑based hybrid post‑training.\n\n**W3. HRPO claims interpretability, but cross-lingual examples suggest abstract…**\n\n- We thank you for the insightful comment. Indeed, HRPO remains surface‑interpretable through its token traces while harnessing abstract latent reasoning. The sampled tokens (and probability distributions of latent tokens) still form coherent chains‑of‑thought, making the model’s reasoning path inspectable. And the latent representations are learnt with the objective of maximizing the outcome rewards rather than explainability, which may account for the cross‑lingual patterns that enable token mixing across languages. Conversely, RL‑based latent reasoning approaches may not yield true intrinsic abstractions, since they optimize solely for reward maximization and high‑probability token outputs without explicitly learning interpretable hybrid representations. Consequently, HRPO prioritizes reasoning efficacy (similar to recent efforts in RLVR) by employing latent features to enrich generation performance, with interpretability naturally emerging from the sampled token‑level outputs.\n\n**Q1. What are training time and computational resource requirements?**\n\n- We appreciate the question and agree that training efficiency is critical for real‑world deployment of HRPO. As such, we would like to highlight that: (i) our gating mechanism adopts the straightforward recurrent‑modeling design [1-2], which adds minimal compute cost at inference (limited to decoding steps, while prefill processes just the prompt tokens). Thus in our experiments, we find no noticeable effect on inference speed across model sizes or generation lengths (see details in our response to Q1); (ii) compared to existing RL algorithms, HRPO only introduces negligible parameters in its gating (in millions compared to billions of LLM sizes), and thus the resulting model sizes, training speed as well as VRAM footprint remain virtually identical from GRPO; (iii) we have implemented parameter efficient / full fine‑tuning with optimized kernels, where the models are patched for efficient rollout and policy update, we also find that for post-training with 8‑bit AdamW could further reduce both memory and compute overhead. Therefore in our experiments, we observe negligible difference of training and inference compute due to the minimal design of HRPO. As an illustration, we evaluate GRPO and HRPO on the 1.5B backbone using the GSM8k and MATH benchmarks, reporting training throughput, peak VRAM usage, and inference latency on their evaluation sets. Despite adding the gating mechanism, HRPO incurs only a minor compute and memory overhead while matching GRPO’s inference speed. These differences are negligible in practice, suggesting HRPO's efficiency and its potential to be trained using only consumer hardware.\n\n- | Metric                          | GRPO    | HRPO    |\n|---------------------------------|---------|---------|\n| GSM8k Training Throughput       | 269.88  | 262.21  |\n| GSM8k Training VRAM (max)       | 11.082  | 11.536  |\n| GSM8k Inference Time            | 16:25   | 16:21   |\n| MATH Training Throughput        | 228.97  | 225.86  |\n| MATH Training VRAM (max)        | 14.154  | 15.222  |\n| MATH500 Inference Time          | 14:15   | 14:04   |\n\n**Q2. The model performs best when relying heavily on either tokens or...**\n\n- Our results in Table 3 reveal a bimodal pattern: performance surges at low and high hidden‑ratio settings but dips around the midpoint. We interpret this as two complementary reasoning regimes: token‑dominant mode generates detailed, step‑by‑step CoT, ideal for tasks demanding precise, literal reasoning. On the other hand, latent‑dominant mode leverages compact, context‑summarizing hidden representations, which excel on abstract or knowledge‑intensive problems (e.g., RAG). Balancing both modes equally appears to dilute each regime’s strengths, introducing interference rather than synergy. We also note that our exploration of these hybrid representations is still in its early-stage, as these are learned solely through reward maximization, and a deeper analysis of their behavior and interpretability remains an important direction for future work.\n\n**Q3. Can HRPO generalize to unseen tasks without RL fine-tuning?...**\n\n- Concurrent to our paper, [1-2] show that latent representations could generalize to unseen tasks for large‑scale post-trained models even without fine-tuning. While these findings establish zero‑shot latent reasoning as viable for post‑trained LLMs, RL‑based fine‑tuning remains impractical as it requires stochastic sampling to optimize the policy model. Therefore, the proposed HRPO represents the best of both worlds by introducing hybrid representations with stochastic token sampling. In addition, HRPO focuses on the algorithmic design of RL-based latent reasoning, and thus remains domain‑agnostic and seamlessly supports generalization via reinforcement learning. For example, we train a single model on the combined MMLU auxiliary dataset and ARC training set, then evaluate it on both benchmarks in our experiments. Similarly, we post-train one model solely on the NQ + HotpotQA data mixture and assess it on multiple knowledge-intensive tasks. These results demonstrate that HRPO can be easily extended to multi‑task / domain generalization settings. Paired with our careful design and implementation, RL-based hybrid reasoning delivers both superior performance and enhanced efficiency on reasoning‑intensive tasks. Regarding larger models, we are unable to perform extensive RL training on large models across all selected datasets due to our limited time and compute resources, we leave this exploration and the analysis of the optimal gating balances across model scales as future work of HRPO.\n\n**Q4. Has the projected hidden state been analyzed?...**\n\n- While we did not perform explicit clustering of the projected embeddings in this work, our qualitative examples (as in Sec. 4 and App. C) reveal that these vectors encode semantically meaningful information: they surface as language‑agnostic token traces and shorten completion lengths when rich context is captured. We also agree that visualized analysis could provide valuable insights of HRPO, especially for numeric and operator tokens in math tasks. Accordingly, we will incorporate additional visualizations (e.g., t‑SNE or PCA of the learnt latent representations) and in‑depth discussions of their structural and semantic properties. We hope these clarifications and analyses address the reviewer’s concerns and highlight HRPO’s contributions.\n\n[1] Zhang, Zhen, et al. \"Soft thinking: Unlocking the reasoning potential of llms in continuous concept space.\" arXiv preprint arXiv:2505.15778 (2025).\n\n[2] Zhuang, Yufan, et al. \"Text generation beyond discrete token sampling.\" arXiv preprint arXiv:2505.14827 (2025)."}]

Author rebuttal:

Meta-review: The paper presents a novel RL-based approach for hybrid latent reasoning in LLMs. The method's key ingredient is a learnable gating mechanism that progressively integrates continuous hidden state representations from previous steps with discrete token embedding. 

The proposed approach is novel and consistently improves upon alternatives on various benchmarks. The paper is well written and presents many valuable insights.

The authors have done an excellent job at addressing the reviewers concerns as appreciated by all reviewers. In particular they have provided statistical significance results, intuitive rationale and  details on the design of the gating mechanism, as well as computational efficiency result, and evaluation against Coconut. 

We strongly urge the authors to incorporate all of the valuable materials and clarifying points into their manuscript.
Decision: Accept (poster)