Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains
Authors: Wenhui Tan
Jiaze Li
Jianzhong Ju
Zhenbo Luo
Ruihua Song
Jian Luan
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
LLM Reasoning
Primary area: deep_learning

Abstract: Large Language Models (LLMs) achieve superior performance through Chain-of-Thought (CoT) reasoning, but these token-level reasoning chains are computationally expensive and inefficient.
In this paper, we introduce Compressed Latent Reasoning (CoLaR), a novel framework that dynamically compresses reasoning processes in latent space through a two-stage training approach.
First, during supervised fine-tuning, CoLaR extends beyond next-token prediction by incorporating an auxiliary next compressed embedding prediction objective. This process merges embeddings of consecutive tokens using a compression factor $c$ randomly sampled from a predefined range, and trains a specialized latent head to predict distributions of subsequent compressed embeddings. Second, we enhance CoLaR through reinforcement learning (RL) that leverages the latent head's non-deterministic nature to explore diverse reasoning paths and exploit more compact ones.
This approach enables CoLaR to: i) **perform reasoning at a dense latent level** (i.e., silently), substantially reducing reasoning chain length, and ii) **dynamically adjust reasoning speed** at inference time by simply prompting the desired compression factor.
Extensive experiments across four mathematical reasoning datasets demonstrate that CoLaR achieves 14.1% higher accuracy than latent-based baseline methods at comparable compression ratios, and reduces reasoning chain length by 53.3% with only 4.8% performance degradation compared to explicit CoT method. Moreover, when applied to more challenging mathematical reasoning tasks, our RL-enhanced CoLaR demonstrates performance gains of up to 5.4% while dramatically reducing latent reasoning chain length by 82.8%.
The code and models will be released upon acceptance.

Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains
Wenhui Tan
Jiaze Li²
Jianzhong Ju
Zhenbo Luo²
Ruíhua Song les
Jian Luan 2
'Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China MiLL Plus, Xiaomi Inc., Beijing, China
"Let's think step by-step:"
LIM
(14 tokens)
11'' 1S
think faster:" "
"Question: 7 spoons cost $21, what's 's the price buy 5
latent.
latent 2
lotent 3
letent
CoLaR Moddl
Annswr: 15"
"Let's think faster:"
Iatent
Latent 2
Rigt
155
Figure I: Our proposed Compressed Latent Reasoning Model (CoLaR) performs dynamic-speed rea- soning by auto-regressively predicting latent variables, each compressing information from multiple word tokens. Simply prompting to reason faster enables CoLaR to predict more informative latents.
Abstract
Large Language Models (LLMs) achieve superior performance through Chain- of of-Thought (CoT) reasoning, but these token-level reasoning chains are compu- tationally expensive and inefficient. In this paper, introduce Compressed La- tent Reasoning (CoLaR), novel framework that dynamically compresses reasoning processes in latent space through two-stage training approach. First, during super- vised fine-tuning CoLaR extends beyond next-token prediction by incorporating auxiliary next compressed embedding prediction objective. This process merges embeddings consecutive tokens using a compression factor c randomly sampled from predefined range, and trains specialized latent head predict distributions of subsequent compressed embeddings. Second, we enhance CoL.aR through rein- forcement learning (RL.) that leverages the latent head's non-deterministic nature to explore diverse reasoning paths and exploit more compact ones. This approach enables CoLaR to: i) perform reasoning at a dense latent level (i.e., silently), substantially reducing reasoning chain length, and ii) dynamically adjust reason- ing speed at inference time by simply prompting the desired compression factor. Extensive experiments across four mathematical reasoning datasets demonstrate that CoLaR achieves 14.1% higher accuracy than latent- bastd baseline methods comparable compression ratios, and reduces reasoning chain length by 53.3% with only 4.8% performance degradation compared explicit CoT method. Moreover, when applied to more challenging mathematical reasoning tasks, our RL-enhanced CoLaRR demonstrates performance gains of up to 5.4% while dramatically reducing latent reasoning chain length by 82.8%.
[30 Corresponding authors: Ruihua Song (rsong @ ruc.edu.cn) and Jian Luan (luanjian @ xiaomi.com). Project page: httpp::/ h/pp::///ithin thub wwwccoomiiiicccee xiaomi- research/colat "This Work was performed when Wenhur Tan was visiting Xıaomi as a research intern.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities in mathematical reason- ing, particularly when employing Chain-of-Thought (CoT) prompting techniques [29] 31]. Recent advances have further highlighted the potential of this approach when combined with reinforcement learning on extended reasoning sequences [66] 12 28 revealing significant "aha-moments" in model performance. Despite these advances, a critical limitation persists: generating lengthy reasoning chains computational costly. impeding efficiency and scalability. This inefficiency becomes par- ticularly evident in world LLM applications, where extended reasoning chains create substantial server load, especially under high-concurrency conditions, underscoring the urgent need for more efficient reasoning methods.
Several approaches have emerged to address this computational challenge 126 71. One line of
research focuses on enhancing efficiency at the token level, primarily by identifying and skipping less informative tokens 30), prompting models to generate more concise reasoning steps [32 11. or
dynamically terminating reasoning when the model exhibits high confidence in a trial answer [35]. While valuable, these methods continue to operate on sparse boken-based representations. A more promising direction explores reasoning within the dense latent space. Initial efforts attempt to "internalize" reasoning knowledge by curriculum learning (5] or knowledge distillation [6]. Some works focus on the potential inside L.LMs by looping or skipping some intermediate LLM layers [ 22 to realize efficient reasoning. Recent innovations have introduced auto-regressive prediction of latent representations for efficient reasoning. Coconut 13
proposes to gradually replace token-level employs self-distillation to transfer CoT knowledge into latent reasoning processes. However, these methods primarily utilize fixed-length reasoning chains, resulting in suboptimal efficiency and limited adaptability. Furthermore, to the best of our knowledge, all these latent-based methods employ deterministic latent reasoning processes, overlooking the potential benefits of exploration-exploitaea
reasoning with latent representations, while CODI
24
capability may bring about.
To overcome these limitations, we introduce Compressed Latent Reasoning (CoLaR), a novel framework that dynamically compresses LLM reasoning chains into latent space while preserving xxploattioneepllotatiio capabilities. Our approach utilizes an auxiliary next compressed embedding prediction task in supervised fine-tuning (SFT) stage. Specifically, each training step, CoLaR first samples a random compression factor c E [1, Cmax] and merges the embeddings of c consecutive reasoning tokens using our Embedding Compress module. A Latent Head is then trained predict the next compressed embeddings from the LLM's output hidden states, which is a fully parallelized process. During inference, CoLaR is capable auto-regressively predict dense and informative latents with the Latent Head, and automatically determine when to terminate the reasoning process with LLM's Language Head. Rather than predicting deterministic values, the Latent Head outputs a probability distribution that produces diverse reasoning pathways for a same question input. Based on this, we further enhance CoLaR through post-training with Group Relative Policy Optimization (GRPO) reinforcement learning algorithm (author?) [23 37, which enables CoLaR to explore correct latent reasoning paths with diverse outputs and exploit those shorter ones.
Our extensive evaluations on four grade-school level mathematical reasoning datasets (GSM8k [4]. Ghrd [9] SVAMP 20), and MultiArith (21) demonstrate that CoLaR achieves a 14.1% improvement in accuracy compared to state-of-the-art baseline methods at comparable compression ratios. Furthermore, CoLaR reduces reasoning chain length by 53.3% I with only a 4.8% performance degradation relative to explicit COT. Finally, experiments on a more challenging dataset MATH 14] demonstrates the potential of CoLaR to reinforcement learning, gaining up 5.36% accuracy while reducing the length of reasoning chain significantly by 82.8%
Our main contribution are three-fold:
We introduce Compressed Latent Reasoning (CoLaR), a novel framework enabling dynamic- speed reasoning by regressesss predicting latent variables that encapsulate the com- pressed semantics of multiple word tokens. This allows for more efficient reasoning by operating in a compressed latent space.
We design CoLaR with a probabilistic Latent Head and demonstrate the effectiveness of reinforcement learning on latent reasoning. This combination improves performance and reduces the length of reasoning chains by encouraging exploration of diverse reasoning paths and exploitation of the shorter ones.

---

Extensive experiments show that CoLaR achieves 14.1% accuracy improvement over existing latent-based methods. Furthermore, reinforcement learning enhances performance by up to 5.36% while simultaneously reducing reasoning chain dramatically length by 82.8%, demonstrating significant efficiency gains.
2 Related Work
2.1
Explicit LLM reasoning
Recent advances have demonstrated the strong reasoning capabilities of large language models (LLMMs.. The explicit reasoning approach, exemplified by Saain-of-Though (Co')) reasoning [29]. propose to prompt LLMs to generate intermediate reasoning steps through sequential token prediction before generating answers 36
31
27
39 17]. Subsequent work demonstrated that reinforcement
learning techniques 23 37 38 can further improve performance on verifiable reasoning tasks
like mathematical problem-solving revealing an "aha-moment" that significantly boosts model's
performances with longer thinking process [16 I2 28).
However, the computational cost of processing these lengthy reasoning chains remains significant bottleneck, motivating research into efficiency optimizations. Current solutions focus on identifying and skipping redundant tokens [30 [35 or encouraging more compact reasoning patterns using mathematical notations and programming language-like expressions [ 32. While these methods reduce reasoning chain length, they are fundamentally limited by the sequential token prediction.
2.2 Latent LLM reasoning
Latent reasoning approaches operate a denser, continuous space, abstracting away from indi- vidual word tokens. These methods can be broadly categorized into three directions: knowledge interalizattion, architectural modifications, and auto-regressive latent reasoning.
The first direction, knowledge internalization, aims to embed reasoning capabilities directly into attempts to internalize reasoning knowledge by progressively removing explicit reasoning steps during training, while Pause [10 proposes training models to reason within
the model. iCoT-SI S
specialized token embeddings.
The second direction exploits the hierarchical structure of transformer layers, with proposals to dynamically skip repeat layer computations [22 3 25 19]. These methods aim reduce
computational cost by selectively processing different layers.
The third direction, and most relevant to work, explores retterrrgreesvve latent reasoning [33 Coconut (13) pioneered this approach by replacing token sampling with hidden state concatenation for breadth-firs reasoning, while CODI [24] introduces auto-regressive latent variable model through self-distillation. However, existing methods like Coconut and CODI are limited by their reliance on fixed-length reasoning chains due to the implicit nature of latent variables. Furthermore, they employ a deterministic approach to auto-regressive latent generation, neglecting the potential for exploration-exploitaiion strategies further enhance model performance, particularly within reinforcement learning framework.
In contrast, CoLaR advances ruto-regressive latent reasoning by introducing a novel next com- pressed embedding objective. This allows the model to capture the semantics of multiple word tokens within single latent variable and reason with dynamic chain lengths, leading to improved efficiency and performance. Moreover, CoLaR achieves significant performance gains and a dramatic reduction in latent reasoning length through reinforcement learning with a probabilistic latent prediction head.
3 Method
In this section, we introduce task, notations, and proposed method CoLaR. We foeus mathematical reasoning tasks using a dataset D, which consists of a question t, = L 1:L, reasoning chain t, = 1,: and an answer a, = 11:La, where I.g. Tor, and I.g denote the respective token lengths. A representative example entry would be: "Question: A set of spoons costs $21. If each
3

---

NLL
2173x13
CE
001: penalize
Answer: 24"
Answer 15
22
Latent Head
Language Head
02i 82 8
Answer: 16"
Compressed Latent Reasoning Model (CoLaRR
1
03:
62 22
Answer 15 dg
49
reinforce
Embed Compress
LLM Taken Embed
e;
Cr
LLM Token Embed
-04
Answer: 15"h a4
27
"Question: Let's
LLM Token Embed
CoLaR-RL RL
"Answer: 15"
think = faster
221 3
"Question: Let's think 2 faster
Figure 2: Our proposed method CoLaR consisting LLM backbone and Latent Head. During the SFT stage (left), for each training step, CoLaR first compresses embeddings e, of the original reasoning chain into compressed embeddings e₂ with compression factor randomly selected from the range [1, Gmaz]- Then, CoLaR is trained to predict: i) the compressed reasoning embeddings via the Latent Head, and ii) the compressed reasoning tokens and answer tokens through the Language Head. During the RL stage (right), for every question input, CoLaR samples a group of G outputs 01:G consisting of the latent reasoning chain and the predicted answer. We then calculate the relative rewards @1:G for each output, and the rewards are averaged on each token (0;)) encouraging CoL to explore diverse latent reasoning pathways and exploit those more compact ones.
spoon would be sold separately, how much would spoons cost?", "Reasoning chain: <21/7=33 5 * 15 <end>", ", and "Answer: 15".
Given an LLM backbone M, the input tokens are first mapped to embedding vectors ey Eq 1:1k
e, = c and e₂2 e.iLa.. These embeddings are processed by .M to produce the hidden states of the final layer, denoted h, h, 1 h, h,: and h I:la, M then predicts distributions a.k.a. logits of next tokens using a Language Head.
To address this issue of lengthy reasoning chains, we propose compressing reasoning processes into a denser latent space, facilitating more efficient LLM reasoning. This requires our method to i) compress reasoning tokens into latent space and understand these dense representations, ii) predict subsequent dense latent representations and determine when to terminate reasoning, and iii) maintain the ability explore diverse latent reasoning paths and exploit shorter latent solutions. CoLaR is designed with these three objectives in mind.
3.1 Reasoning token compression and understanding
As illustrated in Figure 2 the input to CoLaR can be represented as c c= (cc,, Ce: Cal, where [: denotes concatenation. Here, C₂ represents the compressed embeddings derived from Cr, the em- beddings of the original reasoning steps, and the length of compressed embeddings Le = [ To achieve dynamic test-time compression factor c, we begin each training step by randomly sampling [1, Tmax] For every consecutive reasoning token embeddings e lkk+t the Embedding Compress module generates compressed embedding e
straightforward approach apply mean pooling directly to these embeddings. However, due to the high dimensionality of the embedding space (e.g., 2048 dimensions), embeddings from different tokens may be highly uncorrelated. Simply averaging these embeddings can distort the original distribution. For instance, consider two uncorrelated distributions A NN o²) and B ~ NMH,, a²); mean pooling would alter the original distribution to A+B NW, f), effectively scaling the variance by 1 We found that, for most pre-trained LLMs. the distributions of embeddings are centered at RJ O. Thus, to prevent distortion of the original embedding distribution of LLMs, the Embedding Compress module only scales the sum of the e embeddings by V
Intuitively, could be difficult for LLMs to understand these compressed embeddings. simple approach is to supervise M to predict answers with a language modeling loss, which enforces M to model answers with compressed embedding inputs. However, this objective provides supervision signals that are too sparse to converge to near-optimal performance. To address this issue, train

---

CoLaR to predict the compressed reasoning tokens. Ideally, when using compression factor CoLaR should be able read and predict tokens in groups of This means that for each compressed embedding input, CoLaR should be trained to predict all e corresponding tokens. 'oo approximate this multi-label classification task using the single-label prediction capability of L.LM's s language
model head, we randomly sample one token from each group of reasoning tokens tr ( as the ground-truft label. This approach trains the predicted logits to approximate multimodal distribution that represents all potential tokens in each compressed group. This process could be formally represented as:
La the
Ceomp
Σ
log p(tee tol' ea] ea] eg))
(I)
La Le
where tr are sampled from t,
3.2 Next compressed embedding prediction
To enable auto-regressive latent reasoning, we train a Latent Head & (analogous to the Language Head in LLMs) predict the compressed embedding, where & is two-headed MLP. Given the current hidden states hyg output by M, the Latent Head & predicts both the mean p and standard deviation o of the next embedding's distribution.
Unlike previous works that predict deterministic values-whice limits exploration of alternative reasoning pathways-our approach generates probabilistic distribution. During inference, employ the re- e-aaameteriiztionn trick to sample the next embedding: e: = f + o c, where c random noise sampled from a standard Gaussian distribution (0 1).
The Latent Head is primarily trained using the negative ligellkldoood (NLL) loss. For a prediction at position i, this can be formulated as:
(e
P2) + log o'
Iatent (i) = log P/. 6.)
25
This probabilistic formulation enables the model to capture uncertainty in the latent reasoning process and allows for diverse reasoning pathways during generation. The total loss is computed by averaging over all positions in the compressed embedding sequence.
However, empirically found that CoLaR with NLL loss tends to under-fit simpler math reasoning datasets that require less exploration. To address this, we propose a soft-MSH loss that combines Mean Squared Error with entropy regularization term:
Llatent (2) = E. [(pL + ge -
et)
(2re (o:)3)
a
entropy
where a is positive hyperparameter that encourages the model to predict more diverse latents with larger à values. This approach enables CoLaR to better fit simpler datasets while maintaining its exploration capability. We evaluate both the two forms of latent loss in our experiments. We sum up Loomp and Liatent as the final loss to optimize CoLaRin the SFT stage.
3.3 Exploration with reinforcement learning
The next-compressed embedding prediction training enables latent reasoning chains to mimic original chain of thoughts. However, this paradigm inevitably limits the performance of latent reasoning models to their CoT teachers. To further explore the potential of CoLaR, we apply reinforcement learning stage our proposed method.
With the Latent Head trained to predict distributions, we can sample diverse latent reasoning pathways and final answers for the same question g. We then apply Group Relative Policy Optimization (GRPO) algorithm (23 to reinforce correct reasoning chains and answers while penalizing incorrect ones.
Specifically, for each question q, GRPO first samples a group of outputs (01, .., og from the old policy Tew where G the group size. Each output o, consists of a latent reasoning chain and final answer. Then, GRPO optimizes the policy Ty by minimizing the following objective:
5

---

G
Ha (oi
no (oi19) qq)
CGRPO G
Σ
min
A, clip
e,
(4)
G
(o:1g)
Team (o:4q)
1=1
where is a hyperparameter, and A; is calculated as a group-normalized reward:
4 T mean (ri, r,, rG)
(5)
std (71. r2....,rG)
We simply set r; to 1 when an answer is correctly predicted and to otherwise. Following DAPO 37 we remove the KL-regularization term from original GRPO implementation for efficient training.
Notably, LGRP is calculated at the output level, i.e., across the entire latent reasoning chain and predicted answer, but is then averaged when applied to each latent/token. This design encourages CoLaR to balance exploration and exploitation. For instance, in Figure 2 although both 41 = 02 = -1, GRPO penalizes the latents/tokens in 0r more, as there are fewer reasoning steps. This encourages CoLaR to think more deeply to explore correct reasoning paths. Likewise, the latents/tokens in 04 reinforced more as the reward is averaged less, which encourages CoLaR to exploit those more compact latent reasoning paths.
4
Experiments
In this section, we evaluate our proposed method CoLaR against strong baselines, analyze the contributions of different components, and explore the impact of key parameters.
4.1 Experimental setup
Datasets and tasks. Our method is mainly trained and evaluated on GSM8k-Aug 6 an augmented version of the Grade-School level Math reasoning dataset GSM8k 4 GSM8K-Aug comprises approximately 385k training samples and 1k test samples. We also evaluate the trained methods on three out-of-domain math reasoning datasets: (1) GSM-Hard [9, a modified version of GSM8K
with approximately 1k test samples featuring larger magnitude numbers, (2) SVAMP 20 and (3) MultiArith [21]. two simpler math reasoning datasets with 1k and 600 test samples, respectively. Moreover, we train and evaluate our method on a more challenging dataset MATH 14] which consists of 7.5k training samples and 5k test samples, covering algebra, calculus, statistics, geometry, linear algebra and number theory. Following [3], we use two metries: (1) Accuracy (Acc.), which measures the effectiveness of correctly predicting answers and (2) Reasoning chain length (# L), which measures efficiency by averaging the number tokens/latents predicted in reasoning chains.
Baseline methods. We primarily compare against the following baselines: (1) CoT [29]. which is
fine-tuned on complete reasoning chains and answers, and performs token-level reasoning before predicting answers during inference; (2) iCoT (5), which internalizes reasoning knowledge by gradually removing reasoning steps, and directly predicts answers during inference; (3) Coconut [3] which is fine-tuned with a curriculum process to gradually replace token-level reasoning steps with latent reasoning steps, and performs six steps of latent reasoning before predicting answers; and (4) Distill, is our reproduced version of CODI (24] based their implementation details as the code and model are not released. self-distills leven-level CoT into fixed-length latent reasoning steps, with an inference stage same to Coconut.
Implementation details. (1) Base model: unless otherwise specified. all experiments use frozen Llama-3.2 1.---...---B--B- IIstructt (11) backbone with trainable LoRA module [15). Following Coconut, all methods are initialized with weights from CoT-SFT accelerate training. (2) Model checkpointing: for fair comparison, all models are trained for up to 50 epochs or 12 hours, whichever is reached first, and we choose the checkpoint that achieves the best accuracy on the validation set as the final model. (3)
we use the AdamW W [18] optimizer with fixed learning rate of le-4 and to
weight decay le-2 in SFT stage, and set the learning rate to le-6 in RL stage. We set I' Fmazz train CoLaRR During inference, we configure the LLM generation with temperature of and top-p of 0.9. All SFT training processes are conducted a total batch size of 256. For more implementation details, please refer to Appendix Section

---

Table 1: Experiment results of baseline methods and CoLaR on four grade-school math reasoning datasets. We test the methods for five times with different random seeds to report the averaged number and 95% confidence interval (+) on accuracy (Acc. %) and reasoning chain length (# L). CoLaR-c denotes a same CoLaR model tested with different compression factors c. For ablation methods (marked in gray), suffixes DL. OC, MP and NLL denote CoLaR with a Deterministic Latent head, training withOut Compressed reasoning chain in cross entropy labels, using Mean Pooling to compress embeddings, and training with NLL loss, respectively.
GSMSk-Aug
Hard
SVAMP
MuliiAih
Average
Avc.
Ace.
41.
Acc.
#1
Aue.
Acc.
CGT
40.42.72 72
25.6g.11
11.04
34.22.13
30.84
12.12.00
93.24.44
13.72.08
53.6
21.4
19.8 23
0.004..0 .0
3.87 1.874.1
16
0.00=.00 00
36.4+.54 .11
0.00++.00
38.2+.c0 .08
0.00++00
24.6
0.00
Coconut
23.14.26
6.004.00
6.49
6.00-.00 -00
40.74.06 .65
6.001+.00 -.00
41.14.22
6.004.00
27.6
6.00
Dieill
13.32.0
6.004.00 .00
2.97
6.00=
21.7
6.00
19.2#
6.00
14.3
6.00
CoLaR-5
26.84
5.07
0.87
6.53
48.44
2.90
86.44
3.21
41.7
4.57
26.74
5.74
8,20-.00
2.90
84.54. 19
3:22+.01
41.3
5.02
24.8 27
5.14 12
6.46 11
5.49 06
46.5 18
2. 2.85 01
85.94 22
3.13 01
40.1
4.15
MP
20.01.
5.61 a on
4.20
6,18
47.74
2.96 1
80.71.50
3:201.01
38.3
4.49
NLL
20.34.64 .4
5.994..0
4.02
16.0-.22
43.9
3.06
81.0+.2a
3.20+.02
37.6
8.01
CoLaR-2
40.1
12.74
9.08
14.0=
54.94
6.11
91.34
7.35
48.8
10.0
DLL
39.71.
12 12.81..0
8.84
17.2_
54.31
6:10 01
90.11.11
7.461.01
48.2
10.9
39.11. 33
31.01
8.961 01
16.9 13
54.74: 18
6.08 02
90.11: 25
7.301.01
48.2
10.8
MP
36.9 36.91.10
12 1.441.0
8.46
12.0
54.14.4
6.14 01
86, 86.6 20
7.434.01
48.6
9.49
NLL
32.31.51
12 12.21.04
7.
16.62.90
51.01.24
5.50
88.31.41
7.09
44.8
10.3
4.2 Comparison to baseline methods on GSM datasets
Table presents comparison of CoLaR against state-of-the-a baseline methods on four grade- school level math reasoning datasets. CoLaR demonstrates consistent performance gains over existing latent-based reasoning approaches. Notably, CoLaR with a test-time compression factor of 5 (CoLaR- 5) achieves a 14.1% improvement in average accuracy compared to Coconut, and does so with fewer reasoning steps (4.57 vs. 6.00). This advantage stems from our effective next-compressed embedding prediction objective, which efficiently compresses the reasoning process into compact and informative latent variables. This allows for superior performance while maintaining high compression ratio.
Leveraging the dynamic compression design, we evaluated the same trained model with a different test-time compression factor of 2 (CoLaR-2). The resulting accuracy of 48.8% represents only a 4.8% decrease compared to explicit CoT, but with a significant 53% reduction in reasoning chain length.
Furthermore, CoLaR exhibits robust out-of-domain generalization capabilities compared to other latent- baselines, particularly on the MultiArith dataset, where CoLaR shows minimal perfor- mance degradation compared to CoT, while other latent-based methods suffer significant drops.
4.3 Ablation studies of on GSM datasets
We conduct ablation studies with four experimental settings on the GSM datasets, with results illustrated the gray areas of Table Three key findings emerge:
(1) Simple math questions require balanced exploration capability. Comparing CoLaR with CoLaR-DL (trained with deterministic latent head) and CoLaR-NLL (trained with NLL loss), find that a deterministic latent head fits well on simple math datasets but lacks test-time exploration potential, leading suboptimal performance. Conversely, training CoLaR with NLL loss introduces excessive randomness, resulting in poor fit on the training data and worse overall performance.
(2) Dense reasoning supervision signals are crucial. When comparing CoLaR with CoLaR-OC, where we remove the tokens of compressed reasoning chain te and use only the final answer tokens t, as the language modeling supervision signal, performance degrades by 1.6% and 0.6% at compression factors e = 5 and e = 2, respectively. This confirms the importance of dense supervision signals when training latent-based reasoning methods.
(3) Latents under different compression factors should share the same space. CoLaR-MP, which applies Mean Pooling on the compression embeddings, shows 3.4% and 2.2% performance degradation compared to our method. This decline primarily attributed to distribution shifts caused by the compression process, which introduces confusion during model training.
7

---

Table 2: Experimental results on the challenging MATH dataset. We evaluate our proposed method CoLaR on two base models and three settings: -DL denotes using Deterministic Latent head, -NLL denotes CoLaR trained with NLL Loss as Liaten, which is our main method, and /w GRPO denotes the post-trained CoLaR-NLL with GRPO reinforcement learning process. We calculate the performance gain between CoLaR-NLL and CoLaR-NLL-RL to highlight the effectiveness of reinforcement learning. Compression factor and Lmaa are set to 2 and 128, respectively.
DeepSeck- RJ --iisiill-Qweee Qwen-1.5B Acc.
Llama-3.2
Acc.
CoT
23.5
209 +1.6
1.71=.33
210+1.4
CoLaR-DL
9.04
99 99.41.25
3.07=
1344.46
CoLAR-NLL
8.94
56.8
5.28_.16
CoLaR-NLL--RI
14.3
(5.36% 1)
9.79
(82.8% !)
7.08
(1.80% 1)
16.14.14
(80.6% 4)
w/o average
13.84.14
128
0.00=.0
128.0=.or
Less
'20"
'5'
'1'
>>>n'
'
'1'
'16
'2'
<<<'
$4'
Similar
'22'
'6'
4
'4'
$4'
$14"
$21'
22
'2'
<<<
8
2
33'
'2
'7'
"4'
$15'
Most
$21"
'7'
5
4=
$55'
'
$57
'=
Similar
<<<!
4/²
$3'
$3'
$3
$3'
Embedding retrieval (top-5/128k)
22
è2
è3
25
é?
è2
è3
2
CoLaR Model
Question: "A set of spoons costs $21 If each spoon would be sold separately, how much would 5 spponns Ground-trule reasoning chain (not input model): "c- 217---55
Figure 3: A case study on the GSM-8k validation set. We set the compression factor to 2 and which produce two latent reasoning chains in length 7 and 3, respectively. We then retrieve tokens with the predicted latents by embedding cosine similarity, and underscore those informative tokens.
4.4 Reinforcement learning results the MATH dataset
We train and evaluate CoLaR with RL on the challenging MATH dataset, using two base models [12
34 [[]. The results are presented in Table 2 Our analysis of the results yields three key conclusions:
(1) Exploration is crucial for difficult problems. The deterministic latent reasoning process of CoLaR-DL exhibits accuracy comparable or worse than CoLaR-NLL despite longer reasoning chains. This suggests that challenging math problems necessitate exploration of multiple potential solutions, rather than deterministic, step-by-step reasoning. Furthermore, post-training CoLaR-NLL with RL yields significant gains, achieving up to 5.36% higher accuracy and an 82.8% reduction in reasoning length. This highlights the potential of RL. and the importance of balancing exploration and exploitation for latent reasoning models.
(2) Averaged rewards promote exploitation. When training without this averaging (i.e., simply dividing the loss by a constant to normalize the loss scale), we observed that while Qwen-1.5B exhibited performance increase (from 8.94% to 13.8%) similar to averaging the loss, the reasoning length rapidly converged to the pre-defined upper limit. Moreover, Llama- 1B's performance tend towards collapse. This suggests the averaged design encourages CoLaR to exploit more efficient reasoning pathways.
(3) Base model quality impacts RI. effectiveness. Supervised fine-tuning CoTT resulted in varying performance across the two base I.LMs. Meanwhile, CoLaR also demonstrates a significantly larger performance gain on RL when using the higher-quality Qwen-1.5B compared to Llama-1B IB. This

---

50
45-
25
40
20
Aguuniy
35-
30
10
20-
20
10
Compression
Compression Factor
Figure 4: Accuracy and reasoning chain length (# L) of CoLaR on GSM8K dataset when trained with random E [1, 5] (the lines) or trained solely specific (the bars).
Figure 5: Accuracy and reasoning chain length (# L) of CoLaR on GSM8K dataset when trained with {1.3, 3, 5, 7} and tested with extra {2, 4, 6} (under gray bars).
Table 3: Experimental results of CoLaR-2 and TokenSkip on GSM-8k, MATH-500, and GPQA.
Acc. (%)/#L
CoT-IB
CoLaR-IB
CoT-8B
CoLaR-8B
CoLaR-8B-RL
TokenSkip-8B
GSM-8k
47.5/92.6
40.4/49.2
76.5/93.6
70.8/47.3
71.9/13.3
78.2/113
MATH-500
28.6/176
19.8/84.6
54.6/168
45.8/67.7
52.4/17.6
40.2/292
GPQA
26.4/232
26.4/84.1
35.7/216
32,4/101
37.5/66.7
observation aligns with the findings of 8 which suggests that RL substantially activates inherent reasoning capabilities, indicating the importance of base model quality.
We also observed that during the RL training process, CoLaR tends to think longer initially with rapid rise in accuracy, followed by phase of thinking shorter accompanied by more stable increase in accuracy, aligning with the discussion in Section 3.3 Due to space constraints, the detailed training curves are provided in Appendix
4.5 Scaling CoLaR on larger base model
We scale CoLaR the 1-Billion and 8- 8-Billion parameter Llama checkpoints verify that its compression and accuracy gains grow with model capacity. The 8B model also matches the backbone used by TokenSkip, allowing direct comparison under its published protocol. Experiments run on GSM-8k, MATH-500 which are slightly different to Section I align with TokenSkip's settings) and the out-of-domain GPQA benchmark (chemistry, biology and physics). The results shown in Table 3 Moving from B to B parameters raises CoLaR 's accuracy on every dataset while keeping the reasoning chain roughly half the length of the CoT teacher. Reinforcement learning delivers further improvements: +1.1% % on GSM-8k, +6.6% on MATH-500 and +5.1% 1% on GPQA, again with large additional compression. Most notably, CoLaR-SB- reaches 37.5% GPQA, surpassing its 35.7% CoT teacher and shortening the reasoning chain by 69%, confirming that latent RL remains effective models grow.
4.6 Interpreting chains of latent thoughts
To quantatively illustate the effectiveness of proposed CoLaR, and make the latent reasoning process more transparent, we conduct a case study of CoLaR on the GSMSk validation set. The results illlustrated in Figure 3
As latent variables are fundamentally scaled sum of word token embeddings, we could directly calculate the cosine simility between latent variables and the enitre LLM embedding matrix:
When prompted with compression factor == 2, CoLaR auto-regressively produces seven latent variables, and then automatically terminates reasoning process. By calculating similarity scores, each latent variable is capable of retrieving meaningful words such as ">"

---

"21", and the entire chain of latent thoughts could be interpretered as -=217=3><5+==15>> which exactly matches the correct calculation process.
When prompted with higher compression factor = 5. less informative tokens such as "*" are ignored, demonstrating both the effectiveness and efficiency of our proposed dynamic compression mechanism.
4.7 Analyses on dynamic compression factors
We investigate the generalization capability of CoLaR across different compression factors Two key findings emerge from our analyses:
First, illustrated in Figure for each test-time compression factor (except from 5). Co- LaR trained with random e E [1 5] consistently outperforms models trained on single compression factor. These results demonstrate that exposure to diverse training- time compression factors produces complementary benefits for generalization. For example, training with = 2 also improves the performance of testing with 4, highlighting the effectiveness of our dynamic training process.
Second, as shown in Figure 5 we train CoLaR with e E (1,3,5, and evaluate with previously unseen compression factors E {2, 4, 6j. We find that CoLaR successfully generalizes to these unseen compression factors, maintaining expected actual compression rates. Moreover, though worse in absolute values, the slope of the performance curve out-of-domain compression factors closely resembles that of in-domain factors, suggesting robust interpolation capabilities.
5 Limitations
While CoLaR demonstrates superior effectiveness and efficiency in latent reasoning, we acknowledge two important limitations: (1) Though CoLaR surpassing its CoT teacher on GPQA, on the remaining benchmarks, the overall performance of CoLaR currently approximates explicit CoT reasoning without surpassing it. (2) We observe that CoLaR struggles to generalize to non-integer compression factors ge. e 1.5) or values greater than the maximum training compression factor Ymaz- This limitation stems primarily from the discrete tokenization constraints inherent to large language models, which restrict the continuous representation of compression factors. (3) Beyond technical limitations, our work on enhancing reasoning capabilities in LLMs has significant societal implications. On the positive side, CoLaR could significantly boost the efficiency of existing LLM services. However, potential negative impacts include the risk of amplifying existing biases in reasoning processes and possible misuse for generating more convincing misinformation. To mitigate these risks, we recommend careful monitoring of downstream applications.
6 Conclusion
In this paper, we introduce Compressed Latent Reasoning (CoLaR), a framework that dynamically compresses LLM reasoning chains into latent space while maintaining capabilities. Our method centers on three key innovations: (1) compressed latent reasoning through an auxiliary next compressed embedding prediction task that encapsulates the semanties of multiple tokens, (2) dynamic training and inference with variable compression factors that allows for flexible reasoning chain lengths and fully parallelized processing, and (3) a probabilistic latent head for reinforcement learning that enables exploration of diverse reasoning pathways for higher accuracy while exploiting shorter reasoning chains for efficiency. Our experimental results demonstrate that CoLaR achieves 14.1% improvement in accuracy compared to state-of-the the-----tttt----tee latent-based reasoning methods, while reducing reasoning chain length by 53.3% with only a 4.8% performance degradation relative to explicit COT. On the challenging MATH dataset, reinforcement learning techniques further boost CoLaR's performance by 5.36% while dramatically reducing reasoning chain length by 82.8 92.8% Future work will focus on addressing non-integer compression factors, exploring more sophisticated reinforcement learning approaches, and extending our dynamic compression mechanism to more diverse reasoning tasks beyond mathematies.
Acknowledgements. This work is supported by the Beijing Natural Science Foundation (I.233008) and Xiaomi Inc. We acknowledge the anonymous reviewers for their helpful comments.
10

---

References
[1] Simon Aytes, Jinheon Baek, and Sung Ju Hwang. Sketch-of-thought Efficient IIm reasoning with adaptive cognitive-inspired sketching. arXiv preprint arXiv:2503.05173 2025.
[2] Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprint 2ee.22222...88. 2025.
[3] Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint urXiv:2412.13171, 2024.
[4] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14/68, 2021.
[5] Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.4482 2024.
[6] Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.01460, 2023.
[7] Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: A survey, arXiv preprint arXiv:2504. I0900, 2025.
[8] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01300 2025.
[9] Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764-10799. PMLR, 2023.
[10] Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. arXiv preprini arXiv:2310.02222 2023.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint 20xiv:240..21783 2024.
[12] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arxiv:2501.11944 2025.
[13] Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
[14] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
[15] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[16] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Kl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412. 2024.
[17] Zongzhao Li, Zongyang Ma, Mingze Li, Songyou Li, Yu Rong, Tingyang Xu, Ziqi Zhang, Deli Zhao and Wenbing Huang. Star-rl: Spacial transformation reasoning by reinforcing multimodal mms.. arXiv preprint arXiv:2505. 2025.

---

[18] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint drXiv:1711.05101, 2017.
[19] Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Cotformer: More tokens with attention make up for less depth. In Workshop Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurIPS 2023),
2023.
[20] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021.
[21] Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1743-1752, 2015.
[22] Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.14116 2025.
[23] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint xxXiv2242.033300 2024.
[24] Zhenyi Shen, Hanqi Yan, Linhai Zhang. Zhanghao Hu, Yali Du and Yulan He. Codi: Compressing -hain-of-thought into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025.
[25] DiJia Su, Hanlin Zhu, Yingchen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint arXiv:2502.03275. 2025.
[26] Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025.
[27] Wenhui Tan, Boyuan Li, Chuhao Jin, Wenbing Huang, Xiting Wang, and Ruihua Song. Think then react: Towards unconstrained cectiontt-rreaction motion generation. In The Thirteenth International Conference on Learning Representations, 2025.
[28] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1.5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
[29] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24831 2022.
[30] Heming Xia, Yongqi Li, Chak Tou Leong, Wenjie Wang, and Wenjie Li. Tokenskip: Controllable chain-of ofain-of-thhught compression in llms. arXiv preprint arXiv:2502.1200 12067, 2025.
[31] Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung. Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how think with meta chain-of-though. arXiv preprint vrXiv:2501.04682. 2025.
[32] Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint arXiv:2502.18600, 2025.
[33] Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot: Soft chain-of -hain-of-thought for efficient reasoning with lims. arXiv preprint arXiv:2502.12133 2025.
[34] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arX1v:2412.15115. 2024.
12

---

[35] Chenxu Yang, Qingyi Si, Yongje Duan, Zheliang Zhu, Chenyu Zhu, Zheng Lin, Li Cao, and Weiping Wang. Dynamic early exit reasoning models. arXiv preprint arXiv:2504.15895 2025.
[36] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025.
[37] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 115503..476.. 14476, 2025.
[38] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXivv2504.05//66 2025.
[39] Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D) Goodman. Quiet-star: Language models can teach themselves think before speaking. arXiv preprint arXiv:2403.0962,, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly articulate our main contribution of developing a novel approach for compressing explicit reasoning chains into latent space, accurately reflecting the scope of theoretical foundations and empirical results across the mathematical reasoning benchmarks evaluated in our experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: There is a separate "Limitations" section discussion the main limitations on performance compared explicit CoT methods, and worse generalization capability on certain compression factors.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper presents a framework for latent reasoning compression but does not include formal theorems with proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our paper provides comprehensive implementation details necessary for repro- ducing the proposed method. We include detailed descriptions of the model architecture, training procedure, hyperparameters, and evaluation methodology. Additionally, open source our complete codebase, including the environment setup, model implemen- tation, training pipeline, and evaluation framework, facilitate direct replication of our experimental results.
will
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We provide comprehensive implementation details to reproduce our method, but we have chosen not to include anonymous URLs our code repository at the submission stage, to balance scientific transparency with the appropriate protection of intellectual property. We make a strong commitment to release our complete codebase immediately upon acceptance of the paper, including model implementation, training procedures, evaluation framework, and necessary scripts to reproduce all experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guide http://dddeee/eeeuiiiissseeeeeee wwweeeummissiooooooee.eooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //iiss.ccppubll wwabblic/guiiess. httpa///gniddee//eeeuebbmmsseee wwwpeeeSSuumsssoooooo..ooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide comprehensive methodological details, hyperparameter specifica- tions, and experimental protocols for implementation in the manuscript and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
16

---

Answer: [Yes]
Justification: All experimental results in our paper are reported averages across five independent runs with different random seeds. We include 95% confidence intervals for all numerical results, calculated using the standard error of the mean multiplied by 1.96. These confidence intervals are visually represented in our figures as error bars and explicitly stated with + in our tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide complete information about the computational resources used in our experiments in the Appendix, including hardware setup, virtual environment confugura- tion, and total training hours.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpt:tttt:/nt http://errrr..cccuppueccccccccca
Answer: [Yes]
Justification: Our research fully complies with the NeurIPS Code Ethics through our transparent methodology, proper acknowledgment of prior work, and responsible data usage.
We have documented our approach thoroughly, and maintained scientific integrity throughout our work on enhancing reasoning capabilities in Large Language Models.
17

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses both positive societal impacts (improved reasoning capa- bilities that could enhance decision support systems and educational tools) and potential negative implications (risks of amplifying existing biases in reasoning processes and possible misuse for generating misleading content). We outline mitigation strategies including careful monitoring of downstream applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper is primarily analytical in nature, focusing on methods enhance and evaluate reasoning capabilities within existing models rather than releasing new models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
18

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The pre-trained L.Mss and datasets we use are fully open-sourced and correctly cited in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papprssii thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We conduct experiments using existing pre-trained LLMs and datasets, but do not create new assets as part of this publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Since LLMs form the core subject of our research, we have thoroughly documented all aspects of their usage in methodology section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy htappp///eeurpsscc//ooooeeeeee for what should or should not be described.
20

---

A More implementation details
In this section, we provide comprehensive details regarding our model architecture, training hyperpa- rameters, and dataset specifications.
Model hyperparameters. For our experiments, we employ either frozen LLama-3.2- 3.2-1BBIInttruc or SeepSeek-distill-O0 wwen-1.5B our LLM backbone, augmented with a tunable LoRA module. All LoRA modules are configured with a 32 and 128 consistently across all experiments. Our method incorporates Latent Head, implemented as a three-layered MLP with hidden dimensions corresponding to the LLM backbone's dimension (d = 2048).
Training hy perpparameters. We utilize the AdamW optimizer with weight decay of 1e-2 throughout our experiments. The learning rate is set at le-4 for supervised fine-tuning (SFT) and le-6 for reinforcement learning (RL). For SFT experiments, A100 GPUs with total batch size of 256. The RL experiments are conducted on single A100 GPU with rollout batch size of 8, optimizer step batch size of 4, group size G of and clip of 0.2. To ensure reproducibility, we fix the random seed of all libraries (Python, CUDA, Pyrorh and NumPy) for training processes. For evaluation, we use five distinct runs with random seeds sequentially set from 0 to 4.
leverage Distributed Data Parallel across eight
Notably, when training the Latent Head, normalize the target (i.e., the ground-truth compressed embeddings) to ensure training stability. This normalization is implemented by dividing the target by the standard deviation de of the embeddings. Since the embedding distributions are already centered at approximately zero (μ R 0), we do not apply any shift during normalization. During inference, we multiply the predicted embeddings by the standard deviation rescale them to match the LLM's original embedding distribution. These statistics can be either learned during training or calculated in advance; we opt for the latter approach for simplicity. We observe model-specific values, with oe R 0.02 for Llama-3.2 3.2- B-Intrruct and a, = 0.03 for Qwen-l I.5B. This normalization process critical for maintaining numerical stability while preserving the relative relationships between embedding dimensions.
Dataset information. We evaluate our method on four grade-school mathematics Aug, GSM8k-Hard, SVAMP, and MultiArith-as well as the more challenging MATH dataset for advanced mathematical reasoning. Since the original MATH dataset does not provide an official validation set, we randomly shuffle the training set and allocate 10% of the samples for validation purposes.
1.0
Layerwise nom difference
0.9
0.8
0.7
0.6
0
2
4
10
12
14
LLM layer index
Figure Layer-wise norm differences from CoLaR-1 to CoLaR-5.
21

---

B Layer-wise analyses on compression factors
We further investigate how the compression factor inffuences activation patterns across LLM layers. with results shown in Figure 6 Specifically, we tested CoLaR on the same sample as in Section 4.6 with compression factors ranging from to calculating the relative activation norm differences between consecutive LLM layers.
Our analysis reveals distinct patterns across different network depths:
Shallow layers (0-3, near input); CoLaR shows higher activation on smaller compression factors with more pronounced layer-wise changes in magnitude.
Intermediate layers (3-9); Models with different compression factors exhibit similar behavior.
Deeper layers (9-15, near output): Higher compression factors maintain stronger activation patterns.
This phenomenon can be explained as follows: when predicting less informative tokens (e.g.. "<") with lower compression factors (especially with = 1, which uses no compression), the model requires minimal "thinking" and can determine the next token using primarily shallow layers. Consequently, computation in deeper layers is largely underutilized.
In contrast, higher compression factors enable CoLaR process information more densely, with each latent representation carrying richer semantic content. This requires deeper layers to remain actively engaged in analyzing the condensed information and predicting subsequent compressed latents, thereby making more efficient use of the model's computational capacity. These findings align with observations from previous work on internal thinking processes transformer models 2 22
Val/ace
D
C2
vali.Jaten..oowwad
X
3.1
3.34
3.12
01
Figure 7: The validation accuracy and latent reasoning chain length curve MATH dataset.
C RL training curves
Figure 7 presents the training curves from our reinforcement learning phase. The accuracy on the validation set exhibits distinct three-phase pattern. In the initial exploration phase, accuracy increases rapidly from 9% to 14%, accompanied by an expansion of latent reasoning steps from 40 to 60. During this phase, the GRPO algorithm primarily encourages CoLaR to explore more extensively to discover correct reasoning pathways.
In the subsequent exploitation phase, validation accuracy fluctuates between 14% and 16%, while the latent reasoning length decreases from 60 to 20. With the per-token averaged reward/loss, the GRPO algorithm reinforces CoLaRR exploit shorter yet effective reasoning pathways.
Finally, as CoLaRR begins overfit, our early-stopping strategy is triggered to preserve the best- performing checkpoint at approximately 4k steps.
D Scaling properties of CoLaR
Figure iillustrates the performance characteristics of CoLaRR when implemented with foundation models of varying parameter counts, ranging from billion to billion parameters. Our results
22

---

70
Llama3-1B
LJama3-3B
60
Llama3-8B
50
40
30
Compecasion Factor (c)
Figure 8: Performance of CoLaR when implemented with base LLMs ranging from IB to 8B
parameters.
demonstrate that CoLaR follows established neural scaling laws, with performance improvements correlating predictably with increases in the underlying model size. This consistent scaling behavior suggests that the benefits of our approach extend proportionally across different model scales, indicating CoLaR's s architectural effectiveness is not limited to specific parameter regimes.
23


Reviews:
[{"review_id": "hNqoPNA793", "reviewer": "Reviewer_dukW", "summary": "This paper proposes Compressed Latent Reasoning (CoLaR), a framework for accelerating reasoning in large language models by compressing token-level CoTs into dense latent representations. CoLaR trains models to predict compressed latent embeddings and reconstruct tokens from the latent embeddings. It supports dynamic-speed reasoning by controlling a compression factor during inference, making it adaptable and efficient. The authors also introduce a reinforcement learning component to encourage exploration and exploitation of shorter latent chains via probabilistic latent prediction. Empirical results on several math reasoning benchmarks (e.g., GSM8K, MATH) are impressive: CoLaR outperforms previous latent-based methods by a significant margin, and maintains competitive performance to explicit CoT while drastically reducing chain length. While it doesn't yet surpass token-level CoT in absolute performance, its efficiency and scalability make it a compelling direction for future research, especially in resource-constrained or high-throughput scenarios.", "questions": "- What are the implementation details for iCoT and Coconut?\n- Is this method applicable on long reasoning models (like Deepseek-R1 or its distilled versions)? If it doesn’t bring immediate improvement, what are the challenges?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Unlike prior methods that rely on deterministic latent predictions, this work introduces a probabilistic latent head, enabling exploration in the latent reasoning space. This design is well-motivated and experimentally validated through reinforcement learning, demonstrating meaningful gains in both accuracy and reasoning efficiency.\n- The method doesn’t require autoregressive token generation during training, which enables fully parallelized supervision over the entire reasoning chain. This architectural choice improves training efficiency and scalability, making latent space reasoning more practical for large-scale applications.\n- The paper presents nice analysis on the trade-off between compression factor and performance, and the interpolation of compression factor.\n\nWeaknesses:\n\n- Several baselines, such as iCoT and Coconut, perform substantially worse than reported in their original papers on GSM8k, despite using a stronger backbone (Llama-3.2-1B-Instruct vs. GPT-2 Small). The paper does not provide sufficient implementation details or justification for these discrepancies, raising concerns about the fairness of the comparisons and whether the baselines were properly tuned. \n- Although the paper includes experiments with DeepSeek-R1-Distill-Qwen-1.5B, it only fine-tunes them on short CoT-style reasoning traces. This underutilizes the model's potential for generating long reasoning chains, which is precisely where latent reasoning techniques would be more useful and impactful. Including experiments on compressing longer reasoning chains would strengthen the paper’s practical relevance and better showcase the benefits of CoLaR in real-world settings.", "was_revised": true, "final_justification": "Thanks for clarifying the details of experiments. I think the paper delivered a sound method, and more studies need to be done to push the results on long CoT (thousands of tokens). I'll keep my positive score about this paper.", "rebuttal": "Dear Reviewer #dukW,\n\nWe thank you for highlighting both the merits of our probabilistic-latent design and the concerns regarding baseline fidelity and long-chain evaluation. We address each point below.\n\n**Q1: Baseline implementation details.** We strictly followed the original papers and release notes of iCoT and Coconut, applying identical base settings (frozen Llama-3.2-1B-Instruct, LoRA rank 128, batch size 256, learning rate 1e-4, max 50 epochs) and dataset splits; their custom hyper-parameters are documented in line 228 and will be expanded in the final revision.\n\n**W1: Lower baseline scores on a larger backbone.** The drop is expected: LoRA fine-tuning exposes only 13.6 M parameters on Llama-3.2-1B, whereas the GPT-2 baselines in prior work train all 117 M parameters at FP32 precision:\n\n| Model / Param / Trainable / Precision     |    GSM   | GSM-Hard |   SVAMP  | MultiArith |\n|-------------------------------------------|:--------:|:--------:|:--------:|:----------:|\n| Llama-3.2 / **1B** / 13.6M / BF16         |          |          |          |            |\n| CoLaR                                     |   26.8   | **5.87** | **48.4** |  **86.4**  |\n| Coconut                                   |   23.1   |   5.49   |   40.7   |    41.1    |\n| GPT-2 / 117M / **117M** / **FP32**        |          |          |          |            |\n| CoLaR                                     | **35.2** |   5.76   |   32.3   |    70.8    |\n| Coconut (Original)                        |   34.1   |     -    |     -    |      -     |\n\nThe table shows that when we replicate CoLaR on full-parameter GPT-2 its GSM accuracy rises from 26.8% to 35.2%, confirming that parameter count—not backbone capacity—dominates on these short chains. As anticipated, this gain sacrifices out-of-domain generalization (SVAMP and MultiArith drop sharply), so we kept the LoRA setting for fair comparison.\n\n**W2 & Q2: Applicability to long-reasoning models.** Thank you for highlighting the gap in long-chain settings. We agree that compressing truly long reasoning traces is where CoLaR can deliver the most value, and we have already taken the first step in that direction: on DeepSeek-R1-Distill-Qwen-1.5B, we trained and evaluated CoLaR on the MATH dataset whose average reasoning chain is \\~200 tokens—ten times longer than GSM’s. As shown in the paper (Table 2), CoLaR still achieves an 82.8% reduction in chain length with a 5.36% accuracy gain after RL, demonstrating that the method scales gracefully to longer contexts.\n\nWhat we have not yet attempted is compressing chains of several thousand tokens, such as those generated by the full DeepSeek-R1 671 B model. The bottleneck is purely computational: training on such long sequences would require either (1) multi-node GPU clusters to fit the enlarged teacher traces, or (2) an additional “latent pre-training” phase where we first distill very long CoT paths into compressed embeddings offline, then fine-tune the latent head. Both are in our immediate roadmap.\n\nWe are optimistic about scaling the latent reasoning length: because CoLaR’s next-compressed-embedding objective is structurally identical to next-token prediction—the paradigm that has already scaled to million-token contexts.\n\nWe sincerely appreciate your meticulous feedback and remain available for any further clarifications.\n\nAuthors of CoLaR #12942"}, {"review_id": "HfoTLnxeto", "reviewer": "Reviewer_VtEU", "summary": "This paper introduces CoLaR (Compressed Latent Reasoning), a novel framework that compresses LLM reasoning chains into a latent space while aiming to preserve the model’s exploration-exploitation capabilities. Key features of the approach include: (1) a dynamic compression factor that enables flexible reasoning speed at inference time, (2) an embedding compression module that uses scaling rather than simple averaging, (3) a probabilistic latent head to predict the mean/variance of the next embedding, and (4) a two-stage training process combining supervised fine-tuning with auxiliary compressed embedding prediction and reinforcement learning via GRPO. Experimental results show a 14.1% accuracy improvement over latent baseline methods and a 53.3% reduction in reasoning chain length, with only a 4.8% drop in performance compared to explicit CoT methods.", "questions": "- How do efficiency and accuracy characteristics change when using larger foundation models (e.g., 7B, 13B, 70B+)? Do the compression benefits scale proportionally?\n- Have you evaluated the approach on non-mathematical reasoning tasks to assess its broader applicability?", "limitations": "See the weaknesses and questions.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The dynamic compression factor and probabilistic latent head offer a meaningful advance over previous work like Coconut and CODI, and show promise for reinforcement learning applications.\n- The paper provides thorough experiments across multiple math reasoning datasets, with proper statistical reporting (95% confidence intervals over five runs) and relevant baseline comparisons.\n- The GRPO-based reinforcement learning approach delivers substantial gains on challenging tasks (e.g., a 5.36% accuracy boost and 82.8% reduction in chain length on the MATH dataset), highlighting the value of exploration-exploitation in latent reasoning.\n    \n\n**Weaknesses**\n- The method still falls short of explicit CoT reasoning in terms of performance, which significantly limits its practical impact. For real-world applications where users need maximum accuracy, the trade-off between efficiency and accuracy may not be compelling.\n- Most experiments use 1B-parameter models. Modern reasoning tasks often require larger models, where efficiency gains are most valuable. The scalability of the approach to these larger models remains unproven.\n- Latent representations sacrifice the human-readable reasoning chains that make CoT valuable for debugging and verification—an important limitation, especially in high-stakes applications.\n- The paper lacks a deep investigation into when and why latent reasoning fails compared to explicit reasoning, which limits understanding of the method’s boundaries.", "was_revised": true, "final_justification": "The author has solved my problem, and I keep the positive score.", "rebuttal": "Dear Reviewer #VtEU,\n\nWe are grateful for your recognition of our dynamic-compression design, rigorous evaluation, and the documented RL-driven gains. Below we address the four remaining weaknesses that you precisely articulated.\n\n**W1: Efficiency vs. accuracy in practice.** While CoLaR currently lags CoT on raw accuracy, real-world deployments already trade a few percentage points for 2–10× cheaper inference (think Gemini-Flash vs. Gemini-Pro). CoLaR’s carbon footprint drops proportionally with compression factor, making the trade-off attractive for latency-sensitive or eco-conscious applications.\n\n**W2 & Q1-Q2: Scaling to larger models & Non-math datasets.** Yes, CoLaR is capable to scale up to large foundation models, and generalize to non-mathematical reasoning domains.\n\nAppendix E (Figure 8) already shows that CoLaR on GSM8k climbs from 26\\~40% (1B) to 43\\~61% (8B) while retaining comparable compression ratios.\n\nTo extend this evidence, we ran CoLaR-2 on Llama-3.2-1B and Llama-3.1-8B across math and GPQA dataset, which consists high-quality difficulty question on physics, chemistry, and biology: (The reported results of GSM and MATH are different from Table 1 of our manuscript. This is because we are using a slight different version of datasets to align with another baseline method TokenSkip.)\n\n|       |   CoT-1B  |  CoLaR-1B |   |   CoT-8B  |  CoLaR-8B | CoLaR-8B-RL | TokenSkip-8B |\n|-------|:---------:|:---------:|---|:---------:|:---------:|:-----------:|:------------:|\n| GSM8k | 47.5/92.6 | 40.4/49.2 |   | 76.5/93.6 | 70.8/47.3 |  71.9/13.3  |   78.2/113   |\n| MATH  |  28.6/176 | 19.8/84.6 |   |  54.6/168 | 45.8/67.7 |  52.4/17.6  |   40.2/292   |\n| GPQA  |  26.4/232 | 26.4/84.1 |   |  35.7/216 |  32.4/101 |  37.5/66.7  |       -      |\n\nThree patterns emerge:\n- First, bigger backbones improve absolute accuracy for both CoT and CoLaR, yet CoLaR’s compression ratio remains stable.\n- Second, RL post-training raises CoLaR-8B above CoT on GPQA (37.5 vs 35.7) while shrinking reasoning chains by 70%.\n- Third, the gains generalize beyond mathematics, confirming that our paradigm scales and broadens with model size. \n\n**W3: Explainability.** We thank the reviewer for this concern. Appendix B, Figure 5 demonstrates that CoLaR’s latent variables—compressed token embeddings—can be faithfully decoded: a simple cosine-similarity search against the token embedding matrix reliably recovers human-readable fragments such as “21 / 7 = 3 × 5 = 15.”\n\nMore broadly, recent studies [1] reveal that explicit CoT steps can diverge from the model’s actual decision process or harbor silent biases; verbalization is therefore not a guarantee of transparency. Retrieving embeddings from CoLaR offers the same level of interpretability as CoT, while being more concise.\n\nFinally, we argue that human thought, or part of human thought, resides in a latent space. Russian distinguishes голубой (“sky-blue”) from синий (“ocean-blue”)—concepts with no exact English counterpart—yet speakers share the same perceptual embedding in their minds. An old adage captures this: “It can be grasped, yet cannot be spoken.” Echoing Wittgenstein, we believe much of human reasoning unfolds in a non-linguistic, latent form; CoLaR merely gives us direct access to that space and lets us translate it back into words when necessary.\n\n[1] Barez, Fazl, et al. \"Chain-of-thought is not explainability.\" Preprint, alphaXiv (2025): v2.\n\n**W4: Closing the accuracy gap.** Thank you for this insightful question. CoLaR’s SFT stage only distills existing CoT paths, so its ceiling is inevitably “CoT-minus-distillation-loss.” Reinforcement learning, however, lets the model discover shorter latent paths that can outperform the teacher: CoLaR-8B-RL already beats CoT-8B on GPQA and narrows the gap on MATH (52.4 vs 54.6). We expect continued RL at larger scales to erase or reverse the deficit, just as R1 surpassed its CoT teacher. We will add more detailed discussion in Section 4.2 of our final revision.\n\nThank you for your constructive observations; we would be glad to address any additional concerns you may have.\n\nAuthors of CoLaR #12942"}, {"review_id": "PAQrOSHawy", "reviewer": "Reviewer_nEhT", "summary": "This paper proposes a learnable framework for semantic compression in latent space, aiming to reduce the length of reasoning chains. Although the performance slightly drops in certain cases, the method significantly shortens the Chain-of-Thought (CoT) reasoning steps.", "questions": "The paper emphasizes that the method reduces reasoning length. However, during inference, how does the model determine when to stop generating latent steps? Does it rely on checking, at every step, whether the Language Head produces a specific termination token? If so, wouldn’t this introduce additional computational overhead?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a relatively novel approach for compressing reasoning context in latent space.\n2. The integration of GRPO into the compression framework is elegant and well-motivated.\n\nWeaknesses:\n1. The compression mechanism relies on a simple averaging operation. When facing out-of-distribution tokens, this might pose a challenge for the model’s generalization ability.\n2. The termination mechanism for latent generation is not clearly explained—e.g., whether it is based on detecting a special token from the Language Head, fixed length threshold, or another signal.\n3. The experiments are conducted on relatively small backbone models. Additional results on larger models would strengthen the claims and validate scalability.", "was_revised": true, "final_justification": "The main body of the paper employs an averaging-like operation within a mere 1B model to enable the model to learn the ability to predict merge tokens. During inference, it only needs to continuously predict a compressed token, which significantly reduces the inference length. This idea is very novel; however, the authors have not provided sufficient implementation details of this method in the paper.\n\nI had some misunderstandings about the termination mechanism described in the paper, which have been clarified by the authors.\n\nAs all other reviewers have also pointed out, the original paper only used a 1B model. The authors later supplemented their rebuttal with 8B experiments. Still, I believe that drawing the paper’s conclusions at this scale and accuracy level is insufficient. It’s a pity that the authors did not provide experiments on slightly larger models.", "rebuttal": "Dear Reviewer #nEhT,\n\nWe sincerely appreciate your recognition on our latent reasoning framework along with RL pipeline, and highlighting the compression robustness, termination clarity, and scalability concerns. We address each point below.\n\n**W1: Generalization of the averaging operation.** Although the compression module is intentionally lightweight, its robustness is inherited from the backbone LLM.\n\nIn CoLaR, every latent embedding is merely a scaled linear combination of in-vocabulary token embeddings. Because the LLM was trained on massive, diverse corpora, the probability mass over tokens is already smooth; extreme outliers that would make the average OOD are therefore rare. Empirically, we observed no stability issues across three OOD datasets (GSM-hard, SVAMP, and MultiArith), indicating that the exact form of the compression is not a brittle design choice.\n\n**W2 & Q1: Termination mechanism and overhead.** CoLaR decides when to stop via the Language Head, not a fixed length. \n\nAt every latent step we feed the final hidden state into the Language Head and sample the next token; if the token is \\<end-of-thinking\\>, we switch to answer generation. This is identical to how any autoregressive LLM ends a turn, so the extra compute is a single linear layer (hidden-size × vocab-size)  forward pass. \n\nTo quantify the cost, we timed 100 latent reasoning steps on GSM at batch-size 32 on one A100 GPU: LLM layers took 10.8 s, while the Language-Head check added only 0.6 s (5% of total). Replacing the full Language Head with a Tiny Linear layer (by extracting the \\<end-of-thinking\\> token column of the Language Head) keeps accuracy at 26.7% and cuts the check to 0.3 s—noise compared with the 3–10× latency savings we already obtain via shorter chains.\n\n|               | LLM Layers | Language Head | Tiny Linear (90\\% threshold) |\n|---------------|:----------:|:-------------:|:----------------------:|\n| Time Cost (s) |    10.8    |      0.6      |           0.3          |\n| Acc. (%)      |      -     |      26.8     |          26.7          |\n\n**W3: Scaling-up and domain generalization.** Yes, CoLaR is capable to scale up to large foundation models, and generalize to non-mathematical reasoning domains.\n\nAppendix E (Figure 8) already shows that CoLaR on GSM8k climbs from 26\\~40% (1B) to 43\\~61% (8B) while retaining comparable compression ratios.\n\nTo extend this evidence, we ran CoLaR-2 on Llama-3.2-1B and Llama-3.1-8B across math and GPQA dataset, which consists high-quality difficulty question on  physics, chemistry, and biology: (The reported results of GSM and MATH are different from Table 1 of our manuscript. This is because we are using a different version of datasets and prompts to align with another baseline method)\n\n|       |   CoT-1B  |  CoLaR-1B |   |   CoT-8B  |  CoLaR-8B | CoLaR-8B-RL | TokenSkip-8B |\n|-------|:---------:|:---------:|---|:---------:|:---------:|:-----------:|:------------:|\n| GSM8k | 47.5/92.6 | 40.4/49.2 |   | 76.5/93.6 | 70.8/47.3 |  71.9/13.3  |   78.2/113   |\n| MATH  |  28.6/176 | 19.8/84.6 |   |  54.6/168 | 45.8/67.7 |  52.4/17.6  |   40.2/292   |\n| GPQA  |  26.4/232 | 26.4/84.1 |   |  35.7/216 |  32.4/101 |  37.5/66.7  |       -      |\n\nThree patterns emerge:\n- First, bigger backbones improve absolute accuracy for both CoT and CoLaR, yet CoLaR’s compression ratio remains stable.\n- Second, RL post-training raises CoLaR-8B above CoT on GPQA (37.5 vs 35.7) while shrinking reasoning chains by 70%.\n- Third, the gains generalize beyond mathematics, confirming that our paradigm scales and broadens with model size. \n\nWe thank you for your thorough review and look forward to any further questions or suggestions that could strengthen the paper.\n\nAuthors of CoLaR #12942"}, {"review_id": "nGR9bn2013", "reviewer": "Reviewer_MsBS", "summary": "The paper introduces Compressed Latent Reasoning (CoLaR), a novel framework for enhancing the efficiency of reasoning in LLMs by dynamically compressing reasoning chains into latent space. The approach leverages a two-stage training process: SFT with an auxiliary objective for predicting compressed embeddings, followed by RL to explore diverse reasoning paths. The results demonstrate significant improvements in accuracy and substantial reductions in reasoning chain length across various mathematical reasoning datasets.", "questions": "Please refer to \"Weaknesses\"", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. CoLaR introduces a novel method for compressing reasoning chains, allowing for faster and more efficient inference.\n2. The ability to adjust compression factors at inference time demonstrates flexibility and adaptability\n3. Extensive evaluations across multiple datasets demonstrate consistent performance gains.\n\nWeaknesses\n1. The evaluation primarily focuses on math problem solving  tasks, lacking assessments on more complex challenges.  Table 1 includes two version of GSM8k, which may confuse readers as to why two versions of the same dataset were chosen over other diverse datasets.\n2. Testing is only conducted on small models, including 1B and 1.5B model. CoT reasoning typically benefits more from larger models, which may limit the applicability of findings.\n3. The paper does not compare CoLaR with other concurrent reasoning compression methods, such as Tokenskip [1], LightThinker [2] or similar approaches that also optimize reasoning efficiency.\n4. The selection of GRPO over other reinforcement learning algorithms is not justified, leaving questions about this choice.\n5. CoLaR struggles to generalize to non-integer compression factors (e.g.,c=1.5) due to discrete tokenization constraints, which restricts its flexibility. Besides, CoLaR’s intermediate reasoning steps, compressed into latent space, are difficult for humans to interpret, as they lack the explicit, token-level reasoning provided by traditional CoT methods. This may opacity reduces the explainability of the model’s reasoning process.\n\n\n\n\nReference \n\n[1] Xia, Heming, et al. \"Tokenskip: Controllable chain-of-thought compression in llms.\" arXiv preprint arXiv:2502.12067 (2025).  \n[2] Zhang, Jintian, et al. \"Lightthinker: Thinking step-by-step compression.\" arXiv preprint arXiv:2502.15589 (2025).", "was_revised": true, "final_justification": "I appreciate the authors' reply. As most of my concerns have been addressed, I have adjusted my rating accordingly.", "rebuttal": "Dear Reviewer #MsBS,\n\nThank you for acknowledging CoLaR’s novelty and empirical breadth, and for enumerating the remaining gaps in domain coverage, scale, dataset, comparative baselines, algorithmic justification, and explainability. We address each concern below.\n\n**W1.1: Clarification of GSM8k.** Thank you for pointing this out. There are three versions of GSM8k: 1) GSM8k, the original one, where there are 8k samples with natural language reasoning chains, and 2) GSM8k-Aug, the one we used for training, which has extended training samples (\\~385k) and condensed reasoning chains, and 3) GSM8k-Hard, the one we used for OOD testing, where the numbers are in larger range and harder to compute. The test set of GSM8k and GSM8k-Aug are the same. We will clarify this in our final revision.\n\n**W1.2 & W2 & W3: Scaling-up, new baseline, and non-math reasoning tasks.** Yes, CoLaR is capable to scale up to large foundation models, generalize to non-mathematical reasoning domains, and CoLaR surpasses TokenSkip on MATH dataset with fewer reasoning steps.\n\nAppendix E (Figure 8) already shows that CoLaR on GSM8k climbs from 26\\~40% (1B) to 43\\~61% (8B) while retaining comparable compression ratios.\n\nTo extend this evidence, we ran CoLaR-2 on Llama-3.2-1B and Llama-3.1-8B across math and GPQA dataset, which consists high-quality difficulty question on physics, chemistry, and biology: \n\n(The reported results of GSM and MATH are slightly different from Table 1 of our manuscript. This is because we are using a different version datasets, GSM8k instead of GSM8k-Aug, to align with TokenSkip. We are unable to directly compare our CoLaR with LightThinker, which mainly focuses on replacing already-generated cot tokens with dense representations to reduce memory peak.)\n\n|       |   CoT-1B  |  CoLaR-1B |   |   CoT-8B  |  CoLaR-8B | CoLaR-8B-RL | TokenSkip-8B |\n|-------|:---------:|:---------:|---|:---------:|:---------:|:-----------:|:------------:|\n| GSM8k | 47.5/92.6 | 40.4/49.2 |   | 76.5/93.6 | 70.8/47.3 |  71.9/13.3  |   78.2/113   |\n| MATH  |  28.6/176 | 19.8/84.6 |   |  54.6/168 | 45.8/67.7 |  52.4/17.6  |   40.2/292   |\n| GPQA  |  26.4/232 | 26.4/84.1 |   |  35.7/216 |  32.4/101 |  37.5/66.7  |       -      |\n\nFour patterns emerge:\n- First, bigger backbones improve absolute accuracy for both CoT and CoLaR, yet CoLaR’s compression ratio remains stable.\n- Second, RL post-training raises CoLaR-8B above CoT on GPQA (37.5 vs 35.7) while shrinking reasoning chains by 70%.\n- Third, the gains generalize beyond mathematics, confirming that our paradigm scales and broadens with model size. \n- Finally, CoLaR-8B surpasses TokenSkip by a large margin on SFT and RL version with fewer reasoning steps on MATH dataset.\n\n**W4: Reasons to use GRPO.** Thank you for this question. GRPO is selected because the task is a canonical exploration–exploitation dilemma: for each question a combinatoric space of latent reasoning trajectories exists, yet only the subset that yields the correct answer is admissible.\n\nGRPO instantiates this balance explicitly: it first explores by sampling a group of complete latent chains, then exploits by reinforcing those that are both correct and minimal in length.\n\nMoreover, in contrast to PPO, GRPO dispenses with a learned value function and instead relies on the verifiable reward alone, reducing variance and computational overhead while preserving principled policy optimization. \n\nWe will add relevant discussion to Section 3.3 in our final revision.\n\n**W5.1: Generalization to non-integer compression factors.** As acknowledged in the Limitations Section, CoLaR’s reliance on discrete prompting precludes fine-grained, non-integer compression ratios.\n\nWe propose to address this limitation via a stochastic training strategy that blurs compression factor boundaries: at each forward pass, we randomly merge either one or two consecutive tokens, thereby inducing an effective compression factor that fluctuates around 1.5.\n\nThis strategy approximates continuous control while remaining fully compatible with the model’s discrete tokenization, and we will empirically validate its efficacy in our future work.\n\n**W5.2: Explainability.** We thank the reviewer for this concern. Appendix B, Figure 5 demonstrates that CoLaR’s latent variables—compressed token embeddings—can be faithfully decoded: a simple cosine-similarity search against the token embedding matrix reliably recovers human-readable fragments such as “21 / 7 = 3 × 5 = 15.”\n\nMore broadly, recent studies [1] reveal that explicit CoT steps can diverge from the model’s actual decision process or harbor silent biases; verbalization is therefore not a guarantee of transparency. Retrieving embeddings from CoLaR offers the same level of interpretability as CoT, while being more concise.\n\nFinally, we argue that human thought itself resides in a latent space. Russian distinguishes голубой (“sky-blue”) from синий (“ocean-blue”)—concepts with no exact English counterpart—yet speakers share the same perceptual embedding in their minds. An old adage captures this: “It can be grasped, yet cannot be spoken.” Echoing Wittgenstein, we believe much of human reasoning unfolds in a non-linguistic, latent form; CoLaR merely gives us direct access to that space and lets us translate it back into words when necessary.\n\n[1] Barez, Fazl, et al. \"Chain-of-thought is not explainability.\" Preprint, alphaXiv (2025): v2.\n\nYour insightful remarks have been invaluable to the refinement of our work; we stand ready to address any additional concerns you may have.\n\nAuthors of CoLaR #12942"}]

Author rebuttal:

Meta-review: The paper proposes Compressed Latent Reasoning (CoLaR) that trains a model to do latent reasoning by teaching it to predict compressed CoT embeddings. The model is trained in two stages: SFT with next token and compressed embedding prediction losses for varying compression factors, and then with RL to further improve performance and thinking efficiency. Overall the CoLaR outperforms previous latent CoT methods and also provides meaningful reduction in the reasoning chain lengths, particularly after RL.

---

All reviewers were positive about the paper and appreciated the novelty of the CoLaR framework and the ideas therein, the exploration-exploitation design of the method, and the evaluation methodology.

There were some concerns about scale of experimentation, diversity of evals, discrepancy in baseline numbers and interpretability issues with latent embeddings. The authors addressed them by

- Discussing the LoRA vs full FT setup difference

- Llama 8b experiments, GPQA eval, include TokenSkip baseline

- Appendix shows that latent embeddings can be decoded for interpretability

—

Overall the paper proposes an interesting and non-trivial idea that seems more effective than previous latent reasoning approaches. The ideas of token averaging for CoT compression, next embedding prediction and probabilistic sampling of latents are interesting. The sustained gains after RL further strengthens the findings. One weakness is that the results only hold relatively short CoT with ~200 tokens, but this limitation is not restricted to this paper. Overall the paper makes a solid contribution and the recommendation is accept.
Decision: Accept (poster)