SIM-CoT: Supervised Implicit Chain-of-Thought
Authors: Xilin Wei
Xiaoran Liu
Yuhang Zang
Xiaoyi Dong
Yuhang Cao
Jiaqi Wang
Xipeng Qiu
Dahua Lin
Venue: ICLR 2026 Poster (2026)
Keywords: Chain-of-Thought
large language model
math reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Implicit Chain-of-Thought (CoT) methods offer a token-efficient alternative to explicit CoT reasoning in Large Language Models (LLMs), but a persistent performance gap has limited their adoption.
We identify a core latent instability issue when scaling the computational budget of implicit CoT: as the number of reasoning tokens increases, training often becomes unstable and collapses.
Our analysis shows that this instability arises from latent representations becoming homogeneous and losing semantic diversity, caused by insufficient step-level supervision in current implicit CoT methods.
To address this, we propose SIM-CoT, a plug-and-play training module that introduces step-level supervision to stabilize and enrich the latent reasoning space.
SIM-CoT employs an auxiliary decoder during training to align each implicit token with its corresponding explicit reasoning step, ensuring latent states capture distinct and meaningful information.
The auxiliary decoder is removed at inference, preserving the efficiency of implicit CoT with no added overhead.
It also provides interpretability by projecting each latent token onto an explicit reasoning vocabulary, enabling per-step visualization and diagnosis.
SIM-CoT significantly improves both in-domain accuracy and out-of-domain stability of implicit CoT methods, boosting Coconut by +8.2\% on GPT-2 and CODI by +3.0\% on LLaMA-3.1 8B.
It further surpasses the explicit CoT baseline on GPT-2 by 2.1\% with 2.3$\times$ greater token efficiency, while closing the performance gap on larger models like LLaMA-3.1 8B.

Published as a conference paper at ICLR 2026
SIM-COT: THOUGHT
SUPERVISED
IMPLICIT
CHAIN-OF-
Xilin .e Xiaoran Liu Yuhang Zang" 25 Xiaoyi Dong?
Yuhang Cao?, Jiaqi Wang2.4a; Xipeng Qiu 1.4, Dahua Lin 2.3
Fudan University, Shanghai Al Laboratory,
The Chinese University of Hong Kong, Shanghai Innovation Institute
Tteemffuce fudan edu. en, zangyuhang@pjlab. org. rrg.c.
Github: http:://githubb..mmmmntcttttttt
ABSTRACT
Implicit Chain-of-Thought (CoT) methods offer token-efficient alternative explicit CoT reasoning in Large Language Models (LLMs), but a persistent perfor- mance gap has limited their adoption. We identify core latent instability issue when scaling the computational budget of implicit CoT:: as the number reasoning tokens increases, training often becomes unstable and collapses. Our analysis shows that this instability arises from latent representations becoming homoge- neous and losing semantic diversity, caused by insufficient step-level supervision current implicit CoT methods. To address this, we propose SIM-CoT, plug-and- play training module that introduces step-level supervision to stabilize and enrich the latent reasoning space. SIM-CoT employs auxiliary decoder during training to align each implicit token with its corresponding explicit reasoning step, ensuring latent states capture distinct and meaningful information. The auxiliary decoder is removed at inference, preserving the efficiency implicit CoT with no added overhead. It also provides interpretability by projecting each latent token onto explicit reasoning vocabulary, enabling per-step visualization and diagnosis. SIM- CoT significantly improves both in-domain accuracy and out-of-domain stability of implicit CoT methods, boosting Coconut by +8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. further surpasses the explicit CoT baseline on GPT-2 by 2.1% with 2.3x x greater token efficiency, while closing the performance gap on larger models like L.LMMA3.1 8B. Code: https://gituu..com/ M/SIMMCCTT
INTRODUCTION
"Measure what is measurable, and make measurable what is not so."
Galileo Galilei
The strong reasoning capabilities of Large Language Models (LLMs) (OpenAl 2024 Google 2024
Anthropic 2024) are often unlocked through explicit Chain-of-Thought (CoT) prompting (Wei et al. 2022). The explicit CoT approach enables L.MS to solve complex problems in step-by-step manner, yielding high performance in domains like mathematics and programming (Guo et al. 2025 Muennighoff et al. 2025). Despite its advantages, explicit CoT also faces several limitations. For instance, explicit CoT approaches must verbalize intermediate thoughts from fixed vocabulary, thereby precluding the exploration of alternative solution paths (Li et al. 2025 Zhang et al. 2025). Additionally, the generation of extensive intermediate sequences significantly increases inference cost and can result in redundant over-thinking steps or unnecessary verbosity (Chen et al. 2024
To address the flexibility and efficiency issues of explicit CoT methods, recent implicit CoT ap- proaches (Hao et al.] 2025 Zhang et al. 2025 Li et al., 2025) have been proposed by representing reasoning in a continuous latent space rather than as a sequence of discrete text tokens. The implicit CoT methods allow each latent representation to encode richer information than single explicit token, often with a significantly smaller number of latents than the length of an explicit reasoning chain. Early representative implicit work like Coconut (Hao et al. 2025) improves efficiency while still capturing useful intermediate structure. More recent approaches, such as CODI (Shen et al. 2025), further apply trajectory-Ievel distillation from explicit reasoning paths to enhance performance.

---

Published as a conference paper at ICLR 2026
100
Iatent
latent
91.5
89.9
86.6
83.0
80
Accuracy (%)
latent
Iatent
69.5
Number
25
multi-stage training curriculum
51.4
Operator
40
38.3
41.0
Amswer
20
progressively expanding implicit tatents
33.3
27.4
25.9
27.8
performance (12.5%)
Scaling implicir tokens couses Iatent instability isses.
20
5
20
latent
2 Iltent
latent
latent
Failure
Epoch
Latent tokens
(5lteent)
(a) The latent instability issue
(b) Information Loss
40
Janet'
ducks lay 16 eggs per day ...How much in dollars does she make every day at the farmers'
35
25
Vocab Distance
30
I. Latent distance nardows
Latent Distance
market? steps: 9416-3-4-990,
#illg::
25
Iatent distance
20
Normal Implicit Model:
20
Intent 2
15
latent 1: ["3", "4",, "15", "2", 1s", "16", "20", '9"]
latent 2: [*_" *an "4* n-n "y", "3". "-("] (Largor latent distinoo, richer content.)
15
vocab
Intent
distance
Failed Implicit Model:
latent 1: ["16", "3", "WA"". "2", "4", "8", "18", "24"] lalent ["3", "16", "30", "2" "6", "12", "18", "24"]
vocab center
Intent
latent
latent
5 latent
Fallure
(Smallee latent distance, mere uniform.)
(c) Shifted Distance
latent
(d) Semantic Homogenization
Figure (a) The latent instability issue: while using more implicit tokens initially improves
accuracy, training becomes unstable and sometimes collapses. (b) Information Loss: the implicit
tokens of failed models (5 latent tokens) lose crucial information about operators (like +. -). which makes complex reasoning impossible. (c) Shifted Distance: the latent-to-latent distance of failed models shrinks and becomes too similar to each other, while the latent drifts away from the central vocabulary embedding space. (d) Semantic Homogenization: failed models produce similar latent representations, resulting in narrower range decoded tokens, mostly numbers, as opposed to the more varied content generated by a normal model.
Despite these advancements, performance gap still exists between existing implicit CoT methods and their explicit counterparts. The implicit CoT approaches are fast, token-efficient but less accurate, which currently limits their broader application.
To narrow the performance gap, inspired by the success of explicit CoT that scales computational budget for better performance, we explore a similar strategy for implicit CoT methods by increasing the number implicit tokens. However, Fig. (a), we reveal one underlying latent instability issue in current implicit CoT approaches. As we extend the number of implicit tokens from the default three (Hao et al. 2025) to five, the training process initially improves accuracy but becomes unstable and sometimes collapses entirely. To interpret the latent instability issue, analyze implicit tokens from models trained on math reasoning data GSM8K-Aus (Deng et al. (022). We follow previous works (Hao et 20255] Deng al. 2024) project the implicit tokens through the LM head and examine their top decoded tokens for analysis. As shown in Fig. (b), failedd models tend to collapse into homogeneous latent states, While successful reasoning requires capturing both numerical and operator information, the implicit tokens of failed models primarily represent numbers, almost completely losing the critical operator information. Fig. (c) further demonstrates that model's 's collapse is accompanied by two changes: reduction in the inter-latent distance and drift of the latent states away from the central vocabulary embedding space. The latent representations of failed models become too similar and lose their semantic connection to the tokens they meant to represent. Fig. [ (d) provides an example of the semantic homogenization. A normal model (top) maintains large distance between its two latent tokens, allowing them capture distinct information for numbers and operators. In contrast, a failed model (bottom)'s 's latent tokens become homogeneous, with both states decoding to similar information, primarily numbers.
Our observation (Fig. M reveals the reasons for the latent instability issue: lack of sufficient step- level supervision for existing implicit methods to maintain the rich and varied internal representations.
2

---

Published as a conference paper at ICLR 2026
Without stronger guidance, the latent space collapses, losing its diversity and making it impossible to reliably encode the distinct, step-level reasoning needed for complex reasoning tasks. Motivated by our findings, propose Supervised IMplicit- (SIM-CoT). plug-and and-play module that introduces step-level supervision for implicit CoT approaches to alleviate the latent instability issue. Instead of supervising only the final answer (Hao et al., 2025) the trajectory (Shen al. (0225). SIM-CoT uses an auxiliary decoder to align each implicit token with its corresponding explicit reasoning step during training. The step-level supervision for implicit tokens stabilizes optimization, prevents collapse, and ensures that latent tokens capture meaningful reasoning content. Crucially, because the auxiliary decoder is removed during inference, our approach incurs virtually no extra computational cost, making it as efficient as standard implicit CoT approaches. Beyond accuracy, stability, and efficiency, the auxiliary decoder also affords interpretability of implicit reasoning. During training, it defines projection from latent tokens to the explicit reasoning vocabulary, enabling us to decode each latent step into a human-interpretae interpraatteeee summary for verification or crror diagnosis.
Experiments show that SIM-CoT acts as a plug-and-play module that boosts both accuracy and stability. We show that SIM-CoT can be effortlessly combined with various implicit CoT approaches such as Coconut (Hao et al. 2025). CODI (Shen et al., 2025), and training- free approaches (Zhang ett al. 2025) to further enhance reasoning performance. On GPT-2, SIM-CoT surpasses both the strong explicit baseline (supervised fine-tuning on explicit CoT data) by 2.1%, 1%, and outperforms existing implicit methods Coconut and CODI by 8.2% and 4.3%, respectively. The performance trend holds as the method scales to larger models such as the LLaMA series. SIM-CoT achieves improvements over CODI of 3.4% (LLaMA-3.2 IB), 1.5% (LLaMA-3.2 3B), and 3.0% (LLaMA-3.1 8B), in addition to 9.0% gain over Coconut on the LLaMA-3.2 1B model. Furthermore, while previous implicit CoT approaches (e.g., Coconut) collapse when scaled to 8 or 16 implicit tokens, SIM-CoT remains stable and continues to boost performance.
In summary, our contributions are as follows: bility issue of implicit CoT approaches, showing that instability and collapse arise from insufficient supervision. 2) We introduce SIM-CoT, which applies step-level supervision to the model's s implicit tokens. SIM-CoT not only integrates seamlessly with existing implicit CoT approaches and boosts performance with minimal inference overhead, but also affords interpretability of implicit reasoning by projecting each latent token onto explicit reasoning vocabulary, enabling per-step visualization of semantic roles and diagnosis. 3) Through extensive experiments, we demonstrate that SIM- CoT not only improves accuracy in the in-domain dataset, but also generalizes effectively to out-of-domain datasets. The performance gains are consistent across range of LLMs,, including GPT-2 and recent LLaMA models (IB, 3B, and 8B).
1) We provide a systematic analysis of the latent insta-
2 ANALYSIS OF IMPLICIT COT: THE LATENT INSTABILITY ISSUE
We first present an analysis (Fig. D of the limitations in implicit latent CoT approaches. We follow Coconut (Hao al. (0255 and analyze implicit latents by projecting them through the LM head and examining the top-8 decoded tokens to understand the semantic and geometric properties.
Latent Instability Issue. Fig. (a) shows the training process of Coconut when the number of implicit latent tokens is progressively increased. Initially, as the number of latents increases from one to four, the model's accuracy generally improves, suggesting that using more latents can enhance performance. However, a significant drop in accuracy occurs when the number of latents is scaled to five. with performance collapsing to its worst point of 12.5%. The latent instability issue indicates that the implicit reasoning approach is sensitive to the choice of the number of latent tokens, as shown by the sharp drop and subsequent fluctuations in accuracy after adding the fifth latent.
Information Loss. Fig. (b) presents an analysis of how different levels of accuracy are affected by the number of latent tokens, using accuracy metrics at three levels: number, operator, and answer. The bar chart reveals a clear trend: as the number of latent tokens increases from I to 5, there is general decline in performance across all three metrics, especially for the operator accuracy. The strong correlation between increased latent tokens and declining performance, particularly the sharp fall during failure, suggests that implicit latents do not consistently capture the necessary compositional reasoning process without more explicit, fine-grained supervision.
3

---

Published as a conference paper at ICLR 2026
Coconut only answer-level supervision
CE Loss
CODI trajectory-level
CE Loss
CE Loss
implicit supervison
Distil Loss
hidden states
Large Language Model
Large Language Model
Large Language Model
distill
process
SIM-CoT (Ours)
CE Loss
CE Loss
Step prediction
prediction
step-level implicit supervision
Question tokeas
EOS tokens
Large Language Model He
Decoder Model H3
Implicit CeT tokons (1)
Explicit CoT takens (7)
Answer (o)
Figure 2: The framework comparison between Coconut (upper left), CODI (upper right), and our SIM-CoT (bottom). Unlike Coconut and CODI, which apply coarse carse-grained supervision on answers trajectories, SIM-CoT employs decoder to align implicit latents with step-level reasoning. enhancing performance while maintaining inference efficiency.
Shifted Distance. Fig. (c) examines the geometric properties of the latent representations during training. Two metrics are analyzed: the Latent Distance (red), which measures the average distance between pairs of latent vectors, and the Vocab Distance (blue), which measures the average distance from each latent vector to the center of the vocabulary embedding space. When the latent CoT model collapses, the latent distance decreases sharply, indicating that the latent vectors are collapsing and becoming nearly identical, losing their distinctiveness. Simultaneously, the vocab distance increases, showing that these collapsing latents are drifting away from the main lexical embedding space and are no longer grounded in the fundamental token representations used by the model.
Semantic Homogenization. Fig. (d) provides a qualitative analysis of the content of the latent tokens in a normal case versus failed model. In the normal implicit model (middle), the decoded tokens from the latents are diverse and meaningful. the failed implicit model (bottom), the semantic content of the latents becomes highly homogeneous. Latent and Latent 2 contain mainly numbers, lacking operators or symbolic information needed for calculation. This shows that successful training produces latents with step-wins wise reasoning, while without explicit supervision, the latent space collapses into uniform numerical forms.
Summary. Our analysis across Fig. I (a-d) highlights erucial trade-off between diversity and stability. When the model collapses, it loses both its diversity (as the latents become too similar) and its stability (as the latents move away from the token space), leading to catastrophic information loss and complete failure of the reasoning process, as shown by the sharp drop in overall accuracy. These combined findings show that without proper guidance, the latent space degenerates, losing its ability to represent distinct reasoning steps. These challenges motivate our proposed method, which introduces step-level implicit supervision stabilize the training process and enrich unique semantic content of each latent, all while maintaining efficiency during inference.
METHODOLOGY
Overview. As shown in Fig. 2 early implicit reasoning studies differ mainly in supervision granular- ity: Coconut (top left) uses answer-level supervision, while CODI (top right) introduces trajectory- level signals via distillation. Both remain coarse and do not tell the model which latent should encode which step. We propose SIM-CoT, which provides step-level implicit supervision: During an implicit phase, the LLM runs for fixed number K of reasoning steps; each step it takes the last hidden state as the implicit latent 2 and appends to the sequence as the next "token" vector. After K steps, the model switches back explicit decoding over the vocabulary to generate the
4

---

Published as a conference paper at ICLR 2026
final answer. A decoder is used only in training align each 24 with the textual content of the k-th reasoning step; at inference, the decoder is removed, so the runtime is essentially that direct answer generation plus K forward positions, which is far shorter than explicit CoT token lengths.
3.1 NOTATION
Let V be the vocabulary and E E Rivixd the token embedding matrix. A question is æ (XT)..,,TT) E VL with embedded prefix
(1()) (e(z))....,
e(-) R..
We run an autoregressive LLM Fø on any prefix U = (uI..... Um of d-dimensional vectors (tokens or latents). Denote the last-layer hidden state the final position by
Hu ER.
For supervision, the k-th textual step is s (Jk,1..... U., E Vea, and the answer is (a1..... OL.) E Vin The auxiliary decoder has parameters o; the LLM has parameters A.
3.2 IMPLICIT PHASE: LATENT CONSTRUCTION BY LAST HIDDEN STATES
We fix the number of implicit reasoning steps K in advance. For each step k = 1, K,
24 Ho(((((11) R
(7k) (( 2k.
(1)
where @ denotes concatenation along the time axis. The implicit chain-of- is therefore repre- sented as a continuous sequence of hidden states =1:K = (=1,...,, ,zx), which are autoregressively generated and appended the context before the model switches to explicit decoding.
3.3 EXPLICIT PHASE: ANSWER DECODING OVER THE VOCABULARY
After constructing the implicit latents 21:K, the model switches to explicit decoding generate the final answer. Let W. E IR Vixd be the output projection (LM head). With teacher forcing on the partial answer Oct, the generation is
hT+K Ho(U(K) e(acz)),
(2)
po(a; x. 21.K1 act) softmax Wo hoT
(3)
pa x1 II po (ax x. F1:k; act)-
3.4 TRAINING-TIME DECODER AND SEEP-LEVEL SUPERVISION
During training, a decoder Ps (architecturally identical to the LLM) takes only the k-th implicit latent 24 conditioning signal and autoregressively generates the ke-th textual step 8 (gk.1..... Lk. This provides step-level supervision that directly grounds to its corresponding reasoning content:
K
K La
P0(81:K
21:K)
II
(sk =x) = II Polye,t Ek, yk,ct)- 1=1t=1
Parameterization. For step k, the decoder is conditioned on the implicit latent z obtained from the LLM. Since 2ş does not correspond any token in the vocabulary. is included in the loss calculation. Instead, 2ạ is injected as an additional prefix vector that initializes the decoder's 's hidden state for step generation. Concretely, the decoder input sequence
Udec #x e e(vr.1)
L.))
where e(-) denotes the embedding function of the LLM shared between both models. During training with teacher forcing, the decoder predicts each token Sk. autoregressively:
(Bk. 2k. Ve..ce) = softmax dee A
ya.e"

---

Published as a conference paper at ICLR 2026
where "Kt is the decoder hidden state at position t and Wrdes is the LM head of the decoder.
The training loss for step k is then
Cstep.k
log Po (Ukk 2k, Uk,et),
which supervises only the textual step tokens. The decoder is used exclusively for this supervision during training and is discarded at inference.
3.5 OBJECTIVES
Training involves two complementary cross-entropy losses: one for supervising the textual steps through the decoder, and for supervising the final answer through the base LLM.
Step-level supervision. For each implicit latent zg, the decoder generates the corresponding reasoning step 8 1.k.1.--.: :,,,,, Since 2μ: is not vocabulary token, the loss is computed only over the textual step tokens:
K
L
Luep
log Pol Zk, (R.
This loss grounds each latent 2n to a specific reasoning step, ensuring that the latent sequence carries fine-grained semantics.
Answer supervision. After K implicit steps, the LLM Fg switches back to explicit decoding to generate the final answer = (a1..... (L.)) We optimize the standard language modeling loss:
I.
La
log x, #1.K, a.e).
Total objective. The overall loss a weighted sum:
(8) Gradients from Ssep propagate through the decoder into the latent representations 21:K and fur- ther into the LLM (via Eq. equation []. shaping the hidden states to encode step-level reasoning. Meanwhile, fans-tm trains the base model to produce the final answer directly, so the decoder can be discarded inference time without affecting efficiency. Implementation details, inference procedures, and diagnostic analyses are provided in Appendix
C= >sep Luep Am Cuns
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Training Data. We follow previous works (Deng et al. 2024 Hao et al. 2025 to use the GSM8K-Aug dataset Deng al. (2024) for training implicit CoT models. The GSM8k-Aug ex- pands the original GSM8K training set (Cobbe et al. 2021 to 385k examples by using GPT-4 for data generation. To facilitate implieit COT training, the GSM8k-Aug removes the reason- ing chain of natural language, preserving only sequence of structured mathematical expres- sions. Each expression logically linked the previous step, as illustrated by the example: <<12+3=33>><< >=>===->><<>>>> (2t.:66xxxx...xxx11113xx>x<33883
Evaluation Benchmarks. We report results on the GSM8k-Aug test set (Cobbe et 2021) which serves as our in-domain (ID) evaluation benchmark. To further evaluate mathematical reasoning under distribution shift, we also evaluate models on three out-of-domain (OOD) benchmarks: (1) SVAMP PPtell al. 1) a dataset of grade-school arithmetic word problems that introduces simple variations to assess robustness; (2) GSM-Hard (Gao et al., 2022). a modified version of the GSM8k test split where numbers replaced with larger magnitudes increase problem difficulty; Kedziorski et al., 2016) consisting of multi-step arithmetic word problems. Please refer to the Appendix 1) for more details.
and (3) MultiArith (Roy & Roth 2013). a subset of MAWPS -Koncel-Kedziorsk

---

Published as a conference paper at ICLR 2026
Table 1: Main results on GPT-2. We report accuracy (%) on in-domain (GSM8k-Aug) and out-of- domain (GSM-Harg) Hard, MultiArith, SVAMP) benchmarks. Our SIM-CoT is shown to provide accuracy gains on top of existing methods such as Coconut (Hao al. 2025 and CODI (Shen et al. 2025).
In-domain
Out-of-domain
Method
SIM-CoT
GSM8K-Aug
GSM-Hard
MultiArith
SVAMP
Average
Average
Acc. (%)
Tokens
Acc. (%)
Acc. (%)
Ace. (%)
Acc. (%)
Tokens
SFT-CoT
X
42.7
27.6
9.0
85.0
41.6
45.2
24.7
No-CoT
X
19.1
2.2
41.1
16.4
20.6
1.4
iCoT
X
30.1
2.2
5.7
55.5
29.4
30.2
1.4
X
36.6
12.2
8.1
83.5
36.2
42.6
11.4
Coconut
44.8 (+8.3)
12.2
9,3
90.8
40.7
46,9 (+4.)
11.4
X
42.0
12.2
9.4
93.0
41.7
48.0
12.6
CODI
42.6 (40.6)
12.2
92.8
42.6
48.3 (+0.3)
12.6
Table 2: Main results on LLaMA 3.2 1B. We report accuracy (%) on in-domain (GSM8k-Aug) and oft-of-domain (GSM-Hard, MultiArith, SVAMP) benchmarks. Our SIM-CoT builds on CODI to achieve new SOTA in implicit reasoning while setting performance comparable to explicit CoT.
In-domain
Out-of-domain
Method
SIM-CoT
GSM8k-Aug
GSM-Hard
MultiArith
SVAMP
Average
# Average Tokens
Acc. (%)
Tokens
Acc. (%)
(%)
(%)
(%)
SFT-CoT
58.4
25.3
13.9
96.7
65.7
58.8
23.1
No-CoT
28.8
1.2
6.3
50.3
26.7
27.8
1.9
iCoT
X
19.0
1.2
4.4
39.0
40.9
28.1
1,9
X
33,2
13.2
7.0
63.3
43.7
38.0
11.9
Coconut
42.2 (19.0)
13.2
9.3
87.7
43.9
47.0 (-90))
11.9
52.7
13.2
11.9
95.0
60.6
55.8
13.4
CODI
13.2
12.7
96.2
61.5
56.8 (-1.0)
13.4
Implementation Details, We follow the training setup of previous works (Hao et al. 2025 Shen al. 2025), and adopt consistent hyperparameter choices for GPT-2, LLaMA 8B/3B/BB Detailed configurations, such as learning rates, curriculum strategies, are provided Appendix C
4.2
MAIN RESULTS
Baselines. We compare our SIM-CoT against five representative baselines: (1) CoT-SFT: Supervised fine-tuning (SFT) on CoT-annotated data, where the model is trained to generate explicit intermediate reasoning steps followed by the final answer. (2) No-CoT-SFT: Supervised fine-tuning on direct answers only, without producing intermediate steps. (3) iCoT (Deng et al., 2024): A curriculum learning method based on "Stepwise Internalization, which injects CoT reasoning patterns into the model's 's internal representations, enabling it produce more accurate direct answers during inference. (4) Coconut (Hao et al. 2025): A curriculum learning approach that gradually replaces explicit reasoning steps with implicit tokens until the reasoning process becomes fully implicit. This method has shown strong empirical performance and serves as a primary bascline in our experiments. (5) CODI (Shen al. 2025): A distillation-basee method where explicit CoT acts as the teacher and implicit COT as the student. By aligning the last hidden states of the full reasoning trajectory, CODI effectively internalizes knowledge and alleviates catastrophic forgetting.
In-Domain Math Benchmark Results. Table (first column) reports GPT-2 results on GSM8K- SIM-CoT outperforms SFT-CoT and is the first Training-based approach where implicit CoT surpasses explicit COT. With GPT-2 using Coconut as the backbone, it achieves +221 point improvement over SFT-CoT. also exceeds other training-baa based implicit reasoning models; for example, on Coconut, improves by +8.2 points, relative gain of 22.4%. Moreover, when applied on top of CODI-the current SOTA implicit reasoning method Beethod-SIM-orl yields an additional +0.6 point improvement.
Table 2 (first column) shows the results when CODI is used as the backbone. In this setting, our method achieves a substantial +3.4 point improvement. Furthermore, we are the first to achieve performance comparable to SFT-CoT LLaMA-IB IB, reaching 96% of its accuracy. Given that prior studies (Xu et al. 2025 (Shen et al. 2025) reported that curriculum learning in larger models leads to catastrophic forgetting and that homogeneous knowledge harms model training (Zhao al. 2023).
7

---

Published as a conference paper at ICLR 2026
Table Main results on larger LLaMA models (3B and 8B). We report accuracy (%) on in-domain (GSM8k-Aug) and out-of-domain (GSM-Hard, Hard, MultiArith, SVAMP) benchmarks.
In-domain
Out-of-domain
Model
Method
SIM-CoT
GSMBK-Aug
GSM-Hard
MultiArith
SVAMP
Average
# Average
(%)
Tokens
(%)
Acc. (%)
Acc. (%)
(%k
Tokens
SFT-CoT No Co-
71.5
27.7
17.0
98.3
71.0
62.1
22.4
38.3
1.2
9.5
88.7
52.9
50.4
1.4
LLaMA 3.2 3B
CODI
14.3
98.7
73.3
62.1
7.5
62.3 (x1.)
7.2
14.6
98.8
74.9
62.8 (10.7)
7.5
SFT-CoT No-CoT
71.7
27.4
16.5
98.3
73.1
62.6
22.2
LLaMA 3.1
39.5
1.2
9.8
88.0
55.3
51.0
1.6
CODI
61.1
7.2
15.5
99.5
78.1
64.4
7.5
64.1
16.3
100.0
79.4
65.2 (40.8)
GSMRk-Aug
G5M8k--Hrd
MultiArith
SVAMP
45.0
10.0
90.0
40.0
8.8-
75.0
37.0
33.0
60.0
31.0
29.0
6.4
45.0
25.0
SIM-CaT
5:2
SIM-CaT
SIM-CoT
Coconut
Coconut
Coconut
Cocoout
25.0
1-2
3-6
5-10
6-12
8-16
1-2
3-6
6-12
8-18
3-8
5-10
6-12
8-16
1-2
3-8
5-10
8-12
8-16
Latent-Tokros
Latent-Tokoms
Lateni-Tokees
Latect-Tokens
Figure 3: Ablation study on different numbers of implicit latents. The x-axis denotes the number of implicit latents and implicit tokens (joined with *-"), while the y-axis denotes accuracy. The blue line corresponds to our method SIM-CoT, and the orange line corresponds to the baseline Coconut.
we choose CODI as the backbone because its KL-regularized objective constrains the training not to deviate too far from the original model distribution, thereby alleviating catastrophic forgetting.
Out-of-Domain Math Benchmark Results. To evaluate the robustness of our method, we train on GSM8K and evaluate on out-of-domain datasets (GSM-Hard, MultiArith, and SVAMP). From the third column of Table I we observe that SIM-CoT consistently outperforms SFT-CoT, with an average improvement of +4.3 points when using Coconut as the backbone. From the third column of Table 2 our method further improves upon the current SOTA implicit reasoning method CODI by +1.0 point. Moreover, when scaling model size from GPT-2 to LLaMA-18 1B, SIM-CoT enlarges the performance gap against iCoT, Coconut, and other baselines.
We attribute the robustness of SIM-CoT to its step-level implicit supervision. Unlike SFT-CoT, which forces the model to mimic deterministic natural language annotations, and unlike CODI, which applies trajectory-leve alignment to a coarse- coarse-grained reasoning path, our method introduces moderate form of supervision. This design ensures the plausibility of each reasoning step while preserving the diversity of reasoning trajectories, thereby improving generalization to unseen inputs.
Inference Efficiency. In terms of inference speed, our method maintains the other implicit reasoning approaches on both GPT-2 and LLaMA-IB IB. On GPT-2, SIM-CoT not only surpasses SFT-CoT on both in-domain and out-of-domain benchmarks, but also achieves a 2.3x and 2.2x speedup Coconut, respectively. On LLaMA- SIM-CoT remains comparable to SFT-CoT in accuracy while delivering 1.9x and 1.7x speedups on in-domain and out-of-doman benchmarks, respectively. These results demonstrate the effectiveness of our approach in retaining or even enhancing the performance explicit CoT while substantially reducing inference cost.
efficiency as
4.3 ABLATION STUDIES
Ablation on the Number Implicit Tokens. We study the effect of varying the number of implicit latents on GPT-2, comparing SIM-CoT with Coconut trained on GSM8k-Aug and evaluated on GSM8k-Aug, GSM-Hard, MultiArith, and SVAMP (Fig- 3). Following Coconut, each latent corresponds to two tokens. As shown in Fig. 5 most problems involve two to six steps with small proportion of harder cases, so we set the maximum number of implicit latents to 8. For each configuration, we report the best performance, and results show that SIM-CoT provides more stable training and achieves consistent gains over Coconut, indicating that step-level implicit supervision scales effectively with larger latent capacity.
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of (a) LLaMA IB with different decoders and (b) latent token distance analysis. In (a), we evaluate the effect of using larger decoders with a 1B model on both in-domain (GSM8k- Aug) and out-of-domain benchmarks (GSM- Hard, MultiArith, SVAMP). In (b), we report average pairwise distances among latent tokens (Dist.) and their distances to the vocabulary center (Dist. to VC) under different settings, including failed cases and the effect after applying SIM-CoT.
(a) LLaMA IB with different decoders.
(b) Latent token distance analysis.
In-domain
Out-of-domain
Setting
Dist.
Dist. VC
Model
GSMB Aug GSM-Hard MultiArith SVAMP
I latent
20.30
36.20
2 latent
23,46
28.82
Baseline
52.7
95.0
4 latent
27.56
27.83
1B Decoder
56.1
12.7
96.2
61.5
5 latent
28.34
28.34
3B Decoder
50.4
11.6
95.6
59.8
Fail 5 lateut
39.39
94.2
After SIM-CaT
Table Ablation study of soft thinking on LLaMA 3.2 IB. We report accuracy (%) on the in-domain dataset (GSM8k-Augl) and out-of-domall datasets (GSM-Hard MultiArith, and SVAMP). Adding soft thinking consistently improves both Coconut and SIM-CoT across all benchmarks, showing its effectiveness in enhancing implicit reasoning.
Method
GSM8K-Aug
GSM-Hard MultiArith
SVAMP
Coconut
36.6
8.1
83.5
36.2
+ Soft Thinking
36.7
8.3
85.2
36.0
SIM-CoT
44.8
9.3
90.8
40.7
+ Soft Thinking
45.0
9.4
91.5
40.8
Ablation on Scaling to Larger Backbones. examine robustness and scalability, we extend exper- iments to larger LLaMA backbones, including LLaMA 3.2 3B and LLaMA 3.1 8B. Table reports results on GSM8K-Aug (in-domain) and GardH MultiArith, and SVAMP (out-of-domann)
Overall, SIM-CoT scales effectively to larger backbones, consistently surpassing matching explicit CoT on out-of-domain tasks while reducing reliance on trajectory-level supervision.
On LLaMA 3.2 3B, SIM-CoT improves over CODI by +1.5 points on GSM8K-Aug and +1.6 points on SVAMP, while maintaining comparable performance on GSM-Hard and MultiArith. This demonstrates that step-level implicit supervision strengthens strong implicit reasoning baselines even at larger scales.
On LLaMA 3.1 8B, SIM-CoT yields gains of +3.0 points on GSM8k-Aug, +1.3 on SVAMP, and +0.8 on MultiArith relative to CODI, while maintaining stable accuracy on GSM-Hard. Compared with SFT-CoT, it achieves higher accuracy on MultiArith (100.0 vs. 98.3) and SVAMP (79.4 vs. 73.1), while remaining similar on GSM-Hard.
Together, these results confirm that SIM-CoT scales effectively larger backbones, providing consistent gains across both in-domain and out-of-domain benchmarks with reduced reliance on trajectory-level supervision.
Ablation on Different Decoder Sizes. We investigate how decoder size affects performance by replacing the decoder of the LLaMA 1B backbone with larger versions from the same vocabulary family and evaluating on GSM8k-Aug, GSM-Hard, MultiArith, and SVAMP. As shown in Table 4 (a), integrating a 1B-scale decoder leads to consistent improvements across all benchmarks. However, simply scaling the decoder to larger variants (3B or 8B) does not yield additional benefits and instead slightly reduces accuracy.
These results suggest that moderate decoder scaling can enhance reasoning ability, but excessively large decoders may introduce optimization challenges or misalignment with the IB backbone, ultimately limiting generalization. A plausible explanation is that the IB encoder and IB decoder originate from the same model family and thus share a more compatible representation space, facilitating stable learning. In contrast, larger decoders (3B or 8B) may require implicit projection to align with the 1B backbone, which can introduce representational mismatches and hinder training stability.

---

Published as a conference paper at ICLR 2026
[Question] A watermelon farm produced 120 baby watermelons... and found out that 30% of the watermelons were ready for harvest, so she took them home. When she came back two weeks later, 3/4 of the remaining melons were ready, so she harvested them. How many melons were not ready to be harvested after the two weeks?
Large Language Model
Final
Output
Answer: 21
Decoder Model (Optional)
LLM Token Embed
0.3*120-36 36
(120-36)*3/4-0) 84-63=21
Figure 4: SIM-CoT case study on GSM8k. The generated implicit continuous tokens are subsequently interpreted by our decoder, which visualizes the solution intermediate steps leading the final output.
Ablation on Soft Thinking. We also study the effect of integrating soft thinking (Zhang et al. 2025 Wu al,, 2025) with both Coconut and SIM-CoT. For clarity, the detailed experimental setup, results, and analyses provided in Appendix A
Interpretability of Implicit Reasoning. Implicit reasoning models generate continuous latent thoughts that are not directly human-readable We reuse the training decoder to project each latent step into text space, enabling per-step semantic visualization (Fig. Appendix G
To analyze the latent space, examine geometric measures: average pairwise distance (Dist.) and distance to the vocabulary center (Dist. to VC) (Table 4b)), Fig. (6). As latent tokens increase from 5, Dist. grows (20.30-228 indicating better separability: in the failed 5-latent case, collapses (4.21), while SIM-CoT restores and further enlarges (32.81). Dist. to VC decreases from 36.20 28 with more tokens, suggesting improved alignment, but spikes 39.39 in the failed case; SIM-CoT stabilizes it at 29.80. Qualitative results confirm these trends: normal tokens remain separated and grounded, failed tokens collapse and drift, and SIM-CoT recovers a structured, stable latent space.
RELATED WORK
large body of work has studied explicit chain-of-thought (CoT) prompting, including self- consistency (Wei al. 2022 Wang al. 2023). least-to-most prompting (Zhou al., (023). baase reasoning Shinn et al. [023 Madaan et al. 2023). and the integration of external tools Yao al. 2023 Other work investigates step-level supervision to structure explicit reasoning
(Zheng et al. 2023 Wei et al. 2025). While effective, explicit CoT increases inference cost with
longer sequences and often produces redundant steps, limiting efficiency and reasoning diversity
ct al.
2025
Zhang et al. 2025 Xu et al. 2025).
Implicit CoT aims to reduce output length while retaining multi-step reasoning. Prior work explores
knowledge internalization (Deng et al.) 2024). architectural modification (Saunshi et al. 2025 Chen
al. 2025 Cheng & Van Durme 2024 [Su et al., 2025) Mohtashami et 2023] Geiping et al.
2025). , training-free latent construction (Zhang et al. 20255) Wu et al., 2025 and auto-regressive
latent reasoning (Xu et al.) 2025 Tan et 2025). Coconut applies answer-level supervision (Hao)
et al. 2025), and CODI uses trajectory-leve distillation (Shen al. 2025). Our work introduces
step-level supervision, which distributes signals across latent steps and improves stability. See
extended discussion in Appendix B
6 CONCLUSION
We introduce SIM-CoT, training-based implicit reasoning method with step-level supervision on latent tokens. On GPT-2, SIM-CoT outperforms the strong explicit baseline SFT-CoT, while also surpassing implicit baselines such as Coconut and CODI. When scaling larger LLaMA backbones, the performance achieves consistent gains over existing implicit reasoning methods and maintains fast inference efficiency. Ablation studies further show that it improves training stability with more latent tokens and can benefit from integration with training ffning-free free techniques such as soft thinking. Distance analysis confirms that SIM-CoT produces latent representations that are diverse yet stable.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGEMENT
This project is funded in part by Shanghai Artificial Intelligence Laboratory, Shanghai Innovation Institute, the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s 's InnoHK. Dahua Lin is a Pl of CPII under the InnoHK.
REFERENCES
Anthropic. Claude 3,5 sonnet, 2024.
URL https://ww.anttropp.c.
claude-3-55 5-5 --nnnettt
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like Ilms. arXiv preprint arXiv:2412.2111: 187, 2024.
Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking. arXiv preprint arXiv:2502. 2025.
Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv preprint arXiv:2412.13171, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. ArXiv, abs/2405.1488 2024.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211. 1..4555,,,, 2022.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint arXiv:2502.0517.. 2025.
Google. Our next-generation model: Gemini 1.5, 2024. URL https://blog.goont technology/al/ wttp://gggoooelgggeeeee wwwtininneetttttteeeeeee certtteeneraiiioon wwwem-odoo-oooo few.uaryebburrr---2e
next
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1294: 2025.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. COLM, 2025.
Rik Kedzio-kedii Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. MAWPS: A math word problem repository. In NAACL, 2016.
Jindong Li, Yali Fu, Li Fan, Jiahong Liu, Yao Shu, Chengwei Qin, Menglin Yang, Irwin King, and Rex Ying. Implicit reasoning large language models: comprehensive survey, arXiv preprint arXiv.2509.02350, 2025.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. In NeurIPS, 2023.
Meta.
Llama 3.2: Revolutionizing edge and vision with open, cus- tomizable models, 2024.
URL https://aat http://aaaaaaaa..////mm/eaa ctt/bloom Ilama 3-2 connect.. 2024 2724-vissie www.iii-iiiiiiieee mobile mobile-devices
Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. Cotformer: More tokens with attention make up for less depth. In WANT@ NeurIPS 2023, 2023.
11

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19995, 2025.
OpenAl. Hello gpt-4o, 2024. URL https: http:::://peeaa.coo//iio
got--pp--- 40
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In NAACL. 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Technical Report, 2019.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In EMNLP, 2015.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank Reddi. Reasoning with latent thoughts: On the power of looped transformers. arXiv preprint arXiv:2502.17416, 2025.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing cfain-of-thought into continuous space via self-distillation. arXiv preprint arxiv:500..10074 2025.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023.
DiJia Su. Hanlin Zhu, Yingehen Xu, Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint arXiv:2502.03225, 2025.
Wenhui Tan, Jiaze Li, Jianzhong Ju, Zhenbo Luo, Jian Luan, and Ruihua Song. Think silently, think fast: Dynamic latent compression of IIm reasoning chains, arXiv preprint ar51v 2025.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning language models. ICLR, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NIPS, 2022.
Rueg Wei, Haowei Liu, Xuyang Wu, and Yi Fang. A survey on feedback-based multi-step reasoning for large language models on mathematics. arXiv preprint arXiv:2502.1433: 143333, 2025.
Junhong Wu, Jinliang Lu, Zixuan Ren, Ganqiang Hu, Zhi Wu, Dai Dai, and Hua Wu. Llms have a heart of stone: Demystifying the soft thinking ability of large reasoning models. arXiv preprint MrXiv:2508.03440, 2025.
Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. SoftCoT: Soft thain-of-thouuht for efficient reasoning with 1lms. In ACL, 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023.
Zhen Zhang, Xuehai He, Weixiang Yan. Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of llms in continuous concept 2025.
space. arXiv preprint arXiv:2505.10
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang. Junjie Zhang. Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023. URL http::/
wwsp/ass///aooooooaa..../.aaaeee abs/ / 2303, 18223
Chuanyang Zheng, Zhengyyng Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive- hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797,, 2023.
12

---

Published as a conference paper at ICLR 2026
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Ed Chi, and Quoe V. Le. Least-to-mont mastt prompting enables complex reasoning in large language models. In NeurIPS, 2023.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
USAGE OF LARGE LANGUAGE MODELS
In this paper, used LLMs only for minor language polishing and formatting, without generating ideas, analyses, or experimental results.
OUTLINE
In this appendix, we provide additional analyses and supporting materials to complement the main text. In Sec. A present experiments combining soft thinking with SIM-CoT, including setup, results, and a detailed formulation with pseudocode. In Sec. B we provide an overview related work in explicit and implicit chain-of-thougee reasoning. In Secs, C and D we describe our implementation. hyperparameter configurations, boundary conditions for implicit token alignment, training overhead and training/inferrnce procedures, including benchmark and dataset details. In Sec. E we introduce the SIM-CoT training procedure and provide pseudocode for step-level supervision. In Sec. Fi we offer geometric diagnostics of the latent space, analyzing inter-latent distances and distance the vocabulary center. In Sec. G we discuss interpretability analysis, including latent visualization and summary findings. Finally, we provide declarations on LLM usage and additional case studies on GSM8k to further illustrate the reasoning process and visualization choices.
A ADDITIONAL ANALYSIS ON SOFT THINKING
Soft thinking (Zhang al. 2025 Wu et al. 2025) is training-free free method for implicit reasoning in which the latent space is represented as a weighted average over the vocabulary embedding space. In contrast, SIM-CoT' learns latent representations directly from data during training. To knowledge, no prior work has evaluated a combination these two approaches; our experiments provide the first such evaluation.
A.1 SETUP
We apply the proposed soft thinking mechanism on top of both Coconut and SIM-CoT, while adopting GPT-2 as the backbone model. To assess the effectiveness of this approach, we perform evaluations on diverse set of mathematical reasoning benchmarks. The in-domain evaluation is carried out on GSM8k-Aug, which provides augmented training and testing samples closely aligned with the original GSM8k distribution. To further examine generalization beyond the training domain, we include three out-of-domain benchmarks: GSM-Hard, which contains more challenging arithmetic problems with subtle variations in reasoning steps; MultiArith, which evaluates performance multi-step arithmetic operations requiring careful sequencing of addition, subtraction, multiplication, and division; and SVAMP, which focuses variations of elementary word problems designed to test robustness to superficial changes in problem statements.
A.2 RESULTS
Tbb reports the results. Adding soft thinking improves accuracy in most cases. For Coconut, improvements are observed on GSM-Hard (+0.2) and MultiArith (+1.7), with slight decrease on SVAMP (-0.2) For SIM-CoT, soft thinking consistently enhances performance: GSM8k-Aug (+0.2), GSM-Hard (+0.1), MultiArith (+0.7), and SVAMP (+0.1).
A.3 FORMULATION
Let & E 2dd denote a continuous latent token, and E RV be the embedding matrix of the vocabulary V. Our goal is enrich the representational capacity of by incorporating soft thinking, which allows the latent space to draw information not only from its continuous representation but also from the semantic structure of the vocabulary. The process can be described in three steps.
14

---

Published as a conference paper at ICLR 2026
Step 1. Vocabulary distribution. The continuous latent token 2 is first mapped into probability distribution over the vocabulary space:
p (W2)
where W R R is the output projection matrix and p R the resulting distribution. This step can be viewed as interpreting the latent token in terms of vocabulary-leve semantics, where each token V is assigned likelihood according its relevance 2,
Step 2.
Soft-thinking embedding. vocabulary embeddings:
Using the distribution
compute weighted mixture of
Zwoft
E
Σ Po E.; VEV
where E., is the embedding vector corresponding to token e. This operation can be seen as construct- ing a "soft token" that captures multiple semantic hypotheses simultaneously, instead committing to single discrete vocabulary token. As result, Ssoft provides richer and smoother information than a hard token lookup.
Step 3. Combination. Finally, combine the original continuous latent with the soft-thinking emhedding =
z' 0. + 2zoftt
where a cont i mnuuuuusweee ght and 3 so sooft_wel ght are hyperparameters that balance the contribution of the continuous and soft-thinking components. This formulation allows to retain the model's Iearned continuous representations while also grounding them in the vocabulary space. Intuitively. the continuous part encourages compact reasoning within the latent space, whereas the soft soft-thinking component brings in semantic priors from the vocabulary, leading more stable and interpretable reasoning.
The pseudocode implementation of the above process is presented follows.
Algorithm 1 Soft Thinking with Continuous Tokens
Require:
Continuous latent z, embedding matrix E, weights a,
1: if >> 0 then
2:
Compute logits: 1 +- W
3:
Convert to probabilities; p softmax(I)
4:
Form soft embedding: 2soft <- F.
5:
Update latent: z' +- az + 32wn
6:
else
7:
z' f-
8: end if
9: return 2'
A.4 ANALYSIS
The results demonstrate that soft thinking complements training-based implicit reasoning. The hybrid latent integrates semantics learned through training and distributional information from vocabulary mixing, which enables the model to explore diverse intermediate states rather than committing to a single deterministic path. This leads to improvements in both in-domain and out-of-domain benchmarks. Our findings suggest that combining training-free construction with training- based supervision provides gains beyond either approach in isolation.
B RELATED WORK
Explicit chain-of-thought reasoning. Chain-of-thousee th-uuh (CoT) prompting enables large language models (LLMs) to generate intermediate reasoning steps before producing the final answer (Wei et al.) 2022). This approach has been widely studied and extended many directions. Self-consistency samples multiple reasoning paths and selects the majority answer to improve reliability (Wang et al.)
15

---

Published as a conference paper at ICLR 2026
2023).
Least-to-most prompting decomposes a complex question into simpler sub-problems and solves them in order (Zhou et al. 2023). deflection-basee reasoning allows the model to revise or verify its own intermediate steps, leading to better correctness (Shinn al. 2023 Madaan et al.)
2023). Other works focus on using external tools or symholic solvers together with ex explicit reasoning. which further improves accuracy in mathematics and program synthesis (Yao et al. (023). Methods such as progressive-hinl prompting (Zheng et al., (2023) and step-level feedback (Wei et al., (2025) study how supervision can be incorporated into explicit reasoning make reasoning more structured. Despite these advances, explicit CoT has clear draw wawks. Because it generates long token sequences, inference cost grows rapidly with reasoning length, and many intermediate steps are redundant or irrelevant to the final answer. Moreover, since explicit reasoning is restricted to tokens from fixed
vocabulary, it often commits to a single trajectory and shows limited reasoning diversity (Li et al.
2025
Zhang et al. 2025 Xu et al. 2025).
Implicit chain-of-thought reasoning. Implicit CoT performs multi-step computation a continuous latent space instead of emitting long textual traces, reducing decoded length while keeping internal structure. Prior work follows four practical routes. First, knowledge internalization trains models to carry out reasoning internally by progressively removing explicit traces or by using dedicated control embeddings; examples include iCoT-SI (Deng et al., 2024), which removes steps during training to internalize reasoning. Second, architectural modification controls compute by reusing or skipping layers, or by adding light recurrence, so models can refine hidden states without lengthening outputs
(Saunshi et al. 2025; Chen et al. 2025 Cheng & Van Durme 2024] Su et al. 2025 Mohtashami iet al. [2033] Geiping et al. 2025). Third, training-free methods construct continuous latents directly from the model's 's probability distribution over the vocabulary; Soft Thinking mixes embeddings by probability to form "concept" tokens that explore alternative paths without updating weights, which improves efficiency and diversity but does not bind each latent to step-level semantics (Zhang et al.) 2025
Wu et al., 2025).
The fourth route, auto-regressive latent reasoning, updates and concatenates latent states in place of some token-level decoding and is the most relevant to our work (Xu et al. 2025 Tan et al. 2025). Coconut applics answer-level supervision training on the final answer while leaving intermediate latents weakly constrained (Hao et al. 2025). CODI adds trajectory-level distillation by aligning
an implicit trajectory with an explicit CoTl trace, narrowing the gap to explicit CoT but giving only coarse guidance to intermediate steps (Shen al. 2025). However, the implicit token length in CODI is fixed during training, which limits its flexibility and makes it less suitable for scaling to variable or longer reasoning chains. Our framework remains in the auto-regressive setting but changes the supervision: during training, each latent is aligned with its corresponding textual step (step-level supervision), distributing learning signals across the full latent chain to improve stability and semantic fidelity of intermediate states; at inference, the decoder is discarded, ensuring that the decoding cost remains identical to that of standard implicit CoT methods (e.g., Coconut).
C IMPLEMENTATION AND TRAINING DETAILS
We provide the full hyperparameter settings, training procedures, and additional analysis used in our experiments. Unless otherwise specified, we use the AdamW optimizer with 31 = 0.9, Bn = 0.999, and weight decay of 0. Batch size is set to 128 for GPT-2 and LLaMA IB, and 64 for LLaMA 3B and 8B. Early stopping is applied with a patience of 3 epochs. We now describe the training setups for Coconut, CODI, and our SIM-CoT.
C.1 COCONUT TRAINING SETUP
Following Hao et al. 2025 GPT-2 and LLaMA IB (Radford et al. 2019 Metal 2024 are trained with a fixed learning rate of 1 x 10 - One implicit latent corresponds to two implicit tokens. A curriculum is applied: every three epochs, until the maximum number of latent steps is reached. After this expansion, training continues for 15 additional epochs.
explicit reasoning step is replaced by implicit latent
16

---

Published as a conference paper at ICLR 2026
C.2 CODI TRAINING SETUP
For larger backbones such as LLaMA 3B and LLaMA 8B, we adopt task-specific hyperparameter settings to ensure stable training. In particular, we use a learning rate of 3 x 10 - for LLaMA 3B and train for 8 epochs, while for LLaMA 8B the learning rate is reduced to x 10 with 6 training epochs. These choices are motivated by the increased sensitivity of larger models to optimization dynamics, where smaller learning rates and fewer epochs help prevent overfitting and instability.
When reproducing CODI on GPT-2 and LLaMA IB, we strictly follow the configurations reported by Shen al. (025).. Specifically, we a learning rate of 3 x 10-3 with 40 epochs for GPT-2, and a learning rate of 8 x 10-4 with 10 epochs for LLaMA 1B. Adopting these settings ensures that our results are directly comparable prior work and isolates the effect of our proposed method, rather than confounding it with differences in optimization schedules.
C.3 SUMMARY OF HYPERPARAMETERS
Table Training hyperparameters across different models.
Model
Method
LR
Epochs
GPT-2
Coconut
10
15 + curriculum
LLaMA 1B
Coconut
1 1 10
15 + curriculum
GPT-2
CODI
3 10-3
40
LLaMA IB
CODI
8 10
LLaMA 3B
CODI
3x 10-4
LLaMA 8B
CODI 1 10
6
C.4 TRAINING- TIME OVERHEAD
We analyze the computational overhead introduced during training by the auxiliary decoder. Since the auxiliary decoder has the same architecture and number of parameters as the original decoder, and participates an additional forward pass during training, the overall parameter count and memory usage are approximately doubled compared with the implicit baselines Coconut and CODI.
To quantify the training-time overhead, Table 7 reports the wall-clock training hours of SIM-CoT and the corresponding implicit models under identical hardware settings (H800), Across different model scales, SIM-CoT introduces only a moderate increase in training time, ranging from approximately hours for smaller backbones to around 16 hours for larger ones.
Table 7: Training hours of SIM-CoT compared with implicit baselines under identical hardware (H800).
Model
Training Hours
Coconut GPT
180h
SIM-CoT (Coconut GPT)
192h
CODI GPT
~6h
SIM-CoT (CODI GPT)
18.2h
CODI 1B
16.5h
SIM-CoT (CODI 1B)
18.5h
CODI 3B
344
SIM-CoT (CODI 3B)
442h
CODI 8B
~71h
SIM-CoT (CODI 8B)
87hh
These results show that the additional computational cost introduced by SIM-CoT remains modest relative to the improvements in stability and accuracy it provides.
17

---

Published as a conference paper at ICLR 2026
C.5 BOUNDARY CONDITIONS IN IMPLICIT TOKEN- STEP ALIGNMENT
Since the model produces a fixed number of implicit tokens (K) independent of the length of the ground-truth reasoning chain, we formalize the boundary conditions that govern how these tokens are aligned with textual reasoning steps. Let the annotated reasoning contain N steps. Under this formulation, two boundary cases emerge:
Case 1: NN K (Longer reasoning chains). When the reasoning chain contains steps than latent tokens, a direct one-to-one alignment impossible. We adopt many-lo-one strategy: the first K latent tokens aligned individually to the first K reasoning steps, while the final token receives supervision from the concatenation of the remaining steps (steps K through N). This boundary condition ensures that information from longer chains preserved rather than truncated.
Case 2: N K (Shorter reasoning chains). When the reasoning chain is shorter than the number of latent tokens, the first N latent tokens align one-to-one with the available reasoning steps. The remaining latent tokens (#N+1 to = are aligned with the final answer. This encourages the model to utilize its surplus latent capacity to refine the target solution.
These boundary rules guarantee that each implicit token is assigned semantically meaningful supervision signal, regardless of how the reasoning length compares with the fixed latent budget K. They also provide stability during training by preventing both supervision sparsity (when N K) and information loss (when N K).
D TRAINING AND INFERENCE DETAILS
Curriculum for K. We use curriculum schedule to gradually increase the number of implicit steps. Each latent corresponds to two implicit tokens. Let Kmax denote the maximum number of latents. Starting from k(0)) the number of implicit steps after epoch is
(()))
min maxx
Ae
where Ae is the update interval in epochs. Once K(e) reaches K it remains fixed for the remainder of training.
Inference and Efficiency. At inference time, the auxiliary decoder is removed and only the base model is executed:
(0) [e(xi),......... e((r)))),,,,,,T) for h: = 1,..,, K
Z4 Ho( (()) = ((k-1) @ 2k.
and after K implicit steps the model switches back to explicit decoding to generate the final answer sequence a as in Eq. equation 4 The total decoding length is T K + L.s where is the input length, K the number of implicit steps, and L,, the answer length. In practice, the cost is comparable to other implicit reasoning methods because K moderate. In tasks where explicit CoT requires long trajectories (LooT > K), the implicit formulation reduces decoding positions, providing efficiency gains without loss of reasoning accuracy.
Benchmark Detail. SVAMP (Patel al. 2021 (Simple Variations on Arithmetic Math Word Problems) is benchmark dataset designed to test the robustness of math word problem solvers to superficial changes. It contains 1,000 elementary-level arithmetic word problems (grade and below), each involving single unknown and solvable by an arithmetic expression with no more than two operators. The problems are transformations of existing datasets (such as MAWPS and ASDiv-A) with controlled variations in wording. structure, and number values reduce artifacts and superficial cues. SVAMP's average number of reasoning steps required around 1.2, similar to the base datasets, but model performance drops significantly when tested on SVAMP, showing that many models rely on heuristic patterns rather than deep understanding.
GSM-Hard (Gao et al. 2022 is more challenging variant of the GSM8K dataset, intended to test models' ability to cope with harder numerical values. retains the same problem statements the original GSM8K, but replaces many of the numbers with larger and less common numerical
18

---

Published as a conference paper at ICLR 2026
GSM8K Reasoning Step Distribution
Figure 5: Distribution of reasoning steps in the GSM8K-Aug training dataset. Most problems involve two to four steps, with a long-tail of harder cases. For visualization, step counts with fewer than 200 problems are omit- ted, though all examples are used in training.
150000
Number of Probleems
10Kk
100000
75000
50000
25000
TV
36
0
Number of Steps
values, making superficial arithmetic computation and reasoning harder. The dataset has about 1,319 examples (matching the GSM8K test set size).
MultiArith (Roy Roth 2015) is dataset of multi-step arithmetic word problems designed to challenge systems to correctly sequence multiple operations. It contains 600 problems collected from educational sources, each solvable by one equation involving two or more of the four basic operations (addition, subtraction, multiplication, division). The problems require reasoning over multiple sentences to extract and combine numeric quantities, understand the implied operations, and compute the result. MultiArith has been widely used benchmark for evaluating arithmetic reasoning generalization, particularly for models that go beyond single-operation problems. Analyses show that many models struggle on these examples compared to simpler datasets, highlighting the importance of handling compositional and sequential numerical reasoning.
Training Data. GSM8K-Aug (Deng al. 2024) is the only training corpus augmented dataset derived from GSM8K (Cobbe et al. 2021), expanding the original 8.5k training problems to roughly 385k examples through paraphrasing. numerical resampling, and synthetic generation with GPT-4. The distribution of reasoning steps in GSM8K-Aug is illustrated in Fig. 5 where the majority of problems require persists. This balance of common and complex instances makes GSM8K-Aug particularly suitable for training models that need generalize across reasoning difficulty levels.
It is
four steps, while long-tail of six
steps
E SIM-CoT TRAINING IMPLEMENTATION
We provide pseudocode for the SIM-CoT training process, which illustrates how continuous latent embeddings are aligned with explicit supervision in the explicit chain is mapped to corresponding latent representation, and the training objective enforces consistency between the predicted latent tokens and the ground-truth step annotations. This design ensures that the model learns to represent intermediate reasoning steps in compact latent space while still retaining interpretability through explicit alignment. By supervising at the step level rather than only at the final answer trajectory level, SIM-CoT enables finer control over the reasoning process and reduces instability that often arises when scaling to longer chains or larger numbers of implicit tokens.
step level. particular, each reasoning step
F GEOMETRIC DIAGNOSTICS OF THE LATENT SPACE
We analyze the geometry of latent representations with two metrics.
Inter-latent distance.
Dist(21:k) K(K - 1) ISiCiŠK
Σ
1 -
(9)
A larger value indicates better separation, reducing the risk of collapse.
Distance to vocabulary center. Let u 1 Evey Eg denote the mean embedding. Then,
K 1112
Dist = K
Σ 2|
(10)
19

---

Published as a conference paper at ICLR 2026
Algorithm 2 SIM-CoT Training Procedure
Require: Batch size b, number of thoughts C, continuous embeddings Z, tokenized inputs X, embedding matrix
1: for each thought t 1, ....., do
2: for each sample i
Extract continuous embeddings 21,t from Z 4: Obtain token embeddings ci.e from ((xs.)) Concatenate embeddings: hi,t + [2 C2,t Build attention mask mi,t up to BOS
3:
5:
6:
7: Assign position ids Pi.t
8: Prepare labels Vi.t with masked tokens set to -100
9: end for
10: end for
11: Pad and stack {h, m. P. p) to maximum sequence length
12: Prepare 4D attention mask: M < PrepareMask(MM)
13: Forward pass: Ô < ExplainableLLM(HH M P)
14: Extract logits: L < O.logits
15: Shift logits and labels: L' + Ll:: -], Y' + [; 1:
16: Compute cross-entropy loss: = CrossEntropy(L' YY)
17: Normalize & over valid positions
Ensure: Final training loss L
Moderate values indicate that latents remain close enough to the lexical manifold for stability, while avoiding collapse toward the center. These diagnosties are not used training but serve as indicators of diversity and stability in the learned latent space.
G ADDITIONAL DETAILS FOR INTERPRETABILITY ANALYSIS
G.1 MAKING IMPLICIT REASONING VISIBLE
Continuous thoughts produced by implicit reasoning models are represented as latent embeddings that do not correspond discrete vocabulary tokens, and therefore cannot be directly decoded by tokenizer. This makes it difficult to interpret how the model internally organizes multi-step reasoning. To address this, reuse the decoder that was employed for step-level supervision during training, and apply it at inference time to map each latent embedding into a readabeee token sequence.
As illustrated in Figure 4 the process begins with natural language problem (e.g., math word problem) that is embedded and passed into the large language model. The model generates sequence of implicit latent tokens, which capture intermediate reasoning steps in continuous space. These latent tokens are then fed into the optional decoder, which translates them into interpretable expressions. Each latent corresponds one reasoning step, and the autoregressive generation order encodes the dependency structure across steps.
For example, in the GSMSK case study shown in the figure, the first latent is decoded as 0.3x 120 36, representing the number of watermelons harvested initially. The second latent builds upon this result 84 63. and the final latent derives the answer 84 - 63 21. For clarity and to save space, the figure merges the second and third steps into single box, but the actual implicit reasoning unfolds across four distinct latent steps. This sequence of decoded latents mirrors the logic of explicit chain-of-though reasoning, while being produced implicitly within the latent space.
compute 120 36 84, the remaining melons. The third latent then calculates
By projecting implicit tokens into interpretable space, gain direct visibility into how the model structures multi-step reasoning. This not only enables analysis of the correctness and consistency of intermediate steps, but also highlights the dependencies across steps that underlie the final prediction. The visualization confirms that SIM-CoT can encode semantically meaningful and logically ordered reasoning steps in its latent space, bridging the gap between implicit and explicit reasoning.
20

---

Published as a conference paper at ICLR 2026
Leserabi Hoazz ve Vocuisaary trateakingg
Learnalle Tokene vo Woohheary Beeboddlngs
Leurnaioo Telerta vo Vocdbslary Emiteadingg
Implici Tokens2 Vecabriary Tokons
Implicit Tokens2
Wecdaslary Tokuna
Imppliit Tokcns2.
Voestulbry Tikees
Implicit Tokensl
Leseraba Inooza
Loamablee Tokces
Implicit Tokens
Leornatia Tokons
Implicit Tokensl
PCX Dimirann
Normal Implicit
2
Tokens
-2
-1
Learnetee Tokerte Vecalustery Embeedingg
Learzalte Tolece Vecatralery Embroling
Leornchie Tobous vs Vocobulary Embeddiiggs
@Implicit Tokens2
Vocalhetary Toinns
40- @lmplicit Tokens2
Vocatelary Tolots
@lmplicit Tokens!
Souababer Takon
Implicit Tokeus]
Leamatie Tahree
Implicit
Laomable Tabres
Leaenako Tokens
TCA Dionriio
PCA Teeccssion
Failed Implicit Tokens
-4
-2
Leummable Tokena vs Wecabalary Secbedrings
Loumable 153mne Swoabaaary Brabeddlegss
Learzalio Tedoets Ve Wesibatery Eraheeddieps
Implicit Tokens
Woalbiaay lolnes
15.0
Lasmabie Thloos
SCA Deserroon
10.0
PCA Disnsns
After SIM-CoT Implicit
7.5
Implicit Tokens2
Implicit Tokens
Implicit Tokens2
Tokens
Implicit Tokens|
Weotbalary Tokara
Implicit Tokens I Laurtabbee thanto
Wentrzlary Tolose
-1
Figure 6: Visualization of distances among implicit tokens and their distances to the vocabulary center. The first row shows normal implicit tokens with well-separated representations, the second row illustrates failed implicit tokens where distances collapse and drift away from the vocabulary center, and the third row presents implicit tokens after applying SIM-CoT, which restores both separation and stability in the latent space.
G.2 SUMMARY
Overall, the results demonstrate that SIM-CoT establishes a balance between diversity and stability in the latent space. Larger inter-latent distances mitigate representation collapse, while moderate distances to the vocabulary center prevent excessive drift. This equilibrium supports stable implicit reasoning and provides robustness when scaling more latent tokens.
H ADDITIONAL CASE STUDIES ON GSM8K
In practice, implicit reasoning continues to produce latent tokens after the correct answer has been reached. These trailing latents no longer introduce new steps but simply repeat the final prediction. For clarity, we omit such redundant tokens the visualizations. As a result, only the latents that correspond to meaningful intermediate steps are displayed in Figure [ while those mapping directly the final answer are hidden. This choice improves readability without changing the underlying reasoning process.
Notably, the decoded reasoning steps consistently match the semantic structure of explicit chain-of- thought annotations, while being generated implicitly within the latent space. The final predictions align with the ground-truth answers, demonstrating that SIM-CoT is capable of encoding interpretable and step-ordered reasoning without requiring explicit supervision inference time.
21

---

Published as a conference paper at ICLR 2026
Toulouse has twice as many sheep as Charleston, Charleston has times many sheep Seattle. How many sheep do Toulouse, Charleston, and Seattle have together Seattle has 20 sheep?
[Question]
### Answer: 260
4*20=80 latent1
2 80 160
160+80+ 20 20-260
latent2
latent3
[Question] Claire makes a egg omelet every morning for breakfast. How many dozens of eggs will she eat in weeks?
### Answer: 7
3*7=21 latentl
21 84
84/12-7
latent2
latent3
[Question] Billy sells DVDs, He has customers on Tuesday, His first 3 customers buy one DVD each. His next 2 customers buy DVDs each. His last 3 customers don't buy any DVDs. How many DVDs did Billy sell on Tuesday?
### Answer: 7
3*1=3 latentl
2
3+4-7
latent2
latent3
[Question] Richard lives in an apartment building with 15 floors. Each floor contains units, and 3/4 of the building occupied. What's the total number of unoccupied units In the building?
### Answer: 30
15*8=120 latentl
120*3/4-90 latent2
120-90=30
latent3
[Question] Poppy is solving a 1000-piece jigsaw puzzle. She places a quarter of the pieces the board, then her mom places third of the remaining pieces. How many jigsaw pieces are left to be placed?
#### Answer: 500
1000/4=250 latentl
1000-250-755 750
1500332500 latent3
750-250=500
latent2
latent4
The marching band is ordering new uniforms. Each uniform comes with hat that costs $25, jacket that costs three times as much the hat, and pants that cost the average of the costs of the hat and jacket. How much does each uniform cost total?
[Question]
### Answer: 150
25*3=75
25+75=100 latent2
100/2=50
100+50=150 latent4
latentl
latent3
Figure 7: Additional SIM-CoT case studies on GSM8k. Each example illustrates how implicit latent tokens correspond to intermediate reasoning steps. Arrows indicate the dependency relations across steps, while colored spans in the question highlight the textual evidence that supports each step. The decoded sequence of latent steps produces the correct final answer, which matches the ground-truth label.
22


Reviews:
[{"review_id": "sBwhHwfSRK", "reviewer": "Reviewer_4kCa", "summary": "This paper introduces SIM-CoT, a training module that stabilizes implicit CoT reasoning by adding step-level supervision through an auxiliary decoder, preventing latent collapse as reasoning tokens scale. The decoder aligns each implicit token with its explicit reasoning step during training, then is removed at inference, preserving efficiency while boosting accuracy and stability. SIM-CoT improves both in-domain and out-of-domain performance, surpassing explicit CoT and prior implicit methods like Coconut and CODI across GPT-2 and LLaMA models.", "questions": "1. How sensitive is SIM-CoT to the choice or quality of explicit reasoning data used for step-level supervision?\n2. Is the auxiliary decoder’s alignment learned jointly with the main model, or does it require separate supervision or tuning?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces step-level supervision that effectively stabilizes implicit CoT training and prevents latent collapse.\n\n2. Achieves higher accuracy and stability than both explicit and prior implicit CoT methods, while keeping inference fully efficient.\n\n3. Provides interpretability by projecting latent tokens onto explicit reasoning steps, enabling per-step visualization and diagnosis.", "weaknesses": "Evaluation is mainly on math-style reasoning, leaving generalization to other domains less explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVJXXbZuBo", "reviewer": "Reviewer_UZJG", "summary": "This paper aims to improve existing implicit CoT, such as Coconut and CODI. Basically, the framework is on top of CODI, by distilling the knowledge from COT-SFT (teacher model: decode all explicit tokens) to the student model (implicit cot), i,.e., three loss, teacher loss is to align teacher's answer token to be gt, one is student loss to align the final answer token to gt, one is distillation loss to minimising the discrepancy of hidden states of teacher and student.  In the proposed SIM-COT, the authors introduce the K intermediate reasoning token (process reward) as \"step-level\" supervision to align the intermediate latent states, z_1 to z_K.\n\nIn experiment results, (as the framework is more based on CODI, coconut using a different curriculum learning paradigm),  they evaluate GPT2 and Llama3.2-1B, 3B and 8B in both in-domain and out-of-domain evaluation, showing average improvements between 0.3% to 3.4%.", "questions": "See weakness above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Implicit CoT is a promising direction, but the accuracy still requires to be improved. \n2. It introduce the process supervision to improve  the performance.", "weaknesses": "1. For experiment results, SIM-COT is still inferior than COT-SFT, in which the CODI is lower than COT-SFT, even introducing the intermediate supervision.  Even comparing with CODI, the performance boost is not significant, especially in OOD setting, 0.3%, 1.0% and 0.7%, 0.8%. \n2. One of the most distinguished objective is to align the intermediate z to intermediate reasoning steps (tokens), however, this process is not clearly described, and not shown the generalisability to other datasets. For example, \n- only evaluate on math, in which the intermediate reasoning steps are clearly defined and available, how about how datasets without process reasoning process annotation? \n- How is the performance fluctuation, when number of latent tokens in training is different from test (how many implicit tokens set to be, still the pre-defined K)? Table 3 show the performance difference with COCONUT, not CODI. \n- the number of implicit token z_t is fixed, whatever how many the reasoning steps are, so how did you align the z_t with intermediate reasoning steps? any other alternatives, as aligning with intermediate reasoning steps hurts the generalisability (there could be multiple trajectories reach the correct answer)\n- many experiment setups are inspired from CODI, such as In-domain, OOD, Ablation study on different numbers of implicit latents, \n- it is unclear how to visualise the implicit tokens, the top-k decoded token? is it the intermediate result token, or including the operation as well? CODI can only observe some patterns that the intermediate z can decode intermediate result token, what is the advantages after introducing such process supervision?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxdggvOFus", "reviewer": "Reviewer_7Zpg", "summary": "The paper introduces SIM-CoT that improves implicit reasoning in LLMs. Unlike prior implicit CoT methods such as Coconut and CODI, which only provide coarse supervision at the answer or trajectory level, SIM-CoT introduces fine-grained, step-level supervision. Empirical results show that SIM-CoT is a plug-and-play module that can be seamlessly combined with existing implicit reasoning methods to enhance both accuracy and stability.", "questions": "see Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is useful, especially since it functions as a plug-and-play module that can be easily combined with other approaches.\n2. The paper’s presentation is overall clear and well-structured, with a logical and coherent flow of ideas.", "weaknesses": "1. At the beginning, the paper claims that one of the reasons for the instability of prior implicit CoT methods is that implicit tokens **collapse into homogeneous latent states**. This part could be better supported experimentally. Although the authors show that their method is indeed more stable than Coconut when the number of latent tokens increases (figure 3), it would be even more convincing if they included additional measurements or visualizations of the heterogeneity of latent tokens (for example, some measure related to the distance of latent tokens) in the main body.\n\n2. While the proposed method introduces more fine-grained step-level supervision compared to prior approaches, this also adds some trouble to training: the model must rely on **preprocessed, structured**, fine-grained training data, which may limit the scalability of the method.\n\n3. The authors mention that the extra decoder model is architecturally identical to the LLM (line 252). This means the two share parameters, or if the decoder is a smaller, separate model?\n\n4. Although the authors report the average number of tokens used during inference, the introduced decoder model also generates a sequence of tokens at each step for supervision. My concern is that the training cost might be higher than Coconut or even SFT-CoT. A more detailed discussion of training overhead would help clarify the efficiency and practicality of the proposed method.\n\n5. Sections 3.1–3.5 are presented clearly but the notations are quite dense. It would be more reader-friendly if some of the notations were also depicted in Figure 2, which would help distinguish the different symbols more easily.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "861pvpTWm8", "reviewer": "Reviewer_URA3", "summary": "The paper introduces SIM-CoT, a training module that improves implicit Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs). It addresses the performance gap and instability issues found in current implicit CoT methods.\n\nImplicit CoT methods suffer from a \"latent instability\" issue when their computational budget (number of reasoning tokens) is increased. The authors' analysis shows this instability arises because latent representations become homogeneous and lose semantic diversity, which is caused by insufficient step-level supervision. This can cause training to collapse and performance to drop sharply.\n\nDuring training, it uses an auxiliary decoder to align each implicit token with its corresponding explicit reasoning step. This ensures the latent states capture distinct information. The auxiliary decoder is removed during inference, preserving the token efficiency of implicit CoT with no added overhead. The decoder allows for the visualization and diagnosis of each latent reasoning step.\n\nIt provides a systematic analysis of the latent instability issue in implicit CoT approaches, showing it stems from insufficient supervision.\nThe introduction of SIM-CoT, which uses step-level supervision to improve performance and stability with minimal inference overhead.\nExtensive experimental validation showing consistent performance gains across a range of LLMs, including GPT-2 and LLaMA 3 models.", "questions": "This paper introduces SIM-CoT, a novel training module that addresses a key limitation in implicit Chain-of-Thought (CoT) reasoning. The authors identify and provide a rigorous diagnosis of a \"latent instability issue\" that causes performance to collapse when scaling implicit methods. The proposed solution, which uses a training-only auxiliary decoder to apply step-level supervision, is both elegant and effective.\n\nThe paper's strengths are significant. The diagnosis of the problem is original and well-supported by a multi-faceted analysis. The experimental results are strong and comprehensive, demonstrating that SIM-CoT not only boosts the performance of existing state-of-the-art implicit methods across various model scales but can also surpass a strong explicit CoT baseline in both accuracy and token efficiency. The clarity of the presentation, particularly the high-quality figures and diagrams, makes the work easy to understand and appreciate. The contribution is highly significant, offering a practical path toward making efficient reasoning methods more powerful and reliable.\n\nThe paper has a few limitations that should be addressed. The current method relies on a rigid, one-to-one alignment between latent tokens and reasoning steps and uses a fixed number of reasoning steps (K) at inference, which may not be optimal for problems of varying complexity. The authors also do not discuss the training-time overhead introduced by the auxiliary decoder, focusing only on inference efficiency. Finally, the experimental validation is confined to mathematical reasoning tasks, which limits the claims about general \"complex reasoning\".", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The primary originality lies in the identification and framing of the \"latent instability issue\". While the performance gap between implicit and explicit Chain-of-Thought (CoT) was known, this paper is the first to provide a rigorous, multi-faceted diagnosis of why scaling implicit methods fails. It introduces novel analytical concepts like \"semantic homogenization\" to explain the collapse of the latent space, a significant contribution to understanding the failure modes of these models.\nThe proposed SIM-CoT method is a creative and effective combination of existing architectural components. It leverages an auxiliary decoder—a standard component—in a novel way: as a temporary, training-only tool for applying fine-grained, step-level supervision to otherwise unconstrained latent states. This is a departure from prior work that relied on coarser answer-level or trajectory-level supervision.\nThe work directly addresses and removes a critical bottleneck in implicit CoT research: the inability to effectively scale the computational budget (i.e., the number of latent tokens). By stabilizing the training process, SIM-CoT allows for the use of more latent tokens, which in turn boosts performance where previous methods would fail.\nThe central claims are exceptionally well-supported. The diagnosis in Figure 1 is not based on a single metric but is corroborated through a comprehensive analysis of accuracy, information loss (operator vs. number), geometric properties of the latent space, and qualitative examples of decoded latents.\nThe experimental setup is of high quality. The authors compare against a comprehensive suite of strong baselines, including explicit CoT and state-of-the-art implicit methods like CODI. The evaluation is robust, conducted across multiple model families (GPT-2, LLaMA) and sizes (1B to 8B) on both in-domain and out-of-domain benchmarks to ensure the findings are generalizable.\n\nThe paper follows a clear, logical narrative that is easy to follow. It begins by identifying a problem, provides a detailed diagnosis, proposes a well-motivated solution, and validates it with extensive experiments. The writing is precise and unambiguous.\nThis work marks a significant milestone for implicit reasoning. It is the first training-based approach to show that an implicit CoT method can surpass a strong explicit CoT baseline on a standard benchmark, both in accuracy and token efficiency. This result helps to close a critical performance gap and makes implicit reasoning a more viable and attractive alternative for practical applications.\nSIM-CoT is designed as a \"plug-and-play\" module that is shown to boost the performance of multiple existing implicit methods. This high degree of applicability makes the contribution immediately useful to other researchers. The gains in inference efficiency are substantial and directly address one of the most pressing challenges in deploying large-scale reasoning models", "weaknesses": "1. The method enforces a strict one-to-one mapping between implicit tokens and textual reasoning steps, which may not reflect the variable complexity of real-world reasoning.\nSuggestion: Explore flexible alignment mechanisms, such as allowing a variable number of latent tokens per reasoning step based on its complexity.\n\n2. At inference, the model generates a fixed number of implicit tokens (K), which is inefficient as problems require different reasoning lengths.\nSuggestion: Implement a dynamic stopping mechanism, such as training the model to generate a special \"end-of-thought\" latent token to terminate the reasoning phase adaptively.\n\n3. The paper notes that using larger decoders degrades performance but offers only a speculative explanation.\nSuggestion: Conduct a deeper analysis to explain this counterintuitive result, for example, by measuring gradient flows or using representation analysis to check for \"misalignment\" between the LLM and decoder.\n\n4. The paper highlights the lack of inference overhead but does not discuss the increased computational cost (memory, time) during training due to the auxiliary decoder.\nSuggestion: Quantify the training-time overhead (e.g., increase in parameters, memory usage, and wall-clock time) to provide a complete picture of the method's trade-offs.\n\n5. The evaluation is confined to mathematical reasoning benchmarks. Claims about improving \"complex reasoning\" would be stronger if validated on a broader range of tasks.\nSuggestion: Evaluate the method on other domains, such as commonsense (e.g., StrategyQA) or symbolic reasoning, to demonstrate the generalizability of the approach beyond arithmetic problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)