SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens
Authors: Yinhan He
Wendy Zheng
Yaochen Zhu
Zaiyi Zheng
Lin Su
Sriram Vasudevan
Qi Guo
Liangjie Hong
Jundong Li
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models (LLMs)
Chain of Thought (CoT) reasoning
Primary area: applications
TL;DR: We make Chain-of-Thought reasoning in large language models (1) more efficient by creating implicit reasoning with lightweight language models; (2) still effective as the implicit reasoning maintains semantic alignment with ground-truth reasoning.

Abstract: Chain-of-Thought (CoT) enhances the performance of Large Language Models (LLMs) on reasoning tasks by encouraging step-by-step solutions. However, the verbosity of CoT reasoning hinders its mass deployment in efficiency-critical applications. Recently, implicit CoT approaches have emerged, which encode reasoning steps within LLM's hidden embeddings (termed ``implicit reasoning'') rather than explicit tokens. This approach accelerates CoT reasoning by reducing the reasoning length and bypassing some LLM components. However, existing implicit CoT methods face two significant challenges: (1) they fail to preserve the semantic alignment between the implicit reasoning (when transformed to natural language) and the ground-truth reasoning, resulting in a significant CoT performance degradation, and (2) they focus on reducing the length of the implicit reasoning; however, they neglect the considerable time cost for an LLM to generate one individual implicit reasoning token. To tackle these challenges, we propose a novel semantically-aligned implicit CoT framework termed **SemCoT**. In particular, for the first challenge, we design a contrastively trained sentence transformer that evaluates semantic alignment between implicit and explicit reasoning, which is used to enforce semantic preservation during implicit reasoning optimization. To address the second challenge, we introduce an efficient implicit reasoning generator by finetuning a lightweight language model using knowledge distillation. This generator is guided by our sentence transformer to distill ground-truth reasoning into semantically aligned implicit reasoning, while also optimizing for accuracy. SemCoT is the first approach that enhances CoT efficiency by jointly optimizing token-level generation speed and preserving semantic alignment with ground-truth reasoning. Extensive experiments demonstrate the superior performance of SemCoT compared to state-of-the-art methods in both efficiency and effectiveness. Our code can be found at https://github.com/YinhanHe123/SemCoT/.

SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Alineed Implicit Tokens
Yinhan He" University of Virginia Charlottesville, VA nee7ne@virginia.ee
Wendy Zheng* University of Virginia Charlottesville, VA mcdSef@virginia.cm
Yaochen Zhu"
University of Virginia Charlottesville, VA ugp4qhvvirginia.cmm
Zaiyi Zheng University Virginia Charlottesvillle VA s jc4fqvvirginia.c edu
Lin Su LinkedIn Inc. Sunnyvale, CA 1sudlinkedin.c com
Sriram Vasudevan LinkedIn Inc. Sunnyvale, CA svasudevan@linkeednn com
Qi Guo LinkedIn Inc. Sunnyvale CA gguo@linkedin.com com
Liangjie Hong LinkedIn Inc.
Jundong Li University of Virginia. Charlottesville, VA j1 jigk@viigi edu
Sunnyvale, CA
btahhng@@inkkdinncom
Abstract
Chain-of-Thought (CoT) enhances the performance of Large Language Models (LLMs) reasoning tasks by encouraging step-by-step solutions. However, the verbosity of CoT reasoning hinders its mass deployment in efficiency-critical applications. Recently, implicit CoT approaches have emerged, which encode reasoning steps within LLM's hidden embeddings (termed "implicit reasoning") rather than explicit tokens. This approach accelerates CoT reasoning by reducing the reasoning length and bypassing some LLM components. However, existing implicit CoT methods face two significant challenges: (1) they fail preserve the semantic alignment between the implicit reasoning (when transformed to natural language) and the ground-truth reasoning, resulting in a significant CoT performance degradation, and (2) they focus on reducing the length of the implicit reasoning: however, they neglect the considerable time cost for an LLM to generate one individual implicit reasoning token. To tackle these challenges, we propose a novel semantically-alignee implicit CoT framework termed SemCoT. In particular, for the first challenge, we design contrastively trained sentence transformer that evaluates semantic alignment between implicit and explicit reasoning, which used to enforee semantic preservation during implicit reasoning optimization. To address the second challenge, we introduce an efficient implicit reasoning generator by finetuning lightweight language model using knowledge distillation. This generator is guided by our sentence transformer to distill ground-truth reasoning into semantically aligned implicit reasoning, while also optimizing for accuracy. SemCoT is the first approach that enhances CoT efficiency by jointly optimizing token-level generation speed and preserving semantic alignment with ground-truth reasoning. Extensive experiments demonstrate the superior performance of SemCoT compared state-of-the-art methods both efficiency and effectiveness. Our code can be found at httttt::///ttiiiia com wwwp///aaaneeee2/////eaeaee
These authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Shorter than explicit
Fin
Ans
Enput query tokenss
Unembedding layer
Hidden embeddings
Genercted reason, tokens
Gensrated answer Takans
Input emb.
Input emb.
Input query
Fin
Input query
Tokenization
(a) Original CoT
(b) Implicit CoT
Figure I: Illustration of how implicit CoT approaches improve CoT efficiency. "Ans." is the answer. Curl arrows represent that the tokens are autoregressively generated. r,s explicit reasoning tokens.
Introduction
Chain-of-Thon -rrrtghh-TThohht (CoT) [49] is a technique where Large Language Models (LLMs) demonstrate step-by-step reasoning by breaking down complex questions into sequential steps. This approach can be implemented through finetuning with specialized datasets [25] [54] providing crafted prompt
instructions [49]. CoT has achieved remarkable performance in various reasoning-intensive NLP
tasks such mathematical problem solving (20
32 28 and symbolic reasoning (52
In light (CoT) prompting. several
28 42
of the impressive performance gains achieved through Chain-of-Thouse
advanced reasoning models have recently been developed, including OpenAl's o1/o3 models [T8] 55 and DeepSeck RI [13]. These models have been specifically trained to tackle problems that require complex, long-chain reasoning. However, as CoT reasoning becomes particularly verbose for LLMs with extensive parameters, it also significantly extends LLMs' reasoning time. For example, ChatGPT- 4o 117 may generate as many as around five hundred tokens during CoT reasoning (taking up 21.37 seconds (3) to complete grade school-level math problem, whose answer is simply numerical number with two to four tokens. Several works aim to improve CoT efficiency, with one primary strategy being implicil CoT 7 14
53 41 8 shown in Fig. 1 Implicit CoT substitutes the explicit reasoning tokens with a small number of LLM'' first-layer hidden embeddings (each embedding is called implicit CoT token). Implicit CoT can improve efficiency because it not only pursues more concise reasoming (box in the middle Fig. (bb) but also avoids going through unembedding layers (box at the upper side in Fig. (a)) to decode from last-layer hidden embeddings to tokens, and the tokenization (box at the lower side Fig. (a)) of reasoning tokens.
40
3
43
However, current implicit CoT methods encounter two significant challenges: (1) Gap towards Maintaining Effectiveness: Current implicit CoT methods struggle establish semantic aligmment between implicit embeddings (when transformed to natural language) and ground-trute reasoning. This "semantic alignment difficulty" stems from a fundamental format mismatch: implicit CoT exists as hidden embeddings while ground-truth reasoning consists of natural language. Existing approaches inadequately address this by either: (/) completely discarding ground-truth reasoning [12 53 (ii)
matching only keywords from ground-truth reasoning (3), or (iii) first finetune LLMs with ground- truth reasoning, then optimizing implicit reasoning only for generating correct answers 14
(2)
Gap towards Enhancing Efficiency: Although some implicit CoT methods 3 reasoning length, they neglect the high computational cost for generating each reasoning token with the LLM. This challenge becomes particularly important when LLMs scale substantial sizes (hundreds of billions of parameters) nowadays. For example, generating one token can cost as high as approximately 0.1s for DeepSeek-RI R1 [13] 34]. The computational overhead accumulates significantly during reasoning. especially for complex problems requiring extensive step-by-step thinking.
reduce
To tackle the above challenges, in this paper, we propose a novel framework named SemCoT (Semantically-aliung Implicit CoT) to accelerate implicit CoT while effectively preserving the performance benefits of traditional CoT methods. Specifically, to tackle the first challenge, propose a customized sentence transformer to measure the semantic alignment between implicit reasoning and ground-truth reasoning. The customized sentence transformer converts each of the im-
'The figure shows how most (not all) implicit CoT methods perform inference with query.

---

plicit reasoning and ground- reasoning into an embedding vector that characterizes its semantics. Thus, we can compare their semantic alignment by cosine similarity [2]. The well-trained, customized sentence transformer is utilized to optimize implicit CoT generation to be semantically aligned with ground-truth reasoning. To tackle the second challenge, we adopt a lightweight language model (LMM) which is an off-the-shelf LM that is distilled or sheared (i.e., pruned) from the original LLM, to generate implicit reasoning. This approach dramatically reduces the time cost of generating each single CoT token. The lightweight LM is guided by our sentence transformer to distill ground-truth reasoning into semantically aligned implicit reasoning, while also optimized for answer accuracy [.].
The main contribution of this paper is summarized as follows. (1) Problem Identification. We uncover two fundamental gaps existing implicit CoT: the failure to preserve semantic alignment between implicit and explicit reasoning and the computational cost of generating individual reasoning tokens. (2) Method Design. We propose a novel framework named SemCoT, which jointly optimizes the token-level generation speed of implicit reasoning while maintaining semantic alignment with ground reasoning.
(3) Experimental Evaluation. We conduct comprehensive experiments to test SemCoT and state-of-the-art implicit CoT reasoning baselines on multiple LLMs and real-world NLP tasks to verify the effectiveness and efficiency of the proposed framework SemCoT.
Preliminaries and Problem Definition
Preliminaries. We introduce the terminology throughout the work. Let Q denote a query requiring complex reasoning, and Y [y1, .... yN] represent the ground-truth answer of N tokens. The original CoT reasoming process generates M ground truth explicit reasoning tokens R = [r1, r before producing Y. In contrast, our approach leverages implicit reasoning tokens, denoted Z, which are embedding vectors that encode reasoning information without requiring the generation of explicit tokens. The white-box LLM with parameters g is represented as Fg. and within the LLM Ty denotes the mapping converting sentences into LLM input embeddings. The lightweight implicit reasoning generator denoted as I with parameters vé. We use Co to represent our customized sentence transformer with parameters é, which evaluates semantic alignment between the ground- truth reasoning R and the implicit reasoning Z. The probability of the LLM generating token y; given preceding tokens 91:1-1, query Q, and implicit reasoning Z is denoted as P> (B. Z).
Problem 1. Efficient CoT with Implicit Reasoning. Given a query Q, we aim to minimize the total time T for an white-box LLM generate the implicil reasoning Z before achieving the answer Y, i.e. ming T(Q, Z, Y), subject to maximal retention of answer accuracy compared with the original COT.
3 Methodology
In this section, we first present an overview of the proposed framework SemCoT, followed by the detailed elaboration on its two steps performed sequentially: implicit v.s. ground-truth reasoning semantic alignment assessment and efficient implicit reasoning generation.
3.1 Overview
We introduce the workflow of our proposed framework SemCoT, as shown in Fig. SemCoT addresses two key challenges through a two-step process: The first step, implicit vs. ground-truft reasoning semantic aligument assessment, trains customized sentence transformer using contrastive learning. This sentence transformer assesses how semantically aligned the implicit reasoning is with the ground-truth reasoning, addressing our first challenge: the gap towards maintaining CoT effectiveness, by ensuring semantic alignment between implicit and ground- -ruthh reasoning. In the second step, efficient implicit reasoning generation, we finetune a lightweight implicit reasoning generator to produce implicit reasoning efficiently. This addresses our second challenge: enhancing CoT efficiency. The generator is optimized for two objectives simultaneously: (I) semantic alignment with explicit reasoning (guided by our trained sentence transformer) and (2) answer correctness when the L.M generates answers using implicit reasoning. In conclusion, SemCoT efficiently generates implicit reasoning that preserves the semanties of ground-truth reasoning, successfully addressing both challenges of maintaining effectiveness while improving efficiency.
? As the lightweight model may not be large language model, only call it language model.
3

---

Semontic emboddingg
Sam
Ground tru. are
Sectt Ans.
Bera
LLM x
LLM
Pooling & Linear project
Custom ag
Tokerize
Tokenize
Middle five tronsformer layers
Input query
Linear loyer
Input query
Linear loyer
Tokenize Ground- truth reasoning
Implicit reasonings generator
Lightweight LM (distilled/Shee) Sceeledd/eeeeeddd from LLM)
Lightweight LM (dishillledshhere from LLM)
LLM Tokeniiaation
Implicit reasoning generator
Condersed
Ground- truth Ponsorinng
reasening
Input query
scott
rooth
rcote
Input query
Step I: train custom, sentence frons.
Step II: train implicit reason. generator
Ploceholder
(a) Training stage
(b) Inference stage
Figure 2: Overview of the proposed SemCoT. Each cyan box is hidden text embedding within model components, with the text content and model type varying based the box's position in the figure. Fire and snowflake signs mean the component is trained and frozen, respectively.
3.2 Implicit vs. Ground-truth Reasoning Semantic Alignment Assessment
Motivation. As introduced in Section I existing methods struggle to fully capture ground-trute reasoning information in implicit reasoning. We address this challenge with natural assumption: optimal implicit CoT performance is achieved when the implicit CoT semantically aligns with the ground-trun -tutt CoT. This alignment occurs when the implicit Co',, if translated from the LLM's embedding space to human language, would be semantically equivalent the ground-trum -ruth reasoning. However, this "translation" presents a fundamental challenge: we cannot formulate it in explicit formulations due its complexity, nor can we model with neural networks because human language space is discrete and therefore not differentiable. As practical alternative, we measure this alignment within the L.MM's embedding space by comparing the input embedding of ground-truth reasoning (what we call "embedded ground-truth reasoning") with the implicit reasoning. Although one might consider directly using vector distance metrics to measure semantic alignment between implicit reasoning and embedded ground-truth reasoning, such approaches have significant limitations: (I) LLM embeddings are optimized for next-token prediction rather than sentence semantics, vector distance may not faithfully reveal semantic similarity between sentences 38]. (2) LLM embeddings are high dimensional and suffer from the curse of dimensionality [47), where distance metrics become
less meaningful as dimensionality increases [16].
Customized Sentence Transformer. Recent advances in sentence transformers 38 offer state-
of-the-art methods to measure semantic relationships between sentences. They utilize transformer- based architecture combined with pooling and linear layers to produce semantic vector representations of input sentences, enabling the measurement of sentence similarity via cosine similarity. Although traditional sentence transformers cannot be directly applied in our context-since implicit reasoning steps are not explicitly formulated sentences-they inspire our approach. A naive solution would involve bypassing the tokenization step of sentence transformers and directly feeding implicit or embedded ground-truth reasoning into these models. However, this approach is inadequate, as standard sentence transformers and the LLM we examine typically employ different mappings from tokens to input embeddings, resulting in embedding spaces that differ semantically. To address this issue, we design a customized sentence transformer specifically tailored to handle embeddings from I.Mss, thereby accurately measuring the semantic alignment between reasoning.
We design customized sentence transformer, shown in Fig. (a), denoted as C. (with parameters Φ), tailored to comparing semantics between implicit reasoning and ground-truth reasoning, First, we extract the middle five layers of the LLM that we perform CoT on to serve as the backbone of customized sentence transformer, these layers have been shown in prior work (26) to possess optimal language modeling ability and transferability across tasks. Next, we add a pooling layer on
"Impliit CoT performance" refers the LLM accuracy with the implicit CoT reasoning.

---

top of the transformer layers to aggregate token embeddings from the entire reasoning sequence into a unified vector representation. Finally, we employ linear layer to project this unified vector into a lower-dimensional semantic embedding space, facilitating more efficient similarity comparisons.
The customized sentence transformer is trained with specially crafted reasoning pair dataset g, where each data point (R, S) E G is a reasoning pair with At being the ground-truth reasoning and S GPT-4o-mini [29] generated condensed semantically aligned reasoning of R. The prompt to generate S with GPT--4-mini [29] is "Please generate the most condensed reasoning text which semantically aligned with the following reasoning text: R." We train the sentence transformer with a contrastive learning [22] strategy. Specifically, we compute the semantic embeddings of the ground truth reasoning (T= (R.)) and condensed reasoning (Tx (S:)). where Tx is the mapping from tokens to input embeddings of the LLM. The sentence transformer is trained using contrastive learning to maximize the similarity between embeddings of positive pairs (R,, S.), while minimizing similarities between all negative pairs (R;, S,) for * within the same minibatch. Formally, we learn the sentence transformer with
Sim
1 G
Σ (R
log
exp(simleR,
== exp(sim(eR,,) es, (s,)/T)
where ez, := (T=(() and es, == (T=( sim(er,, es, is the cosine similarity between these embeddings, and controls the distribution concentration over negative samples.
3.3 Token-level Efficient Implicit Reasoning Generation
Motivation. LLM typically requires significant time to generate even single implicit CoT token. Thus, we propose using lightweight language models to generate implicit reasoning efficiently. However, a key problem arises: the hidden embeddings of lightweight models do not naturally align with the embedding space of LLMs. This misalignment occurs for two reasons: (1) the embedding dimensions may differ, and (2) the semantic distributions between the embedding spaces of the two models can vary significantly. Nevertheless, we carefully select a compatible lightweight model whose semantic distribution resembles a linear projection of the L.M''s distribution, we can potentially learn a simple linear transformation layer to align their embedding spaces effectively.
Lightweight Implicit Reasoning Generator. We build the lightweight implicit reasoning generator, shown on the right in Fig. 2 (a), based on an off-the-shelf pruned or distilled LM (e.g., Sheared- LLaMA-1.30 (51) from the LLM (e.g.. Llama-2-7b-chatth 46)). We choose to use a pruned or distilled LMs of the original LLMS because existing research [45] 48 suggests that those models preserve crucial semantic properties of their original LLMs in their embedding spaces. Then, incorporate a linear projection layer to map the last-layer hidden embedding generated by the lightweight LM into the embedding space of the LLM to serve as implicit reasoning for the LLM.
More specifically, given query Q, we first append the query with & <CoT> tokens, with & being the length of the implicit reasoning. Here, the <CoT> token is special token added to the lightweight implicit reasoning generator's vocabulary. Then, the implicit reasoning generator, denoted as In with parameters vs, processes the concatenated text and collects the last-layer hidden embeddings of all the <CoT> tokens. Finally, we linearly transform the embeddings to obtain the generated implicit reasoning Z- I (Q). We learn to generate the ground-truth answer Y [M/,,... MN] (N is the length of the answer measured in tokens) with cross-entrops loss:
Epred (1b)
log Pr(vilnaa1,,0
N
Here, is the ith token of the ground-trutt answer, and (3 Z) is the LLM's probability of predicting yi when provided the query, implicit reasoning and all answer tokens before ya- In addition, we adopt a knowledge distillation training objective to encourage the implicit reasoning to be semantically aligned to the ground-truth reasoning. First, we tokenize the ground-truth reasoning R with the I.M and produce its input embedding TRRR). Then, we measure the semantic alignment between the embedded ground-truth and the implicit reasoning by computing their semantic embed- ding vectors generated by the well-trained sentence transformer Co. Formally, we train the implicit
5

---

reasoning with the following semantic alignment loss Lsem
1 D
Laem (v))
sim((o(TT=(RR))Cc222))
Therefore, we train the implicit reasoning generator with Lintat Xa (1 Xpeed where the À is hyperraaammtter controlling the degree to which semantic a alignment between the implicit and ground-truth reasoning is emphasized during the training of the implicit reasoning generator.
3.4 Training and Inference Strategy
Training Strategy. Based on our elaborations above, here we present a summary of the training strategy for the proposed SemCoT. For the first step (left-hand side of Fig. (a)). we optimize the parameters of the sentence transformer & with contrastive learning loss Lsim- For the second step (right-hand side of Fig. 2 (a)), we optimize the parameters of the implicit reasoning generator v with the aid of customized sentence transformer trained the first step. In this second step, the parameters of the sentence transformer á are frozen. The parameters of the implicit reasoning generator & is trained with loss Ltotal. which combines Cpred for enhancing answer accuracy and Leem semantically align the implicit reasoning with the ground-truth reasoning.
At each training step, we begin with warm-up phase that trains only the final linear layer (see Fig. (a)) of the component being updated - either Olinear for the customized sentence transformer in the first step, or Vinnnarr for the implicit reasoning generator in the second step.
Inference Strategy. Fig. (b) illustrates the inference strategy. During inference, given query, we first concatenate the query with & CoT tokens. The concatenated text is then processed by the implicit reasoning generator to generate implicit reasoning. Finally, we concatenate the implicit reasoning to the end of the query's input embedding and input into the LLM generate the final answer.
4 Experimental Evaluations
In this section, we first introduce the experiment setup. Then, we discuss the evaluation results of the SemCoT. Specifically, we aim to answer the following research questions: RQ1: How well can SemCoT improve the CoT reasoning efficiency and retain CoT effectiveness compared with the state-of-the-art efficient CoT baselines? RQ2: To what extent does each component of SemCoT contribute to the overall CoT reasoning performance? RQ3: How do hyperparameters such the number of implicit reasoning tokens affect SemCoT's performance? RQ4: Are there evidence to prove that SemCoT can learn semantically-aligee implicit reasoning while baselines cannot?
4.1 Experiment Settings
We introduce the experiment settings. More details (e.g., hardware information) are in Appendix
Datasets. We adopt five representative datasets from three different semantic domains used for benchmarking CoT performance. Specifically, we apply mathematical reasoning datasets GSM8K (5), SVAMP [6). MultiArith 4 (39], commonsense reasoning dataset CommonsenseQA [4], and symbolic reasoning dataset CoinFlip 21.. Please see Appendix for the metadata of the five used datasets.
Examined LLM & Implicit Reasoning Generator. We examine SemCoT on two representative open source LLMs, Llama-2-7br 2-7bcchtt-- [46] and Mistral-7B- Mistral-7BBIImttruttvO... [19]. For the lightweight LM within the implicit reasoning generator, utilize the distilled/sheared LMs from their corresponding examined LLMs. Specifically, in the cases that the LLM is Llama-2-77 2-b-c------ [46], we employ
Sheared-LLaMA-I.33 (5I) the lightweight LM within the implicit reasoning generator. When the
backbone LLM is
19 we apply 1istral-1.1b-testiine
[33] accordingly.
Baselines. We adopt state-of-the-ar baselines improving CoT efficiency LLMs. Specifically, (1) Pause 12 uses identical implicit reasoning tokens to substitute ground-truth reasoning. Their training strategy includes pretraining and finetuning the L.M for optimal answer accuracy with implicit reasoning. We only implement the finetuning strategy due to the significant cost of L.LM pretraining. (2) Progressive encoding methods, such as ICoT-SI 7 and COCONUT (53), use implicit

---

Table Performance of SemCoT vs. baselines across datasets (best results bold). Green highlight indicates the best values, and blue highlight indicates the runner-up up.
COCONUT
CODI
ICoT-SI
Pause
SoftCoT
SemCoTT (ours)
Acc (%)
77.67
97.33 2377
65.17 =2m
77.67 =:4:
58.33 2085
87.00
CoinFlip
Time (8)
1.58 knm
1.60 =om
1,47 +00)
1,50 kom
1.08 -0m
1.06 aum
Acc (%)
94.67 alas
93.83 23
98.00 4:4
88.67 1143
97.00 1006
98.33 1M0
Common
Time (8)
1.65 40m
1.66
1.46
1.20
1.01
1.06
Acc (%)
3.00 2071
5,00 2041
4.50 204)
4.33 x
8,83 LMB
9.83 29.34
Llama
GSM8K
Time (s)
1.60 auo
1.59 a
1.48 woo
1.46
1.05
1.02
Acc (%)
9.63 20.0
11.85 2005
8.70 1.331
11.85 =6.19
7.96 =0.00
15,93 L.14
MultiArith
Time (s)
1.62 2u04
1.58 200
1.48 was
1.45 2000
1.03 amm
0,94 2.111
Acc (%)
33.00
15.50
26.00 1ox
20.83
38.17
46.33
SVAMP
Time (s)
1.67 2..5
1.58 2001
1.48 000
1.48 200
1,06 20.00
1.00 20.13
Acc (%)
50.00 2400
66.67 247.4
41.83 #302
63,50 =324
49.17 20.40
82.17 23553
CoinFlip
Time (s)
1.75 4us
1.85
1.75 wos
1.70
1.31
1.30
Acc (%)
97.00 +00
99.17 20.7
99.33 6x2
99.67 +
98.50 1144
99.67 a.244
Common
Time (s)
1.38 au0
1.84 200
1.69 wws
1.67
1.18 2013
1.27
Acc (%)
5.83 1.80
6.00 +3:1
10.17 2.0
14.83 1m
12.17 -762
18,50 2.9
Mistral
GSM8K
Time (s)
1.93 2000
1.85 20000
1.69 2018
1.76 8000
1.35 20.0
1.35 20001
Acc (%)
3.33 aut
5.00
1.85 4
40.93
18.70 -091
38.89
MultiArith
Time (8)
1.88 20.0
1.83 20.00
1.73 2000
1.79 5002
1.32 =0.01
1.31 2:000
Acc (%)
45.17 119
13.50 20.71
54.17 2
48.83 1:
29.17 =270
53.83 3
SVAMP
Time (s)
1.84 ause
1.85
1.73
1.71
1.32 2010
1.30
tokens to substitute explicit tokens gradually during LLM finetuning. (3) CODI 41 adopts self- distillation strategy, where the teacher and student are both the LLM. The teacher learns to perform explicit reasoning, while the student learns to generate implicit tokens that lead to the correct answer. Simultaneously, both models are trained to align their token embeddings at the final token position of the query. (4) SoftCoT (53) utilizes small LM to generate LLM's hidden reasoning tokens without compressing the number of hidden tokens compared with the original explicit reasoning chain.
Evaluation Metries. All queries in the udopted datasets are accompanied by ground-truth answers. Therefore, we use answer accuracy as the metric to evaluate CoT effectiveness. This defined as the percentage of test queries for which the LLM,, when equipped with a given efficient CoT framework, successfully generates the correct ground-truth answer in response to the input question. For efficiency, we measure the average wall-clock time for the LLM to generate answer to query.
Implementation Details. We set the output embedding dimension of the customized sentence transformer to 768. The number of implicit tokens during training is five, and, during evaluation, it is set to one. We optimize both the customized sentence transformer and the implicit reasoning generator with AdamW (27) using the best hyperparameters found. For inference, we allow up to thirty answer tokens be generated to enforce the LLM to generate the answer instead of the CoT.
4.2 Effectiveness & Efficiency of SemCoT
In this subsection, we aim to answer RQ1. Specifically, we evaluate our proposed framework SemCoT on two LLMs, 2abch-2-7b (46) and -airral----nstuutt--0.... [19]. We compare the two LLMs' answer accuracy and inference time cost when performing CoT with our SemCoT and the state-of-the-art baselines using one implicit reasoning token. We show the results in Table From Table [ we can make the following observations: (1) from the perspective of effectiveness, our SemCoT achieves the highest answer accuracy compared with the baseline methods in most datasets, showing that our method effectively retains LLM's 's reasoning ability in a wide spectrum of NLP tasks and LLM settings. (2) From the perspective of efficiency, SemCoT achieves nearly the fastest implicit reasoning processing time across all datasets and LLMss. In some datasets, SemCoT is slightly slower than SoftCoT, but SemCoT achieves much higher answer accuracy those cases. In conclusion, SemCoT achieves the optimal performance w.r.t. the trade-off between efficiency and effectiveness.
7

---

CoinFlip
SVAMP
MultiArith
+14.2%
SemCoI-NSA
+13.5%
SemCoI-NST
+51.3%
SemCoT-NLL
100
SemCoTT
100
SemCoT
100
SemCoT
Accuracy (%)
+8.2%
+12.6%
+8.6%
50
50-
50-
+16.2%
+7.5%
+50,9%
CoinFlip
SVAMP
MultiArith
CoinFlip
SVAMP
MultiArith
CoinFlip
SVAMP
MultiArith
(a) S SemCoT-NSA
(b) SemCoT SemCoT-NST
(c) SemCoT
SeMCOT-NLL
Figure 3: Ablation study of SemCoT.
Accuracy
Inference Time
1.6
1.6
45
1.5
Inference Time
Accuracy (%)
40
1.5
Inference Time
Accuracy
1.4
30
1.4
40
1.3
20
1.3
1.2
10
1.2
1.1
0.1
0.3
0.5
0.7
Implicit Tokens
(a)
(b)
Figure Parameter sensitivity of SemCoT.
4.3
Ablation Study
We aim to answer RQ2 by evaluating the effect of different components in SemCoT with its three variants: (1) SemCoT-NST: we remove the customized sentence transformer and modify the semantic alignment loss as the cosine similarity between the mean-pooled embeddings of the implicit reasoning tokens and the ground-truth reasoning: (2) SemCoT-NSA. we remove the entire semantic alignment loss when training the implicit reasoning generator; (3) SemCoT-
replace the lightweight language model in the implicit reasoning generator with the original LLM and finetune the LLM using LoRA 15 to generate implicit reasoning. We conduct experiments with LLama-2--B-chaa on various datasets and observe from Fig. that (1) Generally, SemCoT's performance decrease when any component is removed, showing that each component of the SemCoT positively contributes to the CoT performance; (2) Comparing Fig.3 (a) and Fig. (b), we find that removing the semantic alignment loss results in worse performance compared to that of optimizing with semantic alignment measured with cosine similarity, highlighting the significance of the semantic alignment loss in learning implicit reasoning; (3) From Fig. (c), surprisingly find that finetuning the original LLM to generate the implicit reasoning less effective than utilizing the lightweight LMs, likely attributed catastrophic forgetting induced by LLM finetuning [53].
46
4.4 Parameter Sensitivity
We answer RQ3 by studying the fluctuation of SemCoT performance w.r.t. the change of hyper- parameter X. Here, À controls the weight of the semantic alignment loss when training the implicit reasoning generator. More specifically, we vary À in {0.1, 0.3, 0.5, 0.7, 0.9), and we present the corresponding performance differences of SemCoT when applied to -lama-2-7b-cha---
(46) on
SVAMP 36] In addition, we also investigate the effect of the number of implicit reasoning tokens M when LLM performs inference. We vary its value among M 21, 2, 3, 4,5) and record the performance of SemCoT using the same setting as when evaluating A. The results are shown in Fig. 4 We make the following observations: (1) The LLM answer accuracy generally increases except for when À is 0.5, potentially due to the delicate balance between semantic alignment and prediction accuracy. This is also indicated by the unusual large variance. The inference time is consistent under different X. (2) The LLM answer accuracy decreases as the number of implicit tokens increase,
8

---

SemCoT
Coconut
30
20
10
20
10
10
10
40
-10 20
20
-10 -20
20
40
20
20
40
Figure 5: Case study comparing SemCoT vs COCONUT. PCA plots of implicit reasoning embeddings for 3 SVAMP queries with 20 semantic variants each. SemCoT (blue) has tighter clustering than COCONUT (orange), showing its ability to generate semantic aligned reasoning.
indicating that the implicit reasoning can concisely embed ground reasoning information. We observe similar findings in other datasets well.
4.5 Case Study
We answer RQ4 through a case study by comparing the implicit reasoning of SemCoT with the base- lines. Since directly decoding implicit reasoning to human language is challenging (see Section 3.2]. we adopt a proxy evaluation based on this assumption: if an implicit CoT method effectively generates semantically-
implicit reasoning from ground-trum reasoning, it should produce consistent im- plicit reasoning when processing semantically aligned queries. We thus randomly select three queries from the SVAMP [36] dataset and use GPT--4-mini [29] to generate 20 semantically equivalent
variants for each query. We visualize (with PCA 10] the first implicit reasoning tokens produced by both SemCoT (blue) and the baseline COCONUT (orange) when processing these variants (see case studies for other baseline methods in Appendix D.3). As shown Fig. [5] the implicit reasoning generated by SemCoT is significantly more concentrated compared to that of COCONUT. The tight clustering suggests that SemCoT successfully distills essential reasoning information regardless of surface-level linguistic differences in queries.
5 Related Work
Efficient CoT with Implicit Reasoning. Stepwise or progressive encoding methods encode ground- truth reasoning implicitly by first familiarizing the LLM with the reasoning and then removing it, expecting the LLM to learn hidden embeddings that capture previously learned information 14 40). These techniques are indirect and susceptible noise. Other methods compress parts of the reasoning sequence, with CCoT 3 focusing on keywords and Token Assorted [43] selecting parts through a specially trained neural network. SoftCoT 53 generates the implicit reasoning with small language model and only trains a linear layer to transform the implicit reasoning of the small LM the LLM for answer correctness, completely bypassing ground-trui reasoning. CODI employs self-distillation where the teacher model learns ground-truth reasoning while the student learns to generate correct answers with implicit reasoning, the teacher distills LLM embedding of the last query token to the student in order to encode ground-truft reasoning semantics to the student. This approach lacks verification that the final token fully encodes reasoning semantics. These methods barely consider token-level reasoning generation speed and reasoning semantic aligmment.
Semantic Textual Similarity. Semantic Textual Similarity (STS) has evolved from word-level lexical 24
30 approaches to word-level semantic [3i 37 methods, and finally to sentence-leve semantic 38 I techniques. Early methods relied on lexical overlap and WordNet-basee mea- sures, often struggling with synonymy and polysemy (24). Word-level semantic embeddings like Word2Vec 31 and GloVe 37 marked a significant advancement, with embeddings typically ag- gregated to create comparable sentence representations. Recent specialized neural architectures focus on sentence-level representations, including Sentence-BERRT (SBERT) (38), which modified BERT with siamese networks to derive semantically meaningful sentence embeddings that can be

---

compared using cosine similarity. Similar approaches include Universal Sentence Encoder [ and InferSent [6], which share the goal of creating fixed-length sentence representations. Recent works improve embedding space uniformity through contrastive learning strategies
[9 56 23 However,
these STS methods cannot compare semantics between L.MMs' embedding and natural language text.
Conclusion
In this paper, we introduce SemCoT, novel framework that accelerates Chain-of-Thouglle reasoning while preserving semantic alignment between implicit and explicit reasoning. Our approach addresses two key challenges in implicit CoT methods: maintaining semantic alignment with ground-truth reasoning and optimizing token-level generation speed. Extensive experiments across multiple datasets and LLMs demonstrate that SemCoT achieves superior performance both efficiency and effectiveness compared to state-of state-of-the-rt thehhhr baselines. SemCoT represents significant advancement in making advanced reasoning capabilities more computationally accessible for real-world applications.
7 Acknowledgments
This work is supported in part by the National Science Foundation (NSF) under grants IIS-2006844, IIS-2144209, IIS-2223769, 1IS-2331315, CNS-2154962, BCS-2228534, and CMMI-2411244 the Office of Naval Research (ONR) under grant N000142412636, and the Commonwealth Cyber Initiative (CCI) under grant VV-1Q25-004.
References
[1] D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, S. John, N. Constant, M. Guajardo- Cespedes, S. Yuan, C. Tar, et al. Universal sentence encoder for english. In EMNLP, 2018.
[2] D. Chandrasekaran and V. Mago. Evolution of semantic similarity survey. ACM Comput. Surv, 2021.
[3] J. Cheng and B. Van Durme. Compressed chain of thought: Efficient reasoning through dense representations. arXiv, 2024.
[4] ChilleD. Multiarith dataset. Hugging Face Dataset Repository, 2023.
[5] K. Cobbe, Kosaraju, M. Bavarian, M. Chen, Jun, L. Kaiser, M. Plappert, Tworek,
J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv, 2021.
[6] A. Conneau, D. Kiela, Schwenk, L. Barrault, and A. Bordes. Supervised learning of universal sentence representations from natural language inference data. In EMNLP, 2017.
[7] Y. Deng, Y. Choi, and M. Shieber. From explicit cot implicit cot: Learning internalize cot step by step. CoRR, 2024.
[8] Y. Deng, K. Prasad, R. Fernandez, P. Smolensky, V. Chaudhary, and S. M. Shieber. Implicit chain of thought reasoming via knowledge distillation. CoRR, 2023.
[9] T. Gao, X. Yao, and D. Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP, 2021.
[10] F. L. Gewers, G. R. Ferreira, H. F. D. Aruda, F. N. Silva, C. H. Comin, D. R. Amancio, and L. d. F. Costa. Principal component analysis: natural approach to data exploration. ACM Comput. Surv. 2021.
[11] J. Gou, B. Yu, S. J. Maybank, and Tao, Knowledge distillation: A survey. Int. J. Comput. Vis., 2021.
[12] S. Goyal, 7.. Ji, A. S. Rawat, A. K. Menon, S. Kumar, and V. Nagarajan. Think before you speak: Training language models with pause tokens. In ICLR, 2024.
10

---

[13] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv, 2025.
[14] S. Hao, S. Sukhbaatar, Su, X. Li, Z. Hu, Weston, and Y. Tian. Training large language models to reason in a continuous latent space. arXiv, 2024.
[15] E. J. Hu, Y. Shen, Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 2022.
[16] Y. Huang, J. Xu, J. Lai, Z. Jiang, T. Chen, Z. Li, Y. Yao, X. Ma, L. Yang, H. Chen, al. Advancing transformer architecture in long-context large language models: A comprehensive survey, arXiv, 2023.
[17] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al. Gpt-4o system card. arXiv, 2024.
[18] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. EI-Kishky, A. Low, A. Helyar, A. Madry,
A. Beutel, A. Carney, et al. Openai ol system card. arXiv, 2024.
[19] A. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. Chaplot, D. las Casas, Bressand,
G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023.
[20] Z. Jie, T. Q. Luong. X. Zhang, X. Jin, and H. Li. Design of chain- of---ouutthhhtttt in math problem solving. arXiv, 2023.
[21] S. Krishna, J. Ma, D. Slack, A. Ghandeharioun, S. Singh, and H. Lakkaraju. Post hoc expla- nations of language models can improve language models. Advances in Neural Information Processing Systems, 36:65468-6544
2023.
[22] H. Le-Khac, G. Healy, and A. F. Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 2020.
[23] X. Li and J. Li. Aoe: Angle-optimized embeddings for semantic textual similarity. In ACL, 2024.
[24] Y. Li, D. McLean, Z. A. Bandar, J. D. O'shea, and K. Crockett. Sentence similarity based on semantic nets and corpus statistics. IEEE transactions on knowledge and data engineering, 2006.
[25] H. Liu, C. Sferrazza, and Abbeel. Chain of hindsight aligns language models with feedback. In ICLR, 2024.
[26] N. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith. Linguistic knowledge and transferability of contextual representations. In NAACL-IILT, 2019.
[27] Loshchilov and F. Hutter. Decoupled weight decay regularization ICLR, 2019.
[28] Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, and C. Kallison-Burchh Faithful thain-of-thought reasoning. IJCNLP-AACL, 2023.
[29] J. Menick, K. Lu, S. Zhao, E. Wallace, H. Ren, H. Hu, N. Stathas, and F. P. Such. Gpt-4o mini: advancing cost-efficient intelligence. Open AI: San Francisco, CA, USA, 2024.
[30] R. Mihalcea, C. Corley, C. Strapparava, et al. Corpus-based and knowledge-bate measures of text semantic similarity. In AAAI, 2006.
[31] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. NeurIPS, 2013.
[32] Miner, Y. Takashima, Han, Kouteili, Erata, R. Piskac, and S. J. Shapiro. Scheherazade: Evaluating thain-of-thought math reasoning in 1lms with ohain-of-problenee arXiv, 2024.
11

---

[33] Optimum Team. Mistral-L.1b-tesiingg 1b- testing 2024. Accessed: 2025-04-28.
[34] L. Parisi. How to run DeepSeek-RI IQI_S 1.58bit at 140 token/sec. GitHub Issue, 2025. Issue #1591 in unslothai/unsloth repositor..
[35] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin,
N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
[36] A. Patel, S. Bhattamishra, and N. Goyal. Are NL. models really able solve simple math word problems? In NAACL, 2021.
[37] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP. 2014.
[38] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese nett-nesworks. In EMNLP-IJCNLP, 2019.
[39] S. Roy and D. Roth. Solving general arithmetic word problems. In EMNLP, 2015.
[40] X. Shen, Y. Wang, X. Shi, Y. Wang, P. Zhao, and J. Gu. Efficient reasoning with hidden thinking. arXiv, 2025.
[41] Z. Shen, H. Yan, L. Zhang, Z. Hu, Y. Du, and Y. He. Codi: Compressing chain-of-thought into continuous space via self-distillation. arXiv, 2025.
[42] Z. R. Sprague, F. Yin, J. D. Rodriguez, D. Jiang, M. Wadhwa, P. Singhal, X. Zhao, X. Ye, K. Mahowald, and G. Durrett. To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning. In ICLR, 2025.
[43] D. Su, H. Zhu, Y. Xu, Jiao, Y. Tian, and Q. Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv, 2025.
[44] A. Talmor, J. Herzig, N. Lourie, and J. Berant. Commonsensequ: A question answering challenge targeting commonsense knowledge. In NAACL, 2019.
[45] C. Tao, T. Shen, S. Gao, J. Zhang, Z. Li, Z. Tao, and S. Ma. Lims are also effective embedding models: An in-depth overview. arXiv,, 2024.
[46] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023.
[47] M. Verleysen and D. François. The curse of dimensionality in data mining and time series prediction. In IWANN, 2005.
[48] Q Wang, M. J. Zaki, G. Kollias, and V. Kalantzis. Multi-sense embeddings for language models and knowledge distillation. arXiv, 2025.
[49] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. Ve, D. Zhou, et al. Chain-of- thought prompting elicits reasoning in large language models. NeurIPS, 2022.
[50] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Transformers: State-of-the-art natural language processing. In EMNLP, 2020.
[51] M. Xia, T. Gao, Z. Zeng, and D. Chen. Sheared llama: Accelerating language model pre-training via structured pruning. In ICLR, 2024.
[52] J. Xu, H. Fei, L. Pan, Q. Liu, M.-L. Lee, and W. Hsu. Faithful logical reasoning via symbolic chain-of-thougee In ACL, 2024.
[53] Y. Xu, X. Guo, Z. Zeng, and C. Miao. Softcot: Soft chain-of-thought for efficient reasoning with Mms. CoRR, 2025.
12

---

[54] P. Yu. T. Wang, O. Golovneva, B. AlKhamissi, S. Verma, Z. Jin. G. Ghosh, M. Diab, and A. Celikyilmaz. Alert: Adapt language models to reasoning tasks. In ACL, 2023.
[55] B. Zhang, E. Mitchell, Ren, K. Lu, M. Schwarzer, M. Pokrass, S. Zhao, T. Sanders, A. Kalai, A. Passos, B. Sokolowsky, E. Y. Le, E. Ritter, H. Sheng, H. Wang, I. Kostrikov, J. Lee, J. Ferstad, M. Lampe, P. Radhakrishnan, S. Fitzgerald, S. Bubeck, Y. Dubois, Y. Bai, A. Apple- baum, E. Prochl, E. Mays. J. Parish. K. Liu, L. Maksin, L. Ho. M. Wang. M. Wang. O. Watkins, Chao, S. Miserendino, T. Patwardhan, A. Woodford, B. Hoover, J. Brill, K. Stirman, N. Aj- jarapu, N. Turley, N. Handa, O. Godement, A. Nathan, A. Huang, A. Wang. A. Gohel, B. Eggers, B. Yu, B. Ashley, C. Huang, D. Bogan, E. Sokolova, E. Horacek, F. P. Such, J. Cohen, J. Gross, J. Becker, K. Wu, L.. Lv, L. Byron, M. Liodakis, M. Johnson, M. Trpeic, M. Yesildal, R. Ry- gaard, R. Marsan, R. Ram-chandani, R. Kshirsagar, S. Conlon, T. Xia, S. Fu, Narayanan, S. Choudhry, T. Kaftan, T. Creech, A. Vallone, A. Duberstein, E. Sert, Wallace, G. Zhao,
I. Kofman, J. Yu, J. Q. Candela, M. laine Boyd, M. A. Yatbaz, M. McClay, M. Wang. S. Agarwal, S. Jain, S. Toizer, S. Hernández, S. Mostovoy, T. Li, Y. Cha, Y. Wang, L. Ahmad, T. Peterson,
C. Chang. K. Ying. A. Clark, D. Stuckey, J. Tworek, J. W. Pachocki, J. Heidecke, K. Weil,
L. Fedus, M. Chen, Altman, and W. Zaremba. Openai 03-mini system card.
[56] W. Zhuo, Y. Sun, X. Wang. L. Zhu, and Y. Yang. Whitenedcse: Whitening- based contrastive learning of sentence embeddings. In ACL, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes].
Justification: The abstract and introduction accurately reflect the paper's contributions and scope. The paper clearly states its claim to address key challenges in Chain-of- Thought (CoT) reasoning: preserving semantic alignment between implicit and explicit reasoning. and optimizing token-level generation speed. These claims are consistent with the results presented throughout the paper.
Chaun-of-Thought
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification:
A dedicated section of limitations is provided in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
Justification: The paper does not include formal theoretical results requiring mathematical
proofs. The methodology is explained algorithmically and empirically rather than through formal theorems.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes].
The paper provides sufficient information on the datasets used (Section 4.1), model architectures, training procedures, and implementation details (Section 4.1). The hyperparameters and training details are specified in the Implementation Details subsection, and the code is available the URL provided in the abstract.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes].
Justification: The code available "https:/anonymous. wwwp:/oouuoonosoonnnoeoeeeeeeeee Aopenn.cieencc///eemC""""" Addi- tionally, we use standard, publicly available datasets for evaluation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctipss.cc/pulli//gue
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: The paper specifies all the necessary training and test details in Section 4.1, including dataset splits, model architectures, hyperparameters, optimizer settings, and implementation frameworks used (PyTorch and Huggingface).
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: The paper report error bars for the experimental results. The tables and figures present results with standard deviations across multiple runs.
16

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: The paper provide information about the computing resources used conduct the experiments, such as the type of GPUs/CPUs, memory requirements in Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics stttps://neurr: cwtturrsssccppulcc///eee (eppbllc//thicssuudeellll..
Answer: [Yes]..
Justification: The research conform with the NeurIPS Code Ethics. The work focuses improving efficiency and effectiveness of language models without apparent ethical concerns. No privacy-sensitie data used, and the methods don't present obvious risks of harm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
17

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
It is discussed in Broader Impact in Appendix B.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]]
The paper does not present high-risk models datasets that would require safeguards against misuse. The work focuses on an algorithmic improvement for reasoning efficiency rather than releasing potentially harmful content- geneentingn models.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
18

---

The paper properly cites the original sources for the datasets used (GSM8K, SVAMP, MultiArith, CommonsenseQAA and CoinFlip) and the models (Llama-2-7b-che hf, 1-ad77-----ntuu--O.... Sheared-I.LaMA- mistral-1. 1errral1.1b--eee lb-testing) with appropriate references. The licenses are provided in Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer:[Yes]]
Justification:
The assets are the provided link in the abstract.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: The paper does not involve crowdsourcing or research with human sub- jects. The evaluation conducted existing benchmark datasets rather than new human- annotated data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA)
Justification: Since the research does not involve human subjects, IRB approval was not required for this work.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [Yes].
Justification: LLMs are the research objective of the work, since this work aims to improve the LLMs' reasoning ability.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /tttps://nuurjin CwwearipsscccCCoofeeee / /LLM) for what should or should be described.
20

---

Appendix
A
Limitations
While SemCoT demonstrates promising results in improving both the efficiency and effectiveness of Chain-of-Thoughe reasoning, we acknowledge certain limitations in our current approach. The customized sentence transformer, although effective, requires additional training overhead before deployment, which might be challenging for resource-constrrined environments. Furthermore, our evaluation primarily focuses on standard reasoning benchmarks (mathematical, commonsense, and symbolic reasoning); hence, the performance on more specialized domains or extremely long-chain reasoning tasks remains be explored. Further investigation would benefit the generalizability across different language model architectures beyond the tested ones (I (Ilama-2 and Mistral). Additionally, while we observed consistent performance improvements, there may be specific reasoning patterns where the trade-off between implicit and explicit reasoning is less favorable. Lastly, we acknowledge that implicit reasoning, due to its form of latent token embeddings, reduces the human interpretability of the LLMs' reasoning process. Future work could address these limitations by expanding the evalu- ation scope, exploring more efficient pre-training strategies for the customized sentence transformer component, and training a specialized implicit reasoning decoder to decode the implicit reasoning generated by the contemplation generator.
B Broader Impact
Our work on SemCoT has several potential positive societal impacts. By improving the efficiency of Chain-of-Thouge reasoning in LLMs, we reduce computational costs and energy consumption, leading lower carbon footprints for Al deployments. This efficiency also facilitates broader access to advanced reasoning capabilities in sesource-consrriigg environments such as mobile devices or underdeveloped regions. Thus, more efficient reasoning enables more applications in time- sensitive domains such as healthcare decision support and emergency response systems. However, we also recognize potential negative impacts. As reasoning becomes more efficient, malicious actors could deploy sophisticated reasoning systems at scale for generating misinformation or conducting automated cyberattacks. The improved efficiency might accelerate the deployment of Al systems before adequate safety measures are established. Furthermore, there is a risk that optimizing for computational efficiency might inadvertently prioritize speed over reasoning quality in specific contexts, potentially leading to errors in critical applications if not properly monitored. We encourage addressing these concerns through responsible deployment practices.
C Implementation Details
Dataset Metadata. We show the metadata of the datasets, including the size of train and test sets, along with the reasoning type, in Table For sample data points from each dataset, see Table
Table 2: Sample questions and answers from benchmark datasets
Dataset
Sample Question
Ans. GT-reason
(Continued on next page)
21

---

(Continued from previous page)
Dataset GSM8K
Sample Question
Ans. GT-reason
Janet's ducks lay eggs per day. She
Janet sells -- 1--4=<16--4=awp/ duck eggs day. She makes 9 2 ==9=2=1881 every day at
eats three for break- fast every morning and bakes muffins for her friends every day with four. She sells the remainder the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farm- ers' market?
the farmer's market.
SVAMP
There are 87 oranges and 290 bananas in Philip''s collection. If the bananas are organized into 2 groups and oranges organized into 93 groups. How big is each group of bananas?
145
The question is in type of Common-Diviin and we solve it by calculating / 2.0)
MultiArith
There are 64 students trying out for the school's trivia teams. If 36 of them didn't get picked for the team and the rest were put into 4 groups, how many students would be in each group?
To solve the problem step-by-step, we will follow these steps:1. **Determine the number of students who were picked for the team. **2. **Calculate how many students are in each group. group.**#H* **### Step Calculate the number of students picked for the team We know that there are total of 64 students trying out for the trivia teams, and 36 of them did not get picked. To find out how many students were picked, we subtract the number of students who didn't 't get picked from the total number of students:[ textNumber of students picked textTo- tal students textStudents not picked]Subbtituure the values:[ textNumber of students picked = 36]Calculating this fivessteexNuumbe of students picked 28] Step 2: Calculate how many stu- dents are in each group Now that we know there are 28 students who were picked for the team, we need to divide these students into groups. To find out how many students will be in each group, we di- vide the number of students picked of groups: [textStudents per group = fractextNum- ber of students pickedtextNumber of groups] Sub- stituting the values: per group Caae844Jalculating this cives:textttuddett per group = 7]### Conclusion Each group will have **7 students**."
the number
(Continued next page)
22

---

(Continued from previous page)
Dataset
Sample Question
Ans.
GT-reeson
Commonsense.
The sanctions against
A
To solve this problem, we need to analyze con- text of the sentence and the meaning of each an- swer choice. The sentence states that The sanctions against the school were punishing blow, and they seemed to what the efforts the school had made to change. The key part of the sentence is the phrase Seemed to, which indicates that we are looking for verb that describes how the sanctions relate to the school's 's efforts to change. let's evaluate each option:A. ** **ignore** This suggests that the sanctions did not acknowledge or take into account the school's efforts to change. This could make sense in the context, as sanctions might undermine overlook positive changes. **enforce*** This implies that the sanctions would reinforce or sup- port the efforts the school made change. This does not fit well because sanctions are typically punitive and would not support positive efforts. *authoritarian*** This is an adjective describing a style of governance grammatically in the sentence as it does not com- plete the phrase Seemed to. **yell at This a colloquial expression that implies verbal repri- mand. It does not fit the context of sanctions, which formal penalties rather than verbal actions.E #*avoid** This suggests that the sanctions would lead to evading or sidestepping the efforts made by the school. This does not fit well either, as sanctions more about punishment than avoid- ance. ance.Given the analysis. option A (ignore) the appropriate choice. It indicates that the tions overlooked or disregarded the school's efforts improve, which aligns with the idea of sanctions being punishing blow. Therefore, the answer is ** A**
the school were pun- ishing blow, and they seemed to what the ef- forts the school had made change? "la- bel": ["A", "B", "C", "D", "E" "text": [ ("ignore", "enforce", "authoritarian", "yell at", "avoid"
control. It does fit
CoinFlip
A coin is heads up. sager does not flip the coin. zyheir flips the coin. the coin still heads up?
no
Let's track the state of the coin step by step:1. ** *nitial State*#; The coin heads up.2. **Sager does not flip the coin The state remains heads up.3. **Zyheir flips the coin*#; Flipping the coin changes its state from heads up to tails Ap. the end these actions, the state of the coin is tails up. Therefore, the answer is #* the coin is not still heads up.
ModdentLicensed of Existing Assets. For the two adopted L.Mss, Llama-2-7b-chat-hff [46] has "Llama 2 Community License Agreement," and -inrall----Inttuctt---- 19 has the Apache 2.0 license. For the implementation of the lightweight implicit reasoning generator, both Sheared-LLaMA- and Histral-II.I--estti [33] have the Apache 2.0 license. Please see the licenses of the datasets in Table
[51)
Hardware Information. We perform all experiments on multiple machines with NVIDIA H100 80GB GPUs running CUDA 12.4.
Hyperparameters Setting. Our SemCoT implemented with PyTorch 35 and Huggingface (50 training pipeline. We list the hyperparameters settings in the GitHub repository (found in utils/utils.py) Table shows the average accuracy and time measurements over three independent rounds for each method. During training, the baselines and SemCoT are allotted to five implicit reasoning tokens.
23

---

Table Metadata for benchmark datasets
Dataset
Train Size
Test Size
Reasoning Type
GSM8K S
7,500
1,000
Arithmetic
SVAMP
36
700
300
Arithmetic
MultiArith
39
420
180
Arithmetic
Commonsense 44
9,741
1,140
Commonsense
CoinFlip 21
20,000
3,330
Symbolic
Table 4: License Information for Hugging Face Datasets
Dataset
License Information
SVAMP
MIT License
MultiArith
CC BY 4.0
Commonsense@A
MIT License
CoinFlip
MIT License
GSM8K
MIT License
For evaluation, they are limited to only one reasoning token. This ensures fair comparison across methods, as allowing only one token eliminates the possibility confounding factors, such as directly providing the answer instead of the reasoning and excessively encoding model knowledge.
Text-paired Dataset. We show examples of the input and reasoning pairs datasets in Table
D Supplementary Experiments
D.1 Ablation Study
In this section, we show the supplementary experiment results for the ablation study. Specifically, we
adopt the variants of the SemCoT designed in the Section 4.3 in the main paper and examine their
performance on all datasets (GSM8K [5). SVAMP (36), MultiArith 4 33], CommonsenseeAA [4].
and CoinFlip (21) on both LLMs (Llama-22 2-7b-chat-h [46] and Mistral-7E 7
[9 The
results for
[46] and Mistral- 7B Instruct-vO.2 [19] are shown in the Fig 6 and
Fig. respectievly. From the two figures, we observe that our SemCoT performs better than all
variants in almost all experiment configurations (i.e., the composition of datasets and LLMs), and the three observations from Section also hold.
CoinTlip
SVAMP
GSMBK
MaltArith
CommonsessQQ
100
Tm SemCoT SomCoTNSA SimCuTNNST SenCotiNLL
80
Accuncy (%)
17.0%
15.2%
CoinFlip
SVAMP
GSMBK
MultiArith
Commonsensse@A
Figure Ablation study results on SemCoT compared to its three variants on all five adopted datasets in the Section 4.1 of the paper.
46
We show the performance of our

---

Table 5: Text samples from the text- text-paired datasets for each dataset
Dataset ComFlip
Full Reasoning
Condensed Reasoning
Let's track the state of the coin step by step:1. Initially, the coin is heads up. 2. Mailey does not flip the coin, so the state of the coin remains heads up. 3. Maurisa does not flip the coin, so the state of the coin remains heads up. At the end of these steps, the coin is still heads up. Final answer: yes.
Coin remains heads up; nei- ther Mailey nor Maurisa flipped it.
Common
To determine where Southern Europe you would find many canals, let's analyze each of the answer choices:\nínA. ** **ichigan** -hhs a state in the United States, not in Southern Europe. Therefore, it is not suit- able answer.'nnnl **New York** This is a state in the United States, specifically in the northeastern part of the country. Like Michigan, it is not in Southern Europe, so this option is also not appropriate. AninC. ** *Amsterdam*** While Amsterdam is known for its extensive canal sys- tem, it is located in the Netherlands, which is in Northern Europe, not Southern Europe. Thus, this option does not fit the criteria.\nInD. **Venice***.... Venice is city in Italy, which is located in Southern Europe. It is famous for its canals, making this option a strong candidate for the answer An **Bridge** This is general term and does not refer to a specific location. While bridges may be found in areas with canals, it does not answer the ques- tion of where Southern Europe you would find many canals. An'nAfttr analyzing all the options, the only one that meets the criteria of being in Southern Europe and known for its canals <>:nnnn*DD Venice** \n'nThus, the correct answer D. In#### D
Venice, in Southern Europe, is famous for its canals.
GSM8K
Natalia sold 48/2 = 24 24 clips in May, AnNatalia sold 48+24 248+24=72w2 clips altogether in April and May.
Natalia sold 48 clips in April, 24 in May; total 72.
MultiArith
To determine how much money the magician earned from selling magic card decks, we can follow these steps:lnln#ft## Step 1: Determine the number of decks shed'nThh magician started with 5 decks and ended with decks. To find out how many decks he sold, we sub- tract the number of decks he has left from the number sold = VVtextInitial decks sold = 3 2\nW\/ln\n#### Step 2: Calculate the total earnings from the deeks soldhnEach deck was sold for 2 dollars. To find out how much money he earned from selling the decks, we multiply the number decks sold by the price per
Magician sold 2 decks for $4; earnings totaled $4.
started with:Annnnk/nnnltxxxDDee.ne VtrextRemaining deckktnnl/nMM/uiikeexDDDDee
earnings WtextDecks sold Vtimes Wexxtrrice per deckinn/jluiNN(ulteettte
carnings Vttimes 2 = 4hnlWhnnnut#H Conclusion'uThe magician earned a total of 4 dollars from selling the magic card earnings 4 Mtext dollars\n\I The question is in type of Common-Divisio and we solve by calculating 2900. 2.0)
SVAMP
Divide 290 bananas by 2 groups find group size.
25

---

CoinTTip
SVAMP
GSMRK
MilaArih
CommonsunseQA
100
SimCuT
11.25
SuMCoT
SomCoT-NST
SenCot-NLL
Avcurcy (%)
10.75
1425 12.35
20
CoinFlip
SVAMP
MaltiArith
CommonsenseeAA
Figure 7: Ablation study results on Biirra-7--Instruct-ve [9]. We show the performance of our SemCoT compared to its three variants on all five adopted datasets in the Section 4.1 of the paper.
Accuracy
Inference Time
Commonsense Qn
Multiarihh
Coin Flip
1.000
0.20
1.2
(s)
1.4
inference Time
1.0
Accuracy
Accuracy
0.15
0.975
1.25
0.950
0.10
1.0
0.8
0.925
0.05
0.8
1.20
1.0
3
2
3
2
3
# Implicit Tokens (a)
#
Implicit Tokens (b)
# Implicit Tokens
(c)
GSMBK
Svamp
Inference Time (8)
0.10
1.25
Inference Time
0.5
1.4
Acouracy
Accuracy
0.4
0.05
1.20
0.3
1.2
1.15
0.2
2
3
5
1
2
3
4
5
# Implicit Tokens
# Implicit Tokens
(d)
(e)
Figure 8: Parameter analysis experiment results for -lama-2-7b-chat-l 46)
The accuracy and
inference time of our method using Llama when varying the number of implicit tokens
D.2 Parameter Analysis
Here, we display the supplementary experiment results for parameter analysis. Specifically, we follow the design of Section 4.4 to examine the number of implicit tokens utilized during evaluation. We conduct experiments on all adopted datasets and LLMs; the results for Llama-2-7b- 2-7bc--ttttt- [46] and Mistral-7B -B Instruct-vO.2 [19] shown in Fig. g and Fig, 9 respectively. We find generally consistent observations in Section 4.4 of the main paper.
D.3 Case Study: Semantic Alignment Analysis
conduct extensive experiments to show that our method helps maintain the semantic alignment between ground truth and implicit reasoning. As declared in Section 4.5 we randomly pick three samples from each dataset and generate semantically aligned queries for each query twenty times.
Then design the experiments show that our SemCoT achieves aamantically-alinnee reasoning
26

---

Accuracy
Inference Time
Commmnnennse Qa
Multiarthh
Coin Flip
0.6
1.0
1.6
Inference Time (8)
Inference Time (8]
1.00
1.6
1.55
Accuracy
Accuracy
0.5
Acouracy
0.9
1.50
0.98
1.5
0.4
1.45
0.8
0.96
1.4
0.3
1.40
1.3
2
3
4
5
2
3
4
5
1
2
3
5
# Implicit Tokens (a)
# Implicit Tokens
# Implicit Tokens
(b)
(c)
GSM8K
Svamp
1.60
Accuracy
0.20
1.50
Inference Time
Accuracy
0.575
1.55
0.550
1.50
0.15
1.45
0.525
1.45
0.10
0.500
1.40
Implicit Tokens (d)
Implicit Tokens (e)
Figure 9: Parameter analysis experiment results for Mistral-7B Baraall----Inntttt-tto In-structt-0.22 [99. The accuracy and inference time of our method using Mistral when varying the number of implicit tokens
based on three assumptions: (1) semantically aligned queries will induce semantically aligned reasoning; (2) Different queries and their semantic aligned variants should be well-separated in implicit reasoning space to differentiate their semantics in the implicit reasoning space; (3) Samples from different semantic domains should more separated than those samples from the same semantic domain to differentiate their semantics domain in the implicit reasoning space. Here, the "semantic domain" means the semantic type of the reasoning task. For example, GSM8K (5), SVAMP (36), MultiArith 44, 39 are for mathematical reasoning, they are in the same semantic domain. However, CommonsenseQA [4 for commonsense reasoning. Thus, the samples within GSM8K (5). SVAMP (36). and MultiArith 44 [39] are in the same semantic domain, but their samples are in different semantic domains with CommensenseQA. The results are shown the Fig. [2] Fig. [3 Fig. Fig. 13 Fig. 16 and Fig. 17 For each figure, each subplot of the first row is the implicit reasoning tokens all samples and their semantic-aligned variants from one dataset. the second row, each subplot is the ith sample of all datasets, along with their comantic-aligned variants. We can observe across the five figures that our SemCoT is the only model to maintain low implicit reasoning variance under comantic-alignes queries and appropriately separate different samples in the implicit reasoning space. Meanwhile we can also see that our SemCoT recognizes semantic domains because the implicit reasoning for mathematical reasoning is reasonably separate from other domains, such commonsense reasoning.
27

---

Sample
Sample 2
Sample
Coint top
Commonsense@AA
MultiArthh
SVAMP
50
90
100
100
1000
50
100
100
CoinFlip
CommonsenseQA
MultiArith
SVAMP
Sumple
Sumple
Sumple 3
100
50
50
-0
50
150
100
-200
-50
-100
-100
-150
100
50
Figure 10: Case study for SoftCoT
[53 on Llama-2 2-7b-chat-hi [46
Sumple
Sample 2
Sample
CoinF lip
Qo
MultiArthh
SVAMP
50
50
40
30
-100
100
100
50
100
100
CoinFlip Sumple
CommonsenseeA
MultiArith
SVAMP
Sumple
Sample 3
100
50
50
-50
50
150
0
100
-200
-00
100
100
-150
100
-50
Figure 11: Case study for SoftCoT
[53 on Mistral-7B 7-
[9
28

---

Sample Commssns etaseQ
Sample
Sample
Coinflip
MultiArih
SVAMP
0,0005
0.52
010009
0.50
0.0003
0.002
0.001
0.05
-0,025
0.30
0.00
0.01
-0.00)
0.000
3,025
500
-0.05
Coin Sample
Commonsensee
MulliArith
SVAMP
Sample 2
Sample 3
800
400
250
200
250
250
100
100
1000
100
500
500
0
2000
2000
-00
2009
Figure 12: Case study for SemCoT on Llama-2-7b--haat 46
Sample
Sample 2
Sample
CoinfTlip
CommonsenseQA
MultiArith
SVAMP
0.005
0
11
0.000
25
0.005
25
0.01
-o 016
0.000
-20
20
-20
100
CoinFlip Sample
CommonsenseQA Sumple 2
MultiArith
SVAMP
Sample
200
200
200
100
100
100
100
400
400
200
200
500
-200
500
-200
500
-290
Figure 13: Case study for SemCoT on Mistral- -
29

---

Sample CemsenssenseQA.
Sample 2
Sample
Coinf lip
MultiArith
25
0
25
25
25
-50
25
CoinFlip Sample
CommonsenscQA Sumple 2
MultiArith
SVAMP
Sumple 3
40
20
40
20
25
20
100
-10
-20
100
-75
-50
100
-75
Figure 14: Case study for COCONUT
[53 Llama-22 22 2--iaa-----hhtt 7b-chat-hh
46
Sample CommonsensoCA
Sample 2
Sample 3
CoinFlip
MultiArith
30
000
200)
100
-100
100-100 MultiArith
CoinFlip lip Sample
Commonsense(A Sample 2
SVAMP
Sample
100
200
100
100
100
-200 100
-200
100
0
100
200
300
1000
100
Figure 15: Case study for COCONUT [53] on Mistral- IB-IInsrutttt--
[19
30

---

Sample
Sample 2
Sample
CoinFlip
CommonsscnsoQA
MultArit
SVAMP
0,025
0.000
0.0001
leng
0.0002
-0
0.00005
0.0000
-2525
0 0.0005
0.0209
-0.3
0.0
-00001
2,300
0.0000
0.9
-2.1
CoinFlip Sample
CommonsensoQA
MultiArith
Sample
Sample 3
20
20
20
-00
20
20
40
-40
400
20
20
300
20
20
40
300
Figure 16: Case study for CODI
41
Llama-2- 2-7hn----7--at-thhe 46
Sample
Sample
Sample
CoinFlip
QA
MultiArith
SVAMP
0.0001
0,0005
0.0 0.0000
(0000)
0.00
2.5
000001
9.01
0.022
0.0000
1e-5
-1,00
-02
0.00
0.9
0.01
Caintlip Sample
Commonsansee(A Sample 2
MultiArith
SVAMP
Sumple
20
20
20
20
40
40
20
-20
-20
-20
20
40
20
40
Figure 17: Case study for CODI 41 on Mistral-7B
Instruct-v0.2 7initall-B--Istuuet-tt [9
31


Reviews:
[{"review_id": "S9bBnUGOLE", "reviewer": "Reviewer_CuoZ", "summary": "The authors propose SemCoT, a novel framework for implicitly encoding chain of thought reasoning. SemCoT improves on previous work by addressing two important research gaps: (1) maintains semantic alignment between implicit and ground truth CoT and (2) improves computational cost of generating implicit CoT tokens. The authors evaluate their approach on two LMs, LLaMA-2-7B and Mistral-2, and across five arithmetic, commonsense and symbolic reasoning datasets. The authors show that their method achieves best or second best accuracy while at the same time being computationally efficient. The authors conduct ablation studies and analyses showcasing the robustness and strengths of their approach.", "questions": "- How does SemCoT behave with respect to CoT length compared to baselines? Do you pareto-dominate across the board, and does the performance gap close with longer or shorter CoTs?\n- The authors could consider improving presentation of Figure 2. I would color \"cyan box\" in cyan and add emojis for fire and snowflake for glance value. Some fire emojis could also be aligned a bit better wrt. the boxes, but this is very much a nitpick. I like the presentation overall.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written and presented and easy to follow\n- The experimental results are convincing\n- The analysis and ablation studies are detailed\n\nWeaknesses:\n- The models used in the study are quite dated. LLaMA3 and 3.1 models have been available for a while, while Mistral 2 has performed mediocre in my experience. It would be interesting to see how the method performs on more recent models.\n- It is not clear how the method scales with respect to long-reasoning models such as DeepSeek, which produce thousands of CoT tokens. It would be interesting to explore this.\n\nThe first weakness is something I would wish to be improved upon, while the second one (DeepSeek) is more of a nice-to-have, which would push the paper to a next level. I would however expect an analysis of how SemCoT behaves wrt. CoT length.", "was_revised": true, "final_justification": "The authors have added results on an additional model, Qwen, and responded to most of my comments. The second weakness, applying their method to large reasoning models, is understandably not feasible to address within the discussion period. My opinion is that the paper is good work.", "rebuttal": "We sincerely appreciate your dedicated time and effort in reviewing and providing invaluable feedback. We also thank you for recognizing the novelty and the significance of our contributions. We provide a point-to-point reply below for the mentioned concerns and questions.\n\n----\n\n> W1: Test SemCoT on more recent models.\n\nWe thank the reviewer for this important suggestion regarding testing on more recent models. We performed experiments on the Qwen-2.5 model series (released in **September 2024**). Specifically, we utilize **Qwen-2.5-0.5B as the lightweight implicit reasoning generator** and test the implicit CoT performance with **Qwen-2.5-7B as the LLM**. \n\nFrom the following table, we observe that our proposed SemCoT **consistently outperforms the baselines** across various datasets. We adopted three new datasets—LogiQA [1], StrategyQA [2], and MultihopQA [3]—to provide a comprehensive evaluation of implicit CoT performance across logical reasoning, strategic planning, and multi-hop reasoning tasks, respectively.\n\n| Dataset        | Metric   | Coconut       | ICoT-SI        | Pause        | SoftCoT       | SemCoT           |\n|----------------|----------|---------------|----------------|--------------|---------------|------------------|\n| **LogiQA**     | Acc (%)  | 11.90 ± 0.37  | 8.60 ± 0.86    | 33.20 ± 3.14 | 27.50 ± 2.43  | _64.70 ± 2.27_   |\n|                | Time (s) | _1.38 ± 0.06_ | 1.94 ± 0.20    | 3.20 ± 0.01  | 1.52 ± 0.23   | **1.10 ± 0.02**  |\n| **MultihopQA** | Acc (%)  | 2.30 ± 0.24   | _18.60 ± 2.24_ | 4.10 ± 0.58  | 4.80 ± 1.29   | **31.90 ± 1.71** |\n|                | Time (s) | 1.47 ± 0.10   | 1.91 ± 0.13    | 3.13 ± 0.02  | _1.16 ± 2.11_ | **0.93 ± 0.02**  |\n| **StrategyQA** | Acc (%)  | 3.20 ± 0.24   | 13.30 ± 1.25   | 37.20 ± 1.66 | 36.30 ± 1.17  | **99.10 ± 0.49** |\n|                | Time (s) | 1.56 ± 0.14   | 2.01 ± 0.16    | 3.21 ± 0.02  | _1.11 ± 0.09_ | **1.09 ± 0.01**  |\n\n[1] J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020.\n\n[2] ChilleD. \"StrategyQA.\" Hugging Face Datasets, Hugging Face, huggingface.co/datasets/ChilleD/StrategyQA. Accessed 31 July 2025.\n\n[3] cmriat. \"2wikimultihopqa.\" Hugging Face Datasets, Hugging Face, huggingface.co/datasets/cmriat/2wikimultihopqa. Accessed 31 July 2025.\n\n----\n\n> W2: It is nice and interesting to have the model scale to long-reasoning models such as DeepSeek, which produce thousands of CoT tokens. \n\nWe thank the reviewer for this valuable suggestion about scaling to long-reasoning models. Due to time constraints, we were unable to complete testing of SemCoT and baselines on DeepSeek reasoning models for this submission. We will include these results in the revised paper.\n\n-----\n\n> Q1: How does SemCoT behave with respect to CoT length compared to baselines? Do you Pareto-dominate across the board, and does the performance gap close with longer or shorter CoTs?\n\nWe thank the reviewer for this insightful question about CoT length behavior and performance comparison. \n\nSemCoT's performance with increasing reasoning length is dataset-dependent, as shown in Fig. 4 (main paper), Fig. 8, and Fig. 9 (supplementary materials). \n- For some datasets and LLMs, **one to two** implicit reasoning tokens are sufficient to achieve optimal CoT performance. \n- In other situations, CoT accuracy **increases monotonically** with the number of implicit reasoning tokens.\n\nThe table below shows the performance gap between SemCoT and baselines across different numbers of implicit reasoning tokens (using the same LLMs and datasets as referenced in the answer for W1). We observe that the **performance gap generally increases when incorporating more implicit reasoning tokens**, demonstrating SemCoT's superior scaling behavior.\n\n|       Method | 1                | 2                 | 3                 | 4                 | 5                 |\n|-------------|------------------|------------------|------------------|------------------|------------------|\n| Coconut     | 3.20 ± 0.24      | 6.50 ± 0.45      | 3.90 ± 0.20      | 4.50 ± 0.00      | 5.80 ± 0.51      |\n| CODI        | 48.00 ± 0.00     | 89.20 ± 20.60    | 90.20 ± 18.60    | 40.30 ± 3.36     | 99.50 ± 0.00     |\n| Pause       | 37.20 ± 1.60     | 35.20 ± 3.91     | 30.70 ± 1.96     | 30.30 ± 1.29     | 29.30 ± 1.21     |\n| SoftCoT     | 36.30 ± 1.17     | 21.60 ± 1.36     | 22.10 ± 0.97     | 22.20 ± 0.81     | 22.20 ± 1.81     |\n| SemCoT  (**ours**)   | 98.50 ± 1.26     | 98.70 ± 0.68     | 97.10 ± 1.59     | 95.20 ± 1.44     | 97.90 ± 0.37     |\n\n----\n\n> Q2: Consider improving the presentation of Figure 2, e.g., color the \"cyan box\" in cyan and add emojis for fire and snowflake for glance value; align the fire emojis better with the boxes. \n\nWe thank the reviewer for these helpful suggestions regarding figure presentation and visual improvements. We sincerely appreciate the invaluable suggestions! We will update our manuscript accordingly."}, {"review_id": "lfVhzNF0Oy", "reviewer": "Reviewer_LvAf", "summary": "This paper proposes SemCoT, a method for training a model to perform reasoning using an internal CoT token rather than a discrete CoT. Unlike existing methods, they train the model by encouraging the latent reasoning tokens to be similar in embedding space to the true discrete reasoning chain. This leads to large performance improvements and speeds up inference time.", "questions": "* Why should the statement “optimal implicit CoT performance 3 is achieved when the implicit CoT semantically aligns with the ground-truth CoT” be true?\n* Why does performance decrease with more reasoning tokens?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The paper presents a novel approach to achieving internal LLM reasoning, and the novelty is well presented by directly addressing the related works and their limitations.\n* The paper is well organized and overall clear.\n* Several ablation studies are performed to show the importance of different components of the method and uncover potential limitations.\n* The results look very strong with consistently large performance improvements over baselines.\n\nWeaknesses:\n* Part of the motivation for this work is based on the idea that implicit CoT should be semantically equivalent to normal CoT, but these seem like two potentially distinct methods where the intermediate reasoning does not need to be equivalent between them. The method may still be valuable, but motivating it as a necessary condition seems too far.\n* Overall performance decreases with more than one reasoning token. This seems like a significant limitation if only one reasoning token can be used since not much “reasoning” is really happening implicitly in this case.", "was_revised": true, "final_justification": "The authors mostly answered my concerns during the discussion. I do not know how exactly the wording of the paper will be modified to make the framing of the paper more clear in that semantic alignment is not a strictly necessary condition for implicit CoT, but it's not too major of a change. Also, the results showing that performance decreases with more than one reasoning token makes me partially doubt that the \"implicit CoT\" as described in the paper is really emulating a CoT like process, but the authors pointed me to results in the appendix showing that for some datasets, performance strictly increases with more reasoning tokens. Overall, the reasons to accept the paper outweigh reasons for rejection, so I keep my score of 4.", "rebuttal": "We sincerely appreciate the time and effort you dedicated to reviewing and providing invaluable feedback to enhance the quality of this paper. We provide a point-to-point reply below for the mentioned concerns and questions.\n\n----\n\n> W1: Intermediate (\"implicit\" in paper) reasoning does not need to be equivalent to explicit reasoning to achieve satisfying CoT performance. \n\nWe thank the reviewer for the thoughtful comment and kindly note that **it is a misunderstanding**. In our work, we treat \"implicit reasoning being semantically equivalent to ground-truth explicit reasoning\" as a sufficient condition for optimal CoT performance, not a necessary one.\n\nOur assumption (line 117) is that when semantic alignment is established, implicit CoT will achieve optimal performance. However, implicit CoT could potentially achieve the same optimal performance even when the implicit reasoning semantically differs from the explicit reasoning. This is possible because a single problem can often be solved through multiple valid approaches, each following different reasoning paths while arriving at the same correct answer.\n\n----\n\n> Q1: Why should the statement \"optimal implicit CoT performance is achieved when the implicit CoT semantically aligns with the ground-truth CoT\" be true?\n\n \nThank you for the insightful question. We will explain this statement from two perspectives.\n- Theoretical foundation: This statement is a **core assumption** of our work (line 117), similar to an axiom in a mathematics theory. Therefore, we cannot formally prove it. Still, we consider it intuitive: When the model's internal reasoning process (implicit CoT) matches the correct reasoning steps (ground-truth CoT), we should achieve satisfying CoT performance.\n- Empirical support: Our **ablation study provides evidence** for this assumption. Figure 3 shows that removing semantic alignment (SemCoT-NSA) causes significant performance drops across all datasets, directly demonstrating that semantic alignment improves CoT performance.\n\nWe acknowledge that using \"optimal\" may be misleading. We will revise this to a more moderate term as a substitute.\n\n----\n\n> W2, Q2: Why does the performance decrease with more than one reasoning token? This is limited as not much reasoning can happen with one reasoning token.\n\nThe performance **does not necessarily decrease** with more than one token. Our supplementary materials (Figures 8 and 9) demonstrate that performance with implicit reasoning tokens varies across different LLMs and datasets. The optimal number of reasoning tokens is not universally one—it depends on the specific model-dataset combination:\n- For some cases, 1-2 tokens are indeed sufficient for optimal Chain-of-Thought performance. \n- **For other cases, accuracy continues to improve monotonically as we increase the number of reasoning tokens**. The key finding is that different scenarios require different numbers of reasoning tokens to achieve peak performance."}, {"review_id": "Fzo5Ob6Tn7", "reviewer": "Reviewer_8arc", "summary": "This paper presents a new method designed to make LLMs faster and more efficient at reasoning through complex problems. Traditional CoT often involves generating many words, which can be slow and resource-intensive. The novel method SemCoT solves this by replacing long text explanations with a few hidden tokens inside the model that still carry the same meaning. To ensure these hidden representations stay true to the original reasoning, a special sentence transformer is used to check if the meaning is preserved. Additionally, they train a smaller, faster helper model to generate these hidden tokens efficiently. Experiments show that SemCoT is not only faster but also more accurate than other similar methods, making it a strong solution for improving the speed and quality of reasoning in LLMs.", "questions": "see above.", "limitations": "he authors acknowledge key limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- interesting idea aiming to enhance reasoning with the using fewer hidden tokens instead of long text.\n- Use of a special sentence transformer to keep the same meaning as the full explanation when performing step-by step reasoning. \n- good evaluation and comparaison with SOTA models. \n\nWeaknesses:\n- The fact that reasoning is hidden hinders the transparency and interpretability of the reasoning. How can we ensure that what is hidden is not hallucinated? Matching semantics does not always mean the reasoning is logically correct. The model might preserve “meaning” but still make a wrong or faulty logical step. \n- My feeling is that  SemCoT is quite difficult to learn. While this is mentionned as a limitation of the paper. Any discussion on how we can improve this or reduce complexity? \n- The evaluation is mainly focused on mathematical reasoning. More datasets on reasoning would be interesting to add more complex datasets such as StrategyQA, LogiQA, HotpotQA, 2WikiMultiHopQA, etc.", "was_revised": true, "final_justification": "I am raising my score from 3 to 4, as part of my concerns has been properly addressed.  However, I remain concerned about the interpretability and complexity, which the authors have left for future work.", "rebuttal": "We sincerely appreciate your dedicated time and effort in reviewing and providing invaluable feedback. We provide a point-to-point reply below to clarify certain misunderstandings and provide responses to the mentioned concerns and questions.\n\n----\n\n> W1 (part 1): Hidden reasoning hinders the transparency and interpretability of the reasoning. \n\nWe acknowledge that implicit reasoning reduces transparency. However, this work focuses on scenarios where optimal answer accuracy within minimal time is prioritized over reasoning interpretability. For instance, in recommender systems, users need fast, accurate recommendations rather than detailed reasoning explanations. SemCoT serves as a tool to improve recommendation precision with minimal additional time cost.\n\nFuture work could involve **training a language decoder** that takes implicit reasoning as input and produces interpretable reasoning as output, enabling transparent inspection of the reasoning process.\n\n----\n\n> W1 (part 2): Matching semantics does not always mean the reasoning is logically correct. The model might preserve \"meaning\" but still make a wrong or faulty logical step (i.e., hallucinate). How can we ensure that what is hidden is not hallucinated? \n\nWe acknowledge that aligning semantics may not completely guarantee the reasoning logic's correctness, but we do not optimize for semantics alone. We have $L_{pred}$, which ensures the implicit reasoning must actually produce correct answers. \n\nIf logical steps are faulty, answer accuracy drops, our loss function penalizes this, and restricts the model to follow correct logic informed by the ground-truth reasoning. In fact, Table 1 shows that SemCoT achieves higher accuracy than baselines across all datasets. If we preserved \"meaning\" but broke logic, the CoT accuracy would suffer.\n\n----\n\n> W2: My feeling is that SemCoT is quite difficult to learn. While mentioned as a limitation in the paper, any discussion on how to improve this or reduce complexity?\n\nTo reduce model complexity, we recommend a two-step approach:\n- Step 1: Replace our customized sentence transformer with an **existing well-trained open-source sentence transformer**. This eliminates the need to train a sentence transformer from scratch.\n- Step 2: Design a fast and effective adapter that enables the pre-trained sentence transformer to understand the implicit tokens from our implicit reasoning generator.\n\nHowever, we note that this approach **faces significant challenges**. For example, one challenge lies in the substantial semantic gap between the embedding spaces of the sentence transformer and our implicit reasoning generator. While this reduces training complexity, bridging this semantic gap between the sentence transformer and the implicit reasoning generator remains a difficult research direction that requires careful adapter design.\n\n----\n\n> W3: The evaluation is mainly focused on mathematical reasoning. More datasets on reasoning would be interesting to add more complex datasets such as StrategyQA, LogiQA, HotpotQA, 2WikiMultiHopQA, etc.\n\nWe appreciate this suggestion. While our evaluation does emphasize mathematical reasoning, we actually tested our model across multiple reasoning types:\n\n- Mathematical reasoning: Our primary focus with datasets like GSM8K and MATH\n- Commonsense reasoning: Evaluated on CommonsenseQA\n- Symbolic reasoning: Tested on CoinFlip\n\nFollowing your recommendation, we have now conducted additional experiments on StrategyQA [1], LogiQA [2], and 2WikiMultiHopQA [3] (denoted as \"MultihopQA\" as follows) using the LLaMA series. The results are presented in the table below. Across all datasets, our method consistently achieves optimal Chain-of-Thought (CoT) performance compared to baseline methods, demonstrating the generalizability of our approach beyond mathematical reasoning.\n| Dataset    | Metric   | Coconut             | CODI                   | ICoT-SI            | Pause             | SoftCoT            | SemCoT                 |\n|------------|----------|---------------------|------------------------|--------------------|-------------------|-------------------|------------------------|\n| **LogiQA** | Acc (%) | 11.90 ± 0.37        | **89.60 ± 9.07**       | 8.60 ± 0.86        | 33.20 ± 3.14      | 27.50 ± 2.43      | _64.70 ± 2.27_         |\n|            | Time (s)| _1.38 ± 0.06_       | 8.22 ± 0.28            | 1.94 ± 0.20        | 3.20 ± 0.01       | 1.52 ± 0.23       | **1.10 ± 0.02**        |\n| **MultihopQA** | Acc (%) | 2.30 ± 0.24        | 7.60 ± 2.06            | _18.60 ± 2.24_     | 4.10 ± 0.58       | 4.80 ± 1.29       | **31.90 ± 1.71**       |\n|            | Time (s)| 1.47 ± 0.10        | 2.08 ± 0.17            | 1.91 ± 0.13        | 3.13 ± 0.02       | _1.16 ± 2.11_     | **0.93 ± 0.02**        |\n| **StrategyQA** | Acc (%) | 3.20 ± 0.24        | _48.00 ± 0.00_         | 13.30 ± 1.25       | 37.20 ± 1.66      | 36.30 ± 1.17      | **99.10 ± 0.49**       |\n|            | Time (s)| 1.56 ± 0.14        | 1.89 ± 0.09            | 2.01 ± 0.16        | 3.21 ± 0.02       | _1.11 ± 0.09_     | **1.09 ± 0.01**        |\n\n[1] ChilleD. \"StrategyQA.\" Hugging Face Datasets, Hugging Face, huggingface.co/datasets/ChilleD/StrategyQA. Accessed 31 July 2025.\n\n[2] J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. arXiv preprint arXiv:2007.08124, 2020.\n\n[3] cmriat. \"2wikimultihopqa.\" Hugging Face Datasets, Hugging Face, huggingface.co/datasets/cmriat/2wikimultihopqa. Accessed 31 July 2025."}, {"review_id": "I7LtuK9t1I", "reviewer": "Reviewer_xrMd", "summary": "The paper focuses on implicit chain of thought reasoning, encoding reasoning steps within large language models hidden embeddings rather than explicit tokens. The authors propose a novel semantically-aligned implicit CoT framework termed SemCoT with a contrastively trained sentence transformer that evaluates semantic alignment between implicit and explicit reasoning, and an efficient implicit reasoning generator that finetunes a lightweight language model using knowledge distillation. Experimentally they demonstrate the efficiency and effectiveness of SemCoT.", "questions": "I found the research interesting and the presentation effective. I have some questions that I would be keen to find out more about.\n\nSuggested changes:\n\n\n-\tClarify what “subject to maximal retention of answer accuracy compared with the original CoT” means in the problem statement.\n\nQuestions to think about:\n\n\n-\tCurrently the experimental results are compared with SoftCoT. SoftCot++ may be more similar to SemCoT. Please discuss the differences.\n\n-\tI am interested in discussion around the results that may shed some insight into when the biggest gains are made and when the gains are very limited, e.g. why are there larger gains in accuracy on the SVAMP dataset on both LLMs and the COINFLIP dataset specific to Llama.\n\n-\tThe paper in essence describes how the method is implemented. It would be interesting to understand some of the choices that were made, alternatives that were considered and why they were not adopted.\n\n- I understand how accuracy is measured in your paper but it is unclear to me why the accuracy in your paper differs so much from the reported accuracy in the SoftCoT paper. Please discuss and include a discussion of the advantages and disadvantages of the different accuracy measures.\n\nMinor typos:\n\n\n-\tThe citation to the COCONUT paper on pg 6 is wrong.\n\n-\tIn the problem statement (line 95), “an white-box LLM” should be “a white-box LLM”.\n\n-\tOn line 99, “v.s.” should be “vs.”\n\n-\tOn line 221, “Are there evidence” should “Is there evidence”", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n\n1. The paper is nicely structured and the writing is clear. In particular the research is well motivated and the authors take time in the introduction to explain the context very clearly.\n\n2.\tThe experimentation and results address the research questions. \n\n3.\tThe results are compared against state-of-the-art approaches. The comparisons look fair.\n\n\nWeaknesses\n\n\n1.\tThe originality seems somewhat limited compared SoftCoT++, where they seem to also use the contrastive learning approach.\n\n2.\tThe paper reports results but does not provide any insight into the results, e.g. why are there larger gains in accuracy on the SVAMP dataset on both LLMs and the COINFLIP dataset specific to Llama.\n\n3.\tThere is no future work discussed in the main paper. I suggest this is pulled in from the supplementary.", "was_revised": true, "final_justification": "I have reviewed the author rebuttals and discussion. They have addressed my concerns and  believe that including some of the discussion from the rebuttals would strengthen the paper. A discussion of future work is particularly important.", "rebuttal": "We sincerely appreciate your dedicated time and effort in reviewing and providing invaluable feedback. We also thank you for recognizing the quality and the significance of our contributions. \n\n----\n\n> Suggestion: Clarify what \"subject to maximal retention of answer accuracy compared with the original CoT\" means in the problem statement.\n\nWe thank the reviewer for requesting clarification on this important constraint in our problem formulation. \"subject to maximal retention of answer accuracy compared with the original CoT\" means that while we aim to minimize the total inference time $T$ for generating implicit reasoning $Z$ and the final answer $Y$, we must ensure that the answer accuracy achieved with our implicit CoT approach does not significantly degrade compared to the accuracy obtained using traditional explicit CoT reasoning. Here, the \"Maximal retention\" means that we seek to preserve as much of the original CoT's answer accuracy as possible, ideally achieving comparable or better performance.\n\n----\n\n> W1, Q1: The originality seems somewhat limited compared to SoftCoT++, where they also seem to use the contrastive learning approach. Discuss their differences.\n\nThank you for pointing out SoftCoT++ [1]. We would like to first highlight our independent contribution: SoftCoT++ [1] was submitted to ArXiv the same day as the NeurIPS submission deadline (5/16/2025), and we surely have no knowledge of SoftCoT++. This timeline proves that our work is developed completely independently with originality. \n\nHere is a comparison between SemCoT and SoftCoT++ [1] and clarify the most significant distinctions that demonstrate SemCoT's novelty:\n\n- **Fundamental Problem Focus**: SemCoT addresses **semantic alignment preservation** - ensuring implicit reasoning maintains semantic fidelity to ground-truth reasoning. SoftCoT++ [1] focuses on **test-time scaling** - generating diverse reasoning paths for parallel inference. These are completely different challenges in implicit CoT reasoning.\n\n- **Contrastive Learning Purpose**: While both use contrastive learning, the applications are fundamentally different. SemCoT uses it to **measure and preserve semantic similarity** between implicit and explicit reasoning during training. SoftCoT++ [1] uses it to **maximize diversity** among soft thoughts for better test-time exploration. Same technique, opposite goals.\n\n- **Architectural Innovation**: SemCoT introduces a **customized sentence transformer** specifically designed to evaluate semantic alignment between LLM embeddings and natural language, plus a novel two-step training process. SoftCoT++ [1] focuses on **multiple initial tokens** and scaling mechanisms without addressing semantic preservation.\n\nTo conclude, these works address different critical gaps: SemCoT solves the semantic preservation problem that causes performance degradation in existing implicit CoT methods, while SoftCoT++ [1] tackles test-time scaling challenges. Our experimental results demonstrate that SemCoT achieves superior effectiveness across benchmarks by maintaining semantic fidelity, a problem SoftCoT++ does not address.\n\n----\n\n\n> W2, Q2: The paper reports results but does not provide any insight into the results, e.g., why are there larger gains in accuracy on the SVAMP dataset on both LLMs and the COINFLIP dataset specific to Llama. I am interested in discussion around the results that may shed some insight into when the biggest gains are made and when the gains are very limited, e.g., why are there larger gains in accuracy on the SVAMP dataset on both LLMs and the COINFLIP dataset specific to Llama.\n\nWe thank the reviewer for this insightful question about result interpretation. We indeed analyzed the experiment results in Table 1 and provided some discussion in Section 4.2. Here, we provide a more detailed analysis of the performance variations across datasets:\n- **General Observation**: LLaMA-2-chat-hf demonstrates superior ability compared to Mistral-7B-Instruct-v0.1 in aligning ground-truth reasoning semantics with implicit reasoning tokens. This is evidenced by consistently larger performance gains between SemCoT and the runner-up baseline across all datasets in Table 1.\n- **Why SVAMP shows larger gains across both LLMs**: SVAMP is the simplest mathematical reasoning dataset, with shorter question lengths compared to GSM8K and MultiArith. We attribute the consistently large performance gains across both LLMs to the relative ease of aligning semantics between ground-truth reasoning and implicit reasoning in this simpler dataset. In contrast, GSM8K and MultiArith may require more than a single reasoning token to fully inject ground-truth reasoning into implicit reasoning in a semantically aligned manner, which limits their performance gains compared to baseline methods.\n- **Why CoinFlip shows larger gains specifically on LLaMA**: This relates to LLaMA's superior explicit-implicit reasoning alignment ability mentioned above. CoinFlip shows the largest LLaMA-Mistral performance gap because it is the simplest dataset in our evaluation. Since the task is straightforward for Mistral (which has stronger vanilla question-answering ability than LLaMA), Mistral can achieve good performance without explicit reasoning alignment procedures. This explains why SemCoT shows minimal improvement over baselines on Mistral. However, while LLaMA has better explicit-implicit reasoning alignment ability, the LLaMA model itself has weaker vanilla internal capability for providing correct answers. Therefore, with reasoning alignment, it shows substantial improvement.\n\n----\n\n> W3: There is no future work discussed in the main paper. I suggest this be pulled in from the supplementary.\n\nThank you for the suggestion! We will carefully discuss the future work in the main paper.\n\n----\n\n\n> Q3: It would be interesting to understand some of the choices that were made, alternatives that were considered, and why they were not adopted.\n\nWe have worked on different ideas, in which one closest to our idea is to utilize an existing well-trained open-source sentence transformer to substitute the customized sentence transformer (removing the tokenization process) in our work. However, due to the significant semantic gaps between the sentence transformer and the LLM/implicit reasoning generator, the experiment results are suboptimal. We believe that if appropriate semantic space alignment modules can be designed to achieve satisfying CoT performance, applying existing sentence transformers can be a more computationally viable method.\n\n----\n\n> Q4: I understand how accuracy is measured in your paper, but it is unclear to me why the accuracy in your paper differs so much from the reported accuracy in the SoftCoT paper. Please discuss and include a discussion of the advantages and disadvantages of the different accuracy measures.\n\nThe two papers report different results on SoftCoT [2] for two main reasons. \n- First, our work tested implicit CoT methods with **smaller LLMs** (e.g., LLaMA-2-7B-chat-hf), which have zero-shot vanilla CoT performance of only 8.9% on GSM8K as reported by Cheng et al. [3]. In contrast, the SoftCoT [2] paper used LLaMA-3.1-8B-Instruct, which has significantly higher zero-shot vanilla CoT performance (79.61% on GSM8K). \n- Second, we tested the model with **one reasoning token**, while the SoftCoT paper used **four reasoning tokens**, which also contributes to the performance difference.\n\nThe accuracy measures difference when collecting the results in the main results tables (i.e., Table 1 in SemCoT and Table 2 in SoftCoT [2]) between the two papers reflects different emphasis that the authors would like to demonstrate. Our SemCoT aims to demonstrate how adding a minimum number of (i.e., only one) implicit reasoning token improve LLM performance, as this design introduces minimal additional inference time cost caused by implicit reasoning tokens. We postulate that the SoftCoT work focuses more on demonstrating optimal LLM performance that the SoftCoT framework provides when using the optimal number (in their case is four) of implicit reasoning tokens. There are no direct advantages or disadvantages in their design of the accuracy measure.  \n\n\n[1] Xu, Yige, et al. \"SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoning.\" arXiv preprint arXiv:2505.11484 (2025).\n\n[2] Xu, Yige, et al. \"Softcot: Soft chain-of-thought for efficient reasoning with llms.\" arXiv preprint arXiv:2502.12134 (2025).\n\n[3] Cheng, Jeffrey, and Benjamin Van Durme. \"Compressed chain of thought: Efficient reasoning through dense representations.\" arXiv preprint arXiv:2412.13171 (2024)."}]

Author rebuttal:

Meta-review: The paper presents SemCoT, leveraging "implicit" tokens to accelerate CoT. The "implicit" tokens are learnt from a light-weight LM that semantically aligned with a sentence transformer. Reviewers think the method is novel, experiments are thorough, and findings are well-supported. We'd like to remind the authors to incorporate reviewer's comments into the paper, especially on interpretability and complexity.
Decision: Accept (poster)