Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
Authors: Zhen Zhang
Xuehai He
Weixiang Yan
Ao Shen
Chenyang Zhao
Xin Eric Wang
Venue: NeurIPS 2025 poster ()
Keywords: Chain-of-thought
Reasoning
Efficiency
Continous Space Reasoning
Primary area: deep_learning
TL;DR: Soft Thinking enables large language models to reason more accurately and efficiently by using probability-weighted concept tokens in a continuous space, rather than committing to discrete tokens at each step.

Abstract: Human cognition typically involves thinking through abstract, fluid concepts rather than strictly using discrete linguistic tokens. Current Large Language Models (LLMs), however, are constrained to reasoning within the boundaries of human language, processing discrete token embeddings that represent fixed points in semantic space. This discrete constraint restricts the expressive power and upper potential of such reasoning models, often causing incomplete exploration of reasoning paths, as standard Chain-of-Thought (CoT) methods rely on sampling one token per step. In this work, we introduce Soft Thinking, a training-free method that emulates human-like ``soft'' reasoning by generating abstract concept tokens in a continuous concept space. These concept tokens are created by the probability-weighted mixture of token embeddings, which span the continuous concept space, enabling smooth transitions and richer representations that transcend traditional discrete boundaries. In essence, each generated concept token encapsulates multiple meanings from related discrete tokens, implicitly exploring various reasoning paths to converge effectively toward the correct answer. Empirical evaluations on diverse mathematical and coding benchmarks consistently demonstrate the effectiveness and efficiency of Soft Thinking, improving pass@1 accuracy by up to 2.48 points while simultaneously reducing token usage by up to 22.4\% compared to standard CoT. Qualitative analysis further reveals that Soft Thinking outputs remain highly interpretable and readable, highlighting the potential of Soft Thinking to break the inherent limits of discrete language-based reasoning.

Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
Zhen Zhang!* Xuehai He² Weixiang Yan' Ao Shen' Chenyang Zhao Xin Eric Wang'
University of California, Santa Barbara, University of California, Santa Cruz University of California, Los Angeles, *Purdue University, 'LMSYS Org zhen zhang@uesb. edu, ericxwang@uesb.c edu
Average Accuracy (Hath
Average Longth (Math)
Avsrage Accuracy (Coding)
Average Longth (Coding)
Got thiking
Co Thiking Thisiin
Cal [haoigg Thinkin
Gol thising
ACCURDIO
$4.18
03.50
03.37
CARTE atton
040-246
200-22
010 0-08 Momu
DNECCRRO 12
Figure 1: Soft Thinking vs. Chain-of- Thought thinking on mathematical and coding datasets. Soft Thinking consistently improves both accuracy (with improvements of up to 2.48% pass @ accuracy) and generation efficiency (achieving up to 22.4% % reduction generation length) across both tasks, without any training.
Abstract
Human cognition typically involves thinking through abstract, fluid concepts rather than strictly using discrete linguistic tokens. Current reasoning models, however, are constrained to reasoning within the boundaries of human language, process- ing discrete token embeddings that represent fixed points in the semantic space. This discrete constraint restricts the expressive power and upper potential of such reasoning models, often causing incomplete exploration of reasoning paths, standard Chain-of-Thouur Thought (CoT) methods rely on sampling one token per step. In this work, introduce Soft Thinking, a training-free method that emulates buman--lkke "soft" reasoning by generating soft, abstract concept tokens in a contin- uous concept space. These concept tokens are created by the probability-wiighee mixture of token embeddings, which form the continuous concept space, enabling smooth transitions and richer representations that transcend traditional discrete boundaries. In essence, each generated concept token encapsulates multiple mean- ings from related discrete tokens, implicitly exploring various reasoning paths converge effectively toward the correct answer. Empirical evaluations on diverse mathematical and coding benchmarks consistently demonstrate the effectiveness and efficiency of Soft Thinking, improving pass @ accuracy by up 2.48 points while simultaneously reducing token usage by up to 22.4% compared to standard COT. Qualitative analysis further reveals that Soft Thinking outputs remain highly interpretable and readable, highlighting the potential Soft Thinking to break the inherent bottleneck of discrete language-based reasoning. Code is available here
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The limits of my language mean the limits of my world.
Ludwig Wittgenstein
Introduction
Large Language Models (LLMs) have achieved impressive results across wide range of complex reasoning tasks. key technique contributing to this success is Chain-of-Thoughh (CoT) reasoning [ 4. which enables models to solve problems step-by-step by generating intermediate reasoning steps in natural language. Despite its effectiveness, standard CoT reasoning restricts the model's s outputs to sequences of discrete, predefined tokens, inherently bounded by the expressive limits of human language. Confining LLM reasoning to the discrete natural language tokens may fundamentally restrict their potential to represent and manipulate abstract concepts. Moreover, neuroscientific evidence shows that the human brain represents and stores information at the level of abstract concepts, not merely words, and that reasoning is based non-verbal conceptual processing independent of the language network ($
Another fundamental limitation of standard CoT reasoning is its inherently unidirectional and sequential nature: at each step, the model samples a single token, committing to one specific branch of the reasoning path. In tasks with high uncertainty multiple plausible trajectories, this approach can easily lead the model down an incorrect path, resulting in suboptimal answers or wasted tokens on the wrong path, thus reducing both performance and token efficiency [9] [0]. In contrast, humans do not rely solely on sequentially producing explicit linguistic tokens. Instead, they can simultaneously consider multiple possibilities, integrate abstract concepts, and only later verbalize their thoughts. This allows for more flexible, parallel, and comprehensive reasoning, enabling humans to navigate complex problems more effectively.
In this work, we propose a new perspective: instead of constraining LLMs to reason within the discrete, sequential space of language tokens, we aim to enable LLMs to reason with soft, abstract concepts, which encompass more general and fine-grained semantics and retain information about multiple possible paths. To achieve this, we introduce Soft Thinking, training- -fre method that unlocks the reasoning potential of LLMs in a continuous concept space. Specifically, Soft Thinking replaces the discrete token selection in standard CoT with probabilistic soft aggregation over the entire vocabulary, which we refer to as a concept token. This retains the original distribution of the next step. At each step, construct new embedding from concept token by probability- -weihhtnn all token embeddings, which form the continuous concept token. This approach allows the model represent and process abstract concepts, endowing each output token with more nuanced and fine-grained semantics, and enabling the processing of multiple paths conceptually.
Unlike standard CoT that forces the model to commit to single next token at each step by collapsing the probability distribution, our method naturally preserves a "superposition" " which retains the entire information in each step. As result, we introduce a Cold Stop mechanism to further boost efficiency and address the challenge of generation collapse (e.g.. repetition) caused by outto--distribution (OOD) 11 inputs, where certain concept tokens may be unseen during training. To be specific, Cold Stop monitors the entropy of the model's 's output distribution at each step and terminates the reasoning process early when the model demonstrates high confidence (i.e., low entropy) over several consecutive steps. This mechanism prevents unnecessary computation and mitigates the risk of model collapse when dealing with OOD inputs, ensuring more robust and efficient reasoning.
Soft Thinking offers two major advances. First, by operating in the continuous concept space formed convex combination of all token embeddings, the model can capture and manipulate abstract concepts and detailed semantic information; Second, because each concept token keeps a probability distribution from all possible next tokens, the model can implicitly and efficiently explore multiple reasoning paths in parallel, rather than being limited to a single trajectory. Therefore, Soft Thinking not only improves the comprehensiveness of reasoning but also accelerates convergence toward correct answers.
Empirical evaluations conducted on mathematical and coding benchmarks using mainstream LLM architectures, including Ilama 112 and Qwen [3] with large model sizes 32B and 70B parameters, consistently demonstrate the effectiveness and efficiency of Soft Thinking. The method improves

---

pass I accuracy by up to 2.48 points while simultaneously reducing token usage by up to 22.4% compared to standard COT. Furthermore, qualitative assessments reveal that intermediate reasoning steps generated by Soft Thinking are highly readable, interpretable, and informative. Overall, Soft Thinking presents an alternative reasoning paradigm that breaks the bottleneck of discrete token-based reasoning.
Related Work
Chain-of-Thhuuhh (CoT) Reasoning. CoT reasoning enhances the multi-step reasoning capa- bilities of large language models by introducing explicit intermediate steps. Existing approaches primarily include prompt-based learning methods 113, supervised fine-tuning [4 15), and rein- forcement learning optimization [16 Moreover, according to inference-times scaling laws [20]. model performance continues improve as the length of reasoning chains increases. However, as the chain grows longer, the computational cost also rises, making efficiency a growing concern. To address this challenge, we propose to shift CoT reasoning from the discrete natural language token to a continuous concept space, which is formed as a convex combination of all token embeddings. In this space, the model can select and integrate multiple potential reasoning trajectories at the token level.
Continuous Space Reasoning. 21 constructed datasets for two-hop reasoning tasks and showed that intermediate reasoning variables could be decoded from hidden representations. Building on this, (22) introduced interventions on hidden states to manipulate reasoning outcomes. Parallel latent reasoning paths have also been observed (23
introduced latent planning steps: proposed predicting discrete planning tokens before generating reasoning steps. 25 proposes to do reasoning at an abstract language level beyond tokens and explores an explicit hierarchical structure. [26] proposes extracting text embeddings from the last token of LLMs fine-tuned with instructions contrastive data. COCONUT (27 uses the last hidden state of the model's final layer as the next-step embedding. However, this method still face critical challenges. In language models with fewer than 7B parameters, the input embedding layer and the output language model head are typically weight-tied, enabling continuous-space reasoning by aligning the input and output spaces after extensive training. In contrast, for models with more than 7 billion parameters, these components are typically decoupled, meaning that the hidden states and input embeddings reside in different spaces. Directly using hidden states as input embeddings leads to significant representational mismatch, which is difficult to bridge even with extensive retraining. Such retraining often leads overfitting, catastrophic forgetting, or ineffective performance in practice (28). To address these limitations, we propose training-fire -fre approach that utilizes the distribution over the vocabulary each step as bridge. This method effectively aligns the hidden state output space with the input embedding space, enabling seamless representation alignment during continuous-space reasoning.
3 Methodology
Soft Thinking
In this section, we introduce Soft Thinking, method that generalizes standard Chain-of- Thought (CoT) reasoning by replacing dis- crete one-hot tokens with concept lokens and keeping the entire original probability dis- tribution. As shown Figure 2 the new embeddings are computed using probability- weighted interpolation across all embeddings based on the preceding concept token, facili- tating reasoning within a continuous concept space. Furthermore, we propose the Cold Stop mechanism, which halts intermediate reason- ing steps when overconfident, enhancing in- ference efficiency and preventing generation collapse.
Intereediate Thinking
Y;
Concept Tokens (ct)
Original Dist.
Probability Dist.
Large Language Model
Enbooding
Meightted
Inpurt
Figure 2: Soft Thinking replaces discrete tokens with soft, abstract concept tokens, enabling reasoning in continuous concept space.
3

---

3.1 Preliminary: Standard Shainnof-Thhouht Decoding
Let V be the vocabulary of size IVI, and let E E R Vixd be the token embedding matrix. For any token index k, we denote its embedding by e(k) = E|k] RR. Given an input con- text: F1-L 22, ................................ ,L), the model first generates an intermediate reasoning trace (the Chain-of-Thought) of length m: 1imm (1,,22...., (m), and then produces a final answer of length n: y1.m (y1, U2...., yu.).
At each intermediate thinking step i, the LLM consumes the embeddings of all previously generated tokens and the input, and the next token then sampled discretely:
Pi LLM(e(Z11). e(t1.a1)) E A
(I)
Here, A VI-1 denotes the (IV) 1 --dimensional probability simplex, representing the set of all valid probability distributions over the vocabulary. Decoding continues until the special end-of-thinking
token (/think) is generated, i.e. tm
(accod((/thinkk))
After reasoning, the model switches to answer mode. For each answer position j. it computes the probability yj and then sample one token Vy:
yj q3 = LLM(((ZIa)) e(ti:m), e(y1:)-1)) A
(2)
All tokens in both stages drawn from their respective discrete distributions, committing discrete token id at each step. In the next section, we introduce Soft Thinking, which replaces this discrete sampling with continuous concept tokens, preserving the full distributional information throughout multi-step reasoning.
3.2 Soft Thinking: Reasoning in Continuous Concept Space
Definition 1 (Concept Token). At any intermediate thinking step, let p E A be the LLM- produced probability distribution over the vocabulary. We call this probability vector a concept token, denoted by
:= p.
Unlike traditional step that collapses the distribution to single token id, the concept token preserves the full distribution of every possible next step.
Definition 2 (Continuous Concept Space). Let E RIV the embedding matrix and e(k) = E[k] the embedding of the k-th vocabulary item. The continuous concept space is the convex combination of all embedding vectors.
C C=
{ IVI
k=1 ag E(k) a AIEAI CR
i.e. the set of all probability-weighted mixtures of token embeddings. Note that this is different from the usual semantic space, which modeled as d-dimennional real vector space.
Reasoning Process. Soft Thinking only replaces the intermediate thinking step of the standard CoT. At each step of soft thinking, the model generates concept token defined by Definition [ Then, in the next step, the concept token et is injected back into the LLM by the embedding of concept token:
[V]
IV]
Enext Σ ct(kk] e(k) = Σ p[k]e(k) E EC
(5)
1=1
1=1
When the most probable token for certain concept token is the end-of of-thinking, the intermediate reasoning process stops, and the model switches to generating the output. All output stage tokens Yj are sampled in the usual discrete manner, only the intermediate thinking phase flows through the continuous concept space defined above.
Why Soft Thinking Helps. Using concept tokens allows the model to avoid making hard decisions too early. Instead of selecting a single token each step, the model keeps the full probability distribution over vocabulary. This gives it the flexibility to explore different reasoning paths, especially when it's unsure. By working this continuous concept space, the model can represent more abstract concepts that don't 't map cleanly to single word. These abstract concepts can later evolve into more concrete thoughts as reasoning continues (see Figure This flexibility helps the model think more clearly, avoid early mistakes, and better handle complex multi-step problems.

---

Cold Stop While concept tokens enable more abstract reasoning, feeding in continuous concepr tokens during inference places the model in an out-of-distribution (OOD) regime. This can lead to model collapse if the reasoning process continues for too long without correction. To mitigate this, we propose Cold Stop mechanism that dynamically stops intermediate reasoning when the model becomes overconfident. At each step, we compute the entropy of the concept token:
(p)) = Σ [k] log p(g]
Since Soft Thinking preserves the entire probability distribution at each step, the entropy serves as natural signal for uncertainty, which is often used in LLMs evaluate the quality of generation [29] Low entropy, typically represents "cold" in physics, indicates that the model is confident in its prediction (30, and thus can conclude soon. Given an entropy threshold and required number of consecutive confident steps k, we apply the following rule:
If (p) < T, increment a low-entropy step counter; otherwise, reset the counter.
When the counter reaches we insert an end-of- of-tin-hiin token (/think) to conclude reasoning and begin final answer generation.
This strategy avoids unnecessary computation and prevents the model collapse under OOD conditions, while preserving the benefits of soft thinking through entropy- -baed confidence measure.
Complexity Analysis. Soft Thinking incurs only two lightweight additions to standard CoT. When computing each concept token embedding, first apply top-k top-p filter to the token distribution to remove low-probability noise, select top-n tokens with hightest probability, renormalize, and then perform single dense matrix-vector multiplication over the filtered subset, resulting in O(n d) computational cost per reasoning step (where the embedding dimension). Calculating the entropy for Cold Stop requires O(VV)) time, but adds negligible overhead compared to model forward pass.
3.3 Theoretical Analysis
In this section, we provide theoretical analysis showing how Soft Thinking approximates the full path-summation of standard Chain-of-TT Chain-of-Thought (CoT) by iteratively constructing linear surrogate representations via concept tokens. We begin by rewriting the exact expansion of the marginal likelihood and then derive sequence of linearization steps that culminate in the continuous concept token approximation.
Exact Sath-Summation. Let = Tii denote input context and (iom the first m intermediate reasoning tokens. The true probability of a final answer y V1:1 is obtained by marginalizing over all possible reasoning trajectories of length m:
Σ p(fm x 22t1mm1) p(u x t1m) )) I'
ply xx) Σ p(t |x)
Σ p(2 xx,t)
(7)
This expansion entails an exponential number of paths, since each summation runs over the full vocabulary V.
First-Order Linearization.
Focusing on the outermost summation,
p(u x)) Σ p(t x)p(y) x. f1)
(8)
Viewing the sampled token é as an one-hot vector é E {0, 131 with Σn f1.6 1, its expectation under the multinomial distribution is the first concept token:
ct = B[t]] = Σ p(l (2)4 10:4-p(-11 E AVV-L
(9)
which aligns the concept token in Definition
5

---

By linear approximation of p(y a. around the mean, we obtain ply x) Ep(ti 2)) ply x, (1) = py x. Σ p(t 1)) = p(y I, ct).
(10)
b
Thus, the full outer summation is replaced by single evaluation at the concept token ct1
We now apply the same linearization recursively. Given ct1, the conditional probability expands as
Recursive Approximation.
ply 22 ct1 = Ep(tz p(t2 x, cti) cty) e,,t,,t) t2) = p(y æ, ct,, ct2),
(11)
1/2
where ct2 Ei p(tz x, ct2)
Repeating this process for all m steps yields the continuous expansion:
ply x:) = ply 2, ct1, .t2,...,ccmm
(12)
Comparison to Standard COT. contrast, discrete CoT replaces each summation Σ., p(t) 1-)
with sampling a single token, thereby discarding mass from all other paths. Soft Thinking preserves the full probability distribution at each step through concept tokens, collapsing the exponential path-summation in Eq. iito single forward pass under a sequence of linear approximations.
4 Experiments & Results
4.1 Experiment Setup
Benchmarks. We conduct a comprehensive evaluation of our method on eight benchmark tasks, including Math500 [31, AIME 2024 [32], GSM8K [33], and GPQA-Diamond [34] in the math- ematics domain, well as HvmanEval (35), MBPPMBPP (36), and LiveCodeBench [37] in the
programming domain. Detailed descriptions of these benchmarks are provided in Appendix A.2
Models. We select three widely used open-source LLMs: QwQ-32B 13.. DeepSeek- I-Distill- ---SSeek----istill----------
Qwen-32B (38), and demonstrate the effectiveness and generalizability of the Soft Thinking approach across different model scales (32B and 70B), model architectures (Qwen and LLaMA), and training paradigms (QwQ- 32B is trained with reinforcement learning, while the DeepSeek models are trained via supervised distillation).
38 This diverse selection is designed to
Baseline Methods. We evaluate the performance of Soft Thinking by comparing with two representative baselines. These baselines include Standard CoT Thinking, which employs explicit step-by-step reasoning, and Standard Greedy CoT Thinking, which utilizes greedy decoding at each step of the reasoning process.
Metrics. We use the Pass@) metric to evaluate the accuracy of the model's generated answers. The formula for computing Pass @k is as follows:
Pass @k =
(13)
where n the total number of samples (e.g., 16), c is the number of correct samples, and is the num- ber of samples selected (we set k 1). Therefore, Pass @ 1 g Besides, evaluate the model's reasoning efficiency by reporting the number of tokens generated specifically for correct solutions. These metrics allow comprehensively evaluate the trade-off between computational cost. and performance across different methods.
Implementation Details. We reuse the model's existing embedding matrix without any extra parameters or layers as concept tokens, and the Cold Stop controller monitors decoder entropy and emits an ond-of-thinkin marker when triggered. No model weights update, architecture change, or additional training procedures, Soft Thinking can be plugged into the CoT pipeline of any LLM with minimal engineering effort. We implement our Soft Thinking on SGLang 339, enabling fast inference (see Appendix AA.3 implementation details). We evaluate our method on a server equipped with eight NVIDIA H100 80GB GPUs.

---

Accuracy 1
Generation Length
MATH
500
AIME 2024
GSMSK
GPQA Diamond
Avg.
MATH
500
AIME 2024
GSMSK
Diamond
GPQA
Avg.
Q2Q-328
13
CoT Thinking
97.66
12080
Col Thinking (Gireedy)
97.00
80.00
96.57
65:15
84.68 LT 0.84)
3827
1 11086
1536
7417
5967 (1 7.8%)
Soft Thinking
98.00
83,33
96.81
67.17
86,32 (T 2.48)
3644
10627
1391
7213
5719 (L 11.6%)
DeapSeok-RR-- -a---eea------tttt1----eee
CoT Thinking
94,50
72.08
95.61
63.10
81.32
3543
9347
875
6218
4995
CoTl Thinking (Cärcedy)
95.00
95,30
59.09
77.68 (1 3.64)
8050
1048
8395
Soft Thinking
95.00
83.03 (1 1.71)
6620
(L 22.4%)
DeepSenlz-Ri -a-.aal-------22.211..laa--a-e
38
CoT Thinking
$1.31
CoT Thinking (Greedy)
94,61
73.33
93.60
66.16
81.92 (1 0.61)
2877
9457
606
4443
4345 (L 3.1%)
Soft T Thinking
94,80
73.33
94,90
66.66
82.42 (1 1.11)
3021
6644
597
4470
3683
( 17.9%
Table 1: Comparison of Soft Thinking and various baseline methods accuracy and generation length across mathematical datasets. Best results are highlighted in bold.
Accurncy
Generation Length L
HumanEval
MBPP
LiveCodBencch
Avg.
HumanEval
MBPP
LiveCod-Bench
Avg.
OxQ-328
[3]
Thinking
2557
4899
CoT Thinking (Giredyy
95.73
96.50
57.
83.19 (1 2.51)
2396
2069
70.34
3833 (L 21.8%)
Soft Thinking
98.17
97.66
62.72
86.18 0.48)
2638
2157
7535
1110 16.1%)
DeepSeek- h1 Be--Beek-RiDiettilll Diettilllll-ee Dietill-Qwen-B22 Qwen-32B
68
CoT Thinking
97.25
95.13
$7.33
83.23
3098
2761
8376
4744
CoT'Thinking (Gireddy)
43.36
10.53)
2294
1703
2900 38.9%)
Seft Thinking
97.56
95.33
$9.50
84.13 0.90)
2713
2534
6255
3834 () 19.1%)
38
Thinking
56.94
2711
4472
CoT Thinking (Greedy)
92.07
91.82
48.02
77 37
() 5.84)
2192
1979
5438
3203 () 28.3%)
Soft T Thinking
98.17
94.94
58.42
83.84 0.70)
2498
2214
6512
3741 16.3%)
Table 2: Comparison of Soft Thinking and various baseline methods accuracy and generation length across two coding datasets. Best results are highlighted in bold.
4.2 Hyper-paramett Settings
For all experiments, the maximum generation length was set to 32, 768, the temperature to 0.6, top-k to 30, and top-p to 0.95, unless specified otherwise. The Standard CoT baseline was evaluated using 16 samples per problem to calculate Pass accuracy, whereas the greedy CoT approach utilized a temperature of 0 with single sample.
For Soft Thinking, the concept token was determined using the top-n tokens, where n e 5, 10, 15, 20, 30, along with an entropy threshold 7 chosen from 0.01, 0.05, 0.1, 0.2 and length threshold is selected from 128, 256, 512, 1024. All other settings were kept consistent. We find that 15 yields the best performance for QwQ-32B [13,, while n = 10 is optimal for DeepSeek-RR R1 Results are reported based on the best-perforning combinations of r and k.
models 38
4.3 Results and Analysis
We present the quantitative evaluation results of Soft Thinking and other baseline methods on mathematical and coding datasets in Table and Table 2 respectively.
Improved Pass@1 1 Accuracy. Our proposed Soft Thinking consistently enhances Pass @ accu- racy across all evaluated math and coding benchmarks, demonstrating its broad effectiveness and generalization ability. For instance, on mathematical reasoning tasks, the QwQ-32B model's 's average @ass@1 1 improves from 83.84% (CoT Thinking) to 86.32% (Soft Thinking). representing a notable gain of 2.48% points. On the challenging AIME2024 dataset. the improvement reaches 6.45% and DeepSeek-RI-Disee --siee------issi----aaa------
points. Similarly, for DeepSeek-RP Dissee--RR--issill---eeee increases by 1.71% 1% and 1.11% points, respectively. On coding benchmarks, Soft Thinking also achieves consistent improvements: QwQ-32B sees a 0.48-point increase in average Pass 1, while and DeepSeck-RI- -I--st-tl----i--Lana-----
Pass 1
improve by 0.90% and 0.70%
7

---

Question: 43 34-?
CoT Thinking (157 tokens):
Firet, I recoggize that multiplying 43 by 34 can be simplified by breaking down the numbers into more manageeble parts. I'll decomposse into and IninNext, I'll multiply by 30, account for the 10, resulting in 1, Ansohnnn I'll multiply 43 by which equals 172.ininFinalll, I'll add the two results together: 1,250 plus 172 gives me the finel answer of 1,462.
times can finst multiply by 3 to get 129, then zero
Soft Thinking (96 tokens):
To solve multiplied by 34, I'll start by breaking ng down the numbers into their place values. Vniniirst, I'll multiply 43 by 4, which gives me 172.nninNext, I'll multiply 43 30, resulting 1,290. vnn..lllii, I'll add these two products together: 172 plus 1,290 equals 1,462.
Figure 3: A comparison between standard CoT and Soft Thinking on a multiplication problem. We select the token with the highest probability at each step of Soft Thinking for readability and interpretability. Full distribution is visualized in Figure 4 Red text denotes repetitive, useless words.
points, respectively. These results demonstrate that Soft Thinking provides robust accuracy gains across both math and code domains.
Token Efficiency. A key advantage of Soft Thinking is its significant reduction in generation length, leading to improved token efficiency. On mathematical reasoning benchmarks, Soft Thinking reduces token usage for QwQ-32B by 11.6%,
by 22.4%, and DeepSeek- RI-Distill-Llama-70B by 17.9% compared to standard CoT thinking. Similar trends are observed in and DeepSeek-RR R1-Distilll-Laamaae token usage is reduced by 16.1% %, 19.1%, and 16.3% 3,, respectively. This substantial reduction in token usage not only translates to lower computational and inference costs, but also indicates that the model can reach correct answers through more concise and efficient reasoning. Such token efficiency is particularly valuable for real-world applications, where cost, speed, and scalability crucial.
D--SSee-R--Distil-----
coding tasks: for QwQ-32B, BeepSeck-RI-Distilll
Analysis. Our experimental results demonstrate that Soft Thinking achieves simultaneous improve- ments in both reasoning performance and token efficiency across a diverse set of mathematical and coding benchmarks. This dual gain highlights key advantage of our approach: by leveraging concept tokens that encode richer semantic information at each reasoning step, the model is able to represent and process more abstract or composite ideas within single token. As a result, fewer reasoning steps are required to reach the correct solution, directly translating reduced token usage. Results also stand in stark contrast to the greedy decoding baseline, which, while also reducing token usage, suffers from substantial drops in accuracy, particularly on complex code generation tasks where the loss of diversity in reasoning paths leads to premature convergence on suboptimal solutions.
These findings suggest that the superior efficiency of Soft Thinking is not simply result of more aggressive pruning or shortcutting, but rather reflects a fundamental enhancement in the model's reasoning process. By maintaining the full probability distribution over possible next tokens at each step, our method allows for a "soft" aggregation of multiple reasoning trajectories, effectively broadening the model's 's exploration space without incurring the combinatorial explosion of explicit enumeration. This enables the model to make more informed and confident decisions earlier in the reasoning chain, reducing unnecessary detours and redundant steps.
Overall, the results provide strong evidence that Soft Thinking breaks the traditional trade-off between performance and efficiency in large language model reasoning. Instead of sacrificing one for the other, our approach inherently boosts both, offering more powerful and concise reasoning framework that is readily applicable to wide range of tasks and model architectures.
4.4 Qualitative Results
Visualization of Shortened Examples. Figure demonstrates the comparison between Standard CoT and Soft Thinking. We select the token with the highest probability at each step of Soft Thinking for visualization. can be seen that Soft Thinking has high readability and interpretability. While both methods arrive at the correct answer (1, 462). Soft Thinking produces significantly more concise explanation (96 tokens vs. 157 tokens). This demonstrates Soft Thinking's ability to preserve logical structure while improving token efficiency.
8

---

mitt2ted
bV
5.84
6:93
1.50
1.00
1.00
1.03
1.0
1,03
0.50
0:36
5.07
0.26
C-43
0:33
0.16
6:42
0.50
0.19
14
1
0.24
2.14
1,20
5.10
0.50
6.31
1.20
1.00
1.00
0.45
1.92
1.00
202
1.30
2,02
2.35
6.47
1,00
0.50
0.35
Smeller
2015
ths
de:on
5.95
5.09
0.20
2.45
6.4)
mrake
aeparall
-06
29
45
1
1.50
24
Tese
(WO
prolucEs Toge
0.40
0.03
0.55
1.3
Mia
2.00
1:00
1.03
1.50
00025
1.20
100
1:05
2.00
2.00
E.52
Figure 4: An example illustrating the probability distribution of our proposed Soft Thinking method. At each step, top-k token candidates and their probabilities are shown. Red boxes indicate the selected tokens that form the final generated sequence for readability and interpretability.
Accuracy
Generation Length All ATME LiveCodeBench
Generation Length Correct LiveCndeeeench
2014 1 LiveCodeRench
AIME 2024
COCONUT-TF
0.0
0.0
32,768
32,768
Average Embodding Soft Thinking w/o Cold Stop
6,66
7.49
30,802
30,474
6,556
2,141
73,33
56.98
12,991
13,705
9,457
6,877
Soft Thinking Cold Stop
83.33
62.72
11,445
12,537
10,627
7,535
Table 3: Ablation study of Soft Thinking with QwQ-32B on the AIME 2024 and LiveCodeBench. COCONUT-TF represents training-free COCONUT [27].
Visualization of Embedding Weights. In Figure 4 we present the token probability distributions at each intermediate reasoning step of Soft Thinking. For demonstration, we highlight the top three tokens. During exploratory reasoning phases, such steps 1 1 3, 13 - 14, and 18 20, the token distribution appears more uniform, reflecting the presence of multiple viable paths. contrast, during precise calculations, the token distribution becomes nearly one-hot, indicating that textual elements facilitate path exploration, while numerical components handle exact computations. Notably, at steps 36-37, the model evaluates whether to multiply by 4 or 30, ultimately assigning higher probability to 4. As a result, at step 42, the model selects multiplication by 4. This demonstrates how Soft Thinking integrates path exploration across consecutive concept tokens, thereby enhancing both reasoning flexibility and depth.
4.5 Ablation Study
To comprehensively assess the effectiveness of Soft Thinking, we conducted ablation studies focusing on different strategies for concept token and the impact of the Cold Stop mechanism.
Different strategies for concept token. Specifically, we compared (1) the training-free COCONUT approach [27,, which directly feeds the previous hidden state as the next input embedding; (2) simple average embedding strategy that takes the mean of the top-n token embeddings (we use 5 for ablation); and (3) our Soft Thinking, which computes a probability-weigtted over embeddings. As shown in T we observe that the training-free COCONUT fails entirely, producing no correct answers and always reaching the maximum generation length. The average embedding method performs marginally better, yielding a small number of correct solutions but still suffering from extremely long output. contrast, our Soft Thinking substantially improves both accuracy and token efficiency.
Impact of Cold Stop. We further analyze the Cold Stop by comparing Soft Thinking with and without it. Without Cold Stop, the model is prone to generation collapse, especially due to out-of-

---

distribution (OOD) issues because models have never been trained on concept tokens. It will begin to repeat until it hits the maximum generation length. This OOD-induced collapse significantly increases the averagee generation length across all problems. However, for problems that are solved correctly (typically those requiring shorter reasoning chains), the average length remains relatively low, as these cases are less likely to trigger collapse. When Cold Stop is activated, generation collapse is effectively mitigated, and the model avoids unnecessary exploration along overconfident paths, resulting in significant reduction in average generation length for all problems. Interestingly, Cold Stop allows the model to correctly solve more challenging problems that require longer reasoning chains, the average length for correct solutions increase. Nevertheless, Thinking with Cold Stop not only solves more problems than standard CoT but also achieves greater overall efficiency, confirming the effectiveness of our approach.
demonstrated in Tables and 2 Soft
5 Conclusion
In this work, we present Soft Thinking, a novel, training-free free framework that enables large language models to reason in a continuous concept space by leveraging probabilisticllly weighted concepr tokens instead of traditional discrete tokens. By aggregating information across the entire vocabulary at each reasoning step, our method allows the model to implicitly explore multiple reasoning paths in parallel, leading to both higher accuracy and greater token efficiency. Experiments on mathematical and coding benchmarks demonstrate that Soft Thinking consistently improves pass@) 1 accuracy and reduces generation length, all without any additional training or architectural modifications. Qualitative analyses further show that the reasoning process remains interpretable and concise. Future work may explore integrating training- based approaches to adapt the concept token, with the goal of further improving performance and stability when faced with -uitt--distriuution inputs.
6
Acknowledgments
We would like to express our sincere gratitude to Yue Fan, Saaket Agashe, Liliang Ren, Hao Cheng, Baolin Peng, and Yiping Wang for their valuable feedback and constructive discussions. We thank Orby Al for generously providing the computational resources. Additionally, we appreciate the SGLang Team's assistance during development.
References
[1] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 248377, 2022.
[2] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. 2022.
arXiv preprint arXiv:22
[3] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.1066 2022.
[4] Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang- Towards revealing the mystery behind chain of thought: a theoretical perspective. Advances in Neural Information Processing Systems, 36777777-7777788 2023.
[5] Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representation by single neurons in the human brain. Nature, 435(7045):1102 100---1777111 2005.
[6] Evelina Fedorenko and Rosemary Varley. Language and thought are not the same thing: evidence from neuroimaging and neurological patients. Annals of the New York Academy of Sciences, 1369(1): (1:111))332111 10111122133331 2016.
[7] AA Ivanova, Mineroff, V Zimmerer, N Kanwisher, Varley, and E Fedorenko. The language network is recruited but not required for non-verbal semantic processing. biorxiv, 696484, 2019.
10

---

[8] Yael Benn, Anna A Ivanova, Oliver Clark, Zachary Mineroff, Chloe Seikus, Jack Santos Silva, Rosemary Varley and Evelina Fedorenko. The language network not engaged in object categorization. Cerebral Cortex, 13(11):10380-1100
2023.
[9] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances in neural information processing systems, 36:1 36:11809-11 1822, 2023.
[10] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint ar arXiv:2203.11117 2022.
[11] Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting -istr---isttiittionnn robustness in nip: Benchmark. analysis, and Ilms evaluations. arXiv preprint arXiv:2306.04660 2023.
[12] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint urXiv:2407.2177: 2024.
[13] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025.
[14] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.056633 2023.
[15] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.1222 2023.
[16] Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math- Math-shepherd: Verify and reinforce Iims step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 9426-9439, 2024.
[17] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi- Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint 2ee::2403.0442 2024.
[18] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03331 2024.
[19] Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Efficient training of Ilm policy with divergent thinking. arXiv preprint trXiv:2406.05673, 2024.
[20] Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of coppute-optimal inference for problem-solving with language models. arXiv preprint arXiv:2408.0001 00724, 2024.
[21] Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? arXiv preprint 4626 2024.
[22] Eden Biran, Daniela Gottesman, Sohee Yang, Mor Geva, and Amir Globerson. Hopping too late: Exploring the limitations of large language models on multi-hop queries. arXiv preprint arXiv:2406. 12775, 2024.
[23] Yuval Shalev, Amir Feder, and Ariel Goldstein. Distributional reasoning in Ilms: Parallel reasoning processes in multi-hop reasoning. arXiv preprint arXiv:2406. /3858, 2024.
[24] Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessandro Sordoni. Guiding language model reasoning with planning tokens. arXiv preprint arXiv:2310.05777, 2023.
11

---

[25] Loic Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R Costa-jussà, David Dale, et al. Large concept models: Language modeling in sentence representation space. arXiv preprint arXiv:2412.08821, 2024.
[26] Liang Wang. Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint Sid1:2401.003666 2023.
[27] Shibo Hao, Sainbayar Sukhbaatar, Dilia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. arXiv preprint arXiv:2412.06769, 2024.
[28] Yige Xu, Xu Guo, Zhiwei Zeng, and Chunyan Miao. Softcot: Soft chain-of-thought for efficient reasoning with llms. arXiv preprint arXiv:2502.12I: 2025.
[29] Mengting Hu, Zhen Zhang. Shiwan Zhao, Minlic Huang, and Bingzhe Wu. Uncertainty in natural language processing: Sources, quantification, and applications. arXiv preprint arXiv:2306. 04459, 2023.
[30] Alfréd Rényi. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pages 547-562. University of California Press, 1961.
[31] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 2021.
[32]
AMC. American invitational mathematics examination. http://oooooop//aooooooooooooooa com/ http:/ees.ppptmmmmmmaaaaaaaaaaaa Bat.aaaaaaaaaaatatttaecaaaaaaaaa 2025.
[33] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2 14/68, 2021.
[34] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First Conference on Language Modeling, 2024.
[35] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
[36] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint
2021.
[37] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezana Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint Stt::2403.07974, 2024.
[38] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 2025.
Ilms via reinforcement learning, arXiv preprint
12

---

[39] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Livia Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in Neural Information Processing Systems, 37:62557-62583 2024.
[40] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2335.200550 2023.
13

---

A Appendix
A.1
Limitation
While Soft Thinking demonstrates significant improvements in both reasoning accuracy and token efficiency without requiring any additional training, its training-free nature also introduces certain limitations. Specifically, since current large language models are trained exclusively on discrete token sequences, they have never encountered concept lokens, the wrbbbblltyy-eiighee mixtures of token embeddings, during pre-training or fine-tuning. As a result, feeding such continuous concept tokens into the model during inference places it in out-of-distributie lead to instability or generation collapse, especially when the reasoning chain becomes long or the input distribution diverges from the model's training data. Although our Cold Stop mechanism helps mitigate these issues by terminating reasoning when the model is sufficiently confident, it does not fundamentally resolve the OOD problem. Future work should explore training strategies that explicitly expose the model to concept tokens, thereby aligning its internal representations with the continuous concept space and improving robustness and generalization under soft thinking paradigms.
(OOD) regime. This can
A.2 Benchmarks
The evaluation covers four mathematical benchmark tasks and three programming benchmark tasks.
Mathematical Benchmarks: Math500 40 is diverse subset of 500 problems selected from the MATH dataset (31), covering seven mathematical disciplines. AIME 2024 [32] is drawn from the American Invitational Mathematics Examination, this dataset features challenging problems that serve as rigorous benchmark for both accuracy and token efficiency. GSM8K 33 a benchmark comprising 1,319 grade-school-level math word problems designed to assess multi-step arithmetic reasoning. EPQA-Diamont [34 is subset focusing on high-difficulty problems requiring deep. multi-step reasoning.
Coding Benchmarks: HumanEval 35 is widely used benchmark for evaluating functional correctness in code generation. Each problem consists of Python function signature and natural language description, with correctness measured by executing unit tests. MBPP (36) contains introductory-level coding tasks paired with simple specifications and measures a model's ability to understand and execute basic programming logic. BiveCodeBench (37 is a dynamic, contamination- free code generation benchmark. For our study, we select 279 problems released between August 2024 and January 2025 to ensure evaluation on unseen tasks.
A.3 Implementation of Soft Thinking on SGLang
In this appendix, we describe the engineering modifications made to the SGLang inference engine (v0. .4.6 post1) support our proposed Soft Thinking method. We highlight the core code changes, affected files, and provide high-level code snippets to clarify the new reasoning flow.
A.3.1 Overview of Modifications
Our implementation introduces new inference mode, soft thinking, in which intermediate reasoning steps use "concept tokens" (i.e., full token probability distributions) rather than discrete token ids. This required changes to the input/output interface. sampling, embedding, and state management in SGLang. The main modifications are summarized as follows:
Configuration: New flags and parameters enable soft thinking and control its behavior.
Sampler: Modified to output top-k probability distributions (concept tokens) instead of single sampled token.
Embedding Layer: Added support for probability-weighted interpolation of token embed- dings.
Forward Pipeline: Adapted to accept and propagate concept tokens through the model.
Cold Stop: Entropy- based carly stopping logic for intermediate reasoning steps.
14

---

A.3.2 Key Files and Logic Changes
1. model_ model_config py & server_args.
Purpose: Add configuration options for soft thinking.
Key additions:
enable_ sonplle_sfftthhnieeee thinkingg: bool max topk: int # Command line flags: --enable thinkkng --man topk, think end str
2. sampler py (Sampling Logic)
Purpose:
Output concept tokens (top-k probabilities and indices) instead of only discrete token ids.
Key changes:
if shable_soft_thikkiee
Compute top-k probabilities and indices
topk.probs, topk_indices torch. topk (probs, k-max_topk, dim=-1)
Normalize
topk_probs topk.proba / topk.probs... (dim=-1, keepdim=
True)
logits_ topk-probs topk_probs logits_outpur topk indices indicindices For nent token id, use argmaz or sample from topk as needed
batch token, ids topk_indices 0]
10
else:
Standard discrete sampling batchh_nxx_tooem_idd torch. argmax (probs, -1)
13
14
Entropy calculation for Cold Stop entropy -torch. sum (probs torch log (probs clamp
15
dim=-1)
Nogits_output.entaa entropy entropy
3. vocab, vocab_ppaaallal www.parlllllembeddnnnn py (Embedding Layer)
Purpose:
Support probability-weightedd embedding computation.
Key changes:
def weighted_.fovarddssll topk_probs, topk_indices) Compute weighted sum of embeddings
topk jere_eemeddigns self qquat__metiii embedding (self,
fopk_indices.len (ong)) # [8, DJ
Normalize probabslities
topk_probs topk_probs / topk,probs sum (dim=-1, keepdin True)
eww_ddmedd = torch sum (topk unsqueeze (-1)
topk_ embeddings, dim=1)
return nev_embedding
models/ 11ama. py, mwwmmmolelle......mmmm py (Model Forward Pass)
Purpose: Accept and process concept tokens as input.
Key changes:
15

---

In model. oodel.forwards
forward_batch, topk_probs is not and forvard_batch. topk_indices is not None:
if self self. tp_size 1:
hidden_statan self embed, tokens forbbeddforwadddee
bervard_batchl topk.probs, forward batch topk_indices
else:
hidden_states = self embed_ embed_tokens forward_ topk_probs, arward_batccl topk_indiee
tokens. weighted
elif
is None: hidden_states self enbed_tokens (input_ids)
5. schedule_ batch. py, scheduler.
py, .y, Purpose: State management and output tracking for soft thinking.
Key changes:
# Pssedocode for Cold Stop
if entropy entropy- threshold: low_entropy. stepe else:
stepa
low coo_eenrroy-stepa
if low_entropy_steps >= length_thresholdd
Insert end thinking token, switch mode
self outputu
self sampling. think_end,str.ff
6. sampl paramm py, sampling.batch_ingg py
Purpose: Add soft thinking-specific parameters and per-batch flags.
Key changes:
Parometers for after- thinking sampling, entropy thresholds,
etc.
hww..sstooing.eetrroyttteeeeeeee float
sarly_stopping thresholrre int soft_thinkkig_mood Optional [torch. Tenaor]
A.3.3 High-Level Soft Thinking Inference Flow
1. Initialization: If enable_soft, thinking is set, the model enters soft thinking mode for reasoning steps.
2. At Each Reasoning Step:
The sampler outputs a concept token: top-k probabilities and indices (not just a single token id).
The embedding layer computes the next input embedding weighted sum over token embeddings, using these probabilities.
The model forward pass consumes this weighted embedding.
3. Cold Stop (Early Termination):
At each step, compute the entropy the concept token.
If entropy is below threshold for several consecutive steps, insert the end-of-thinking token to terminate reasoning.
4. Answer Generation: After reasoning. the model switches back standard discrete decoding for the answer.
The above changes enable SGLang to support soft thinking as described in our paper, allowing for continuous, distributional reasoning and entropy- early stopping, all with minimal overhead to the standard inference pipeline.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: These sections comprehensively outline the research objectives, methodology, and expected outcomes, closely aligning with the in-depth analysis and results presented in the full text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: See Section 3.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have added all data and code to the supplementary materials to facilitate reviewers to reproduce our work. We have detailed all experimental details (e.g., model selection, hyperparameters, and experimental equipment) in Sections[4.1]aa a. the paper. Besides, we have implemented our code in the popular open-sourgs source inference framework SGLang and highlighted the key code content in the appendix A.3 to facilitate reproduction. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We add all the data, and code results the supplementary material, which makes easy reproduce
work. We make them public GitHub. Besides, have implemented our code in the popular open-source inference framework SGlang and highlighted the key code content in the appendix A.3 to facilitate reproduction.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.co ctltps.cc/pbbi hubbli//guuddss/ddeuuuiiiii
www.c//gwgeeeeeeee Code//bmeeeeeeee SusSSuumissss citammsssios sssisssssiccciiiii for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have detailed all experimental details (e.g., model selection, hyperparame- ters, and experimental equipment) in Sections 4.1 and 4.2 of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports averaged results over 16 runs to ensure stability, with detailed information provided in Section 4.2. This adequately supports the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have detailed all computer resources (e.g., type of GPUs) in Sections 4.J of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The data we used is all open source, including mathematics and code, and does not contain any content that violates ethical requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
20

---

Our paper does not specifically discuss potential positive or negative societal impacts, as its primary focus is on methodological innovation and technical evaluation. This section is not directly applicable to our work.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not describe specific safeguards for responsible release, as does not involve the release of new high-risk models or sensitive datasets. This section is not applicable to our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, all creators and original owners assets (such as code, data, and models) used in Section[4.Ijor our paper properly credited through citations. We use only publicly available resources with appropriate licenses, and all terms use are respected and explicitly referenced in the manuscript.
21

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
No new assets are introduced in our paper. Therefore, this section is not applicable.
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing experiments research with human subjects. Therefore, this section is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
22

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLM to check syntax and calibrate typos, and not for other purposes.
Guidelines:
The answer NA means that the core method development in this research does not non
involve LLMs as any important, original, non- standaaddard components.
Please refer to our LLM policy for what should or should not be described.
23


Reviews:
[{"review_id": "IVw0srMbrs", "reviewer": "Reviewer_zaBE", "summary": "This paper proposes to modify the inference process in generating CoT tokens by using a soft mix of embeddings instead of sampling the next token and then embedding it. The mixture weights are based on the output distribution, but with top-p and top-k constraints and renormalization. In order to make it work in practice, this work also introduced \"Cold Stop\", where when the entropy is lower than a fixed threshold for a set number of steps, they stopped using soft embeddings and revert back to normal decoding. Experiments on multiple recent state-of-the-art language models show that the proposed approach slightly improves reasoning accuracy with lower inference cost.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is training-free and does not require finetuning the model.\n2. The experiment results show that the proposed method can improve accuracy while using fewer CoT tokens for reasoning.\n\nWeaknesses:\n1. My primary concern is how practical the method is, due to the many hyperparameters introduced. For example, there's top-p, top-k, top-n, entropy threshold $\\tau$, and length threshold $k$. This paper stated that \"Results are reported based on the best-performing combinations of $\\tau$ and $k$\", but without a sensitivity analysis, it is hard to know whether we can use this method in practice. For example, if sensitivity analysis found that the improvements are observed across a large spectrum of hyperparameters, this method could be adapted in all language model decoding implementations, which would be a significant contribution (and I'd significantly increase my score); on the other hand, if it only works for specific combinations of hyperparameters (or the hyperparameters heavily depends on the dataset and the model), then the performance gains might more likely come from hyperparameter tuning.\n2. Table 3 shows that Cold Stop significantly improves performance of soft thinking, but why not directly apply cold stop to the baseline normal decoding? Does the length reduction (or even accuracy improvement) come from cold stop itself?\n3. Some related works in \"Continuous Space Reasoning\" are missing, such as https://arxiv.org/abs/2311.01460.", "was_revised": true, "final_justification": "Apologies for my delay. The author response does address my primary concerns:\n\n1. Regarding how sensitive the method is with respect to hyper-parameters, the added experiments fully address my concerns.\n\n2. Regarding cold stop, after reading authors' explanations, I think I misunderstood the method in my initial review.\n\nGiven this, I'm increasing my rating to 5.", "rebuttal": "We sincerely appreciate the reviewer’s thorough and constructive feedback on our paper. Your valuable insights have helped us identify areas for improvement and refine our work further. Below, we provide detailed responses to all the questions and concerns raised.\n\n---\n\n## **1. Hyperparameter Sensitivity Analysis**\nWe conducted hyperparameter sensitivity analysis on QwQ-32B and DeepSeek-Distill-Qwen-32B across AIME2024 and LiveCodeBench datasets. \n### **1.1 QwQ32b (aime2024)**\n\n#### **Length = 256, Varying Entropy**\n\n| Entropy      | 0     | 0.01  | 0.05  | 0.075 | 0.1   | 0.2   | 0.3   | 0.5   |\n| ------------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 73.33 | 80.00    | 76.66 | 83.33 | 83.33 | 83.33 | 83.33 | 80.00    |\n| Tokens       | 9457  | 11428 | 9910  | 10537 | 10514 | 11045 | 11873 | 10611 |\n\n#### **Entropy = 0.1, Varying Length**\n\n| Length       | 64   | 128   | 256   | 512   | 1024  |\n| ------------ | ---- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 80.00   | 80.00    | 83.33 | 83.33 | 73.33 |\n| Tokens       | 9933 | 10433 | 10514 | 11228 | 9027  |\n\n---\n\n### **1.2 QwQ-32B (livecodebench)**\n\n#### **Length = 512, Varying Entropy**\n\n| Entropy      | 0     | 0.01  | 0.05  | 0.075 | 0.1   | 0.2   | 0.3   | 0.5   |\n| ------------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 60.93 | 62.72 | 62.72 | 62.72 | 62.72 | 62.36 | 60.93 | 60.93 |\n| Tokens       | 7378  | 7564  | 7535  | 7463  | 7808  | 7591  | 7630  | 7295  |\n\n#### **Entropy = 0.05, Varying Length**\n\n| Length       | 64    | 128   | 256   | 512   | 1024  |\n| ------------ | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 60.21 | 60.21 | 62.36 | 62.72 | 62.72 |\n| Tokens       | 7521  | 7652  | 7711  | 7535  | 7820  |\n\n---\n\n### **1.3 DeepSeek-R1-Distill-Qwen-32B (aime2024)**\n\n#### **Length=256, Varying Entropy**\n\n| Entropy      | 0    | 0.01  | 0.05  | 0.075 | 0.1   | 0.2   | 0.3   | 0.5  |\n| ------------ | ---- | ----- | ----- | ----- | ----- | ----- | ----- | ---- |\n| Accuracy (%) | 70   | 76.66 | 73.33 | 76.66 | 76.66 | 73.33 | 73.33 | 70   |\n| Tokens       | 5923 | 6497  | 6275  | 6533  | 6620  | 6377  | 6462  | 6531 |\n\n#### **Entropy = 0.1, Varying Length**\n\n| Length       | 64   | 128   | 256   | 512   | 1024  |\n| ------------ | ---- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 70   | 76.66 | 76.66 | 76.66 | 73.33 |\n| Tokens       | 5890 | 6543  | 6620  | 6888  | 6457  |\n\n---\n\n### **1.4 DeepSeek-R1-Distill-Qwen-32B (livecodebench)**\n\n#### **Length=512, Varying Entropy**\n\n| Entropy      | 0     | 0.01 | 0.05 | 0.075 | 0.1  | 0.2   | 0.3   | 0.5   |\n| ------------ | ----- | ---- | ---- | ----- | ---- | ----- | ----- | ----- |\n| Accuracy (%) | 56.98 | 59.5 | 59.5 | 59.5  | 59.5 | 58.78 | 58.42 | 56.63 |\n| Tokens       | 6877  | 6351 | 6275 | 6199  | 6335 | 6419  | 6253  | 6127  |\n\n#### **Entropy = 0.05, Varying Length**\n\n| Length       | 64    | 128   | 256   | 512  | 1024 |\n| ------------ | ----- | ----- | ----- | ---- | ---- |\n| Accuracy (%) | 53.76 | 56.63 | 58.78 | 59.5 | 59.5 |\n| Tokens       | 6935  | 6139  | 6213  | 6255 | 6402 |\n\n### **1.5 Analysis**\nThe results demonstrate that our method is robust to hyperparameter variations across a large spectrum:\n\n- **Entropy Sensitivity**:  \n  For QwQ-32B on AIME2024, entropy threshold values ranging from 0.075 to 0.3 consistently achieve optimal results, exceeding the baseline. Similarly, for LiveCodeBench, entropy threshold values between 0.01 and 0.2 outperform the baseline. For DeepSeek-32B, the trends are similar, showing robustness across varying entropy thresholds.\n\n- **Length Sensitivity**:  \n  On AIME2024, lengths threshold from 64 to 512 work effectively, while on LiveCodeBench, lengths threshold from 256 to 1024 exceed baseline performance.  \n\nOptimal hyperparameter ranges vary slightly across tasks but share a large overlap and are independent of the models used. Coding tasks require lower entropy and longer lengths threthod to generate longer code blocks, as overly aggressive stopping can truncate meaningful outputs.\n\n**For general settings**, entropy = 0.1 and length = 512 perform well across all datasets and models, consistently improving upon the baseline. In our paper, for optimal results, we selected entropy = 0.1 and length = 256 for math tasks, and entropy = 0.05 and length = 512 for code tasks.\n\n**Therefore, our method has good robustness to parameters and does not require heavy parameter tuning.**\n\nWe will add these important sensitive analysis to camera ready version.\n\n---\n\n## **2. Explanation of Cold Stop Mechanism**\n\n### **2.1 How Cold Stop Works**\nCold Stop does not fundamentally alter the model’s reasoning ability or enable it to produce correct reasoning paths where it previously could not. Instead, its purposes are:  \n1. **Make the answer easy to parse**: By stopping reasoning once the model reaches high confidence, Cold Stop ensures that correct answers are output cleanly and easily to parse by our evaluation scripts automatically.\n2. **Reducing Token Usage**: It halts unnecessary reasoning steps once the model has already reached the correct conclusion, thereby saving tokens.\n\n**Therefore, for those samples where the judgment changed from incorrect to correct after using Cold Stop, the correct answer has already been output before stopping.**\n\nIn addition, since generation rarely happens in baseline (CoT), Cold Stop will not improve the accuracy.\n\nBelow is a simple example to illustrate how Cold Stop works. We only show the top 1 token of Soft Thinking for visualization:\n```\nQuestion:\n1+1=?\n\nWithout cold stop:\n<think>Let me solve this question. … (some thinking) …. Therefore the correct answer is 2. Emmm, let me verify my answer in another way. Let’s Let’s Let’s Let’s Let’s …  (repetition occurs until the maximum number of tokens is reached)\n\nWith cold stop:\n<think>Let me solve this question. … (some thinking) …. Therefore the correct answer is 2. Emmm, let me verify my answer in another way. Let’s Let’s Let’s (cold stop here)</think> The final answer is 2.\n```\nFor the example without cold stop, the correct answer appears in the thinking process but not in the output, and can not be parsed for evaluation. With Cold Stop, we stop the generation collapse properly and let the model conclude so that we can parse the answer.\n\n### **2.2 Improvement from Cold Stop**\nAccuracy improvements stem from Soft Thinking itself, which enables the model to explore abstract and parallel reasoning paths to find correct solutions. Cold Stop primarily serves as a utility to enhance token efficiency and prevent collapse during inference. Because it cannot make the model generate more content, it will instead reduce the generated content.\n\nRegarding token efficiency, we analyzed the token reductions contributed by Soft Thinking and Cold Stop. The following table summarizes the results:\n\n| Method                      | Accuracy (AIME2024) | Length All | Length Correct | Accuracy (LiveCodeBench) | Length All | Length Correct |\n| --------------------------- | ------------------- | ---------- | -------------- | ------------------------ | ---------- | -------------- |\n| CoT (baseline)              | 76.88               | 13449      | 12080          | 62                       | 13743      | 9986           |\n| Soft Thinking w/o Cold Stop | 73.33               | 12991      | 9457           | 56.98                    | 13360      | 6877           |\n| Soft Thinking w/ Cold Stop  | 83.33               | 11445      | 10627          | 62.72                    | 12537      | 7535           |\n\nBoth Soft Thinking (with or without Cold Stop) reduce token usage in terms of total and correct solution length. Cold Stop further mitigates generation collapse, ensuring correct answers are cleanly output while reducing tokens. **As a result, accuracy improvement is from Soft Thinking and efficiency improvement is from both Soft Thinking and Cold Stop.**\n\nFor baseline (CoT), generation collapse (repetition) is rare. Since Cold Stop is a method to let model generate output faster, it will not improve the accuracy.\n\n---\n\n### **3. Missing Related Work**\nWe appreciate the reviewer pointing out missing references. In the camera-ready version, we will update the \"Related Work\" section to include relevant papers such as:\n\n- Implicit Chain of Thought Reasoning via Knowledge Distillation  \n- From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step  \n- Follow-up works building on these ideas  \n\nThank you for highlighting this oversight, and we will ensure these contributions are properly acknowledged.\n\n---\n\n### **4. Paper Formatting**\nRegarding the term \"Visualization of Shorted Example,\" \"Shortened\" is a more precise term to describe the concise reasoning chains generated by Soft Thinking. We will update this phrasing in the final version.\n\n---\n\n### **Acknowledgment**\nWe sincerely respect and appreciate the hard work and professionalism demonstrated in your review. Your thoughtful feedback has been instrumental in improving our paper, and we hope the explanations provided above address all concerns. We look forward to your feedback on our revisions and thank you again for your valuable time and effort."}, {"review_id": "PTzSqEas5u", "reviewer": "Reviewer_m4ot", "summary": "This paper proposes a training-free reasoning approach for large language models that conducts reasoning in the continuous latent space rather than at the discrete token level (in contrast to traditional Chain-of-Thought reasoning). Through comprehensive experiments across multiple benchmarks, the authors demonstrate performance improvements using this continuous space reasoning methodology. The approach aims to leverage the representational richness of the latent space to enhance reasoning capabilities without requiring additional model training.", "questions": "Please refer to the Weaknesses section above", "limitations": "Please refer to the Weaknesses section above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Proposes a training-free reasoning approach that operates in the continuous latent space rather than discrete token space, offering a fundamentally different perspective from traditional Chain-of-Thought (CoT) methods\n- Clear methodological exposition: Provides a clear explanation of how the method differs from original CoT and specifically describes how reasoning is conducted in the continuous latent space. The paper also includes detailed theoretical derivations that support the method design and provide mathematical justification for the approach\n\n\nWeaknesses:\n\n- The paper claims that \"the method preserves the uncertainty and richness inherent in reasoning by aggregating over all possible next steps,\" suggesting that soft thinking can overcome the limited sampling path diversity of traditional CoT. However, the explanation of how continuous latent space sampling enables token diversity and the ability to \"explore multiple reasoning paths in parallel, rather than being limited to a single trajectory\" requires further clarification. The mechanism by which this parallel exploration occurs is not convincingly demonstrated.", "was_revised": true, "final_justification": "I still maintain my original score which has indicated an \"accept\"", "rebuttal": "We truly value the your comprehensive and insightful feedback. Below, we offer detailed replies to your questions.\n\n## **Explanation to Parallel Thinking**\n### **1. From a qualitative perspective**\nSoft Thinking retains the original probability distribution, preserving more information compared to traditional sampling methods. In standard sampling, the probability distribution collapses into a one-hot vector (where the position corresponding to the selected token is set to 1). This increases certainty, as the entropy of a one-hot distribution is zero, leaving no room for uncertainty.  \n\n### **2. From a theoretical perspective**\nSoft Thinking also provides a theoretical explanation for how it enables parallel reasoning.\n\nIn Section 3.3, Equation (10) demonstrates:  \n\n$$\np(y \\mid x) = \\sum_{t_1} p(t_1 \\mid x) p(y \\mid x, t_1) \\approx p\\left(y \\mid x, \\sum_{t_1} p(t_1 \\mid x) t_1 \\right) = p(y \\mid x, ct_1). \\tag{10}\n$$  \n\nHere, we prove that the concept token is a linear approximation of all possible discrete tokens at this step. The full outer summation over the first token is replaced by a single evaluation at the *concept token* $ct_1$ via linear approximation.  \n\nSubsequently, in Equations (11) and (12), we apply the same linearization recursively:\n\nGiven \\( ct_1 \\), the conditional probability expands as: \n\n$$\np(y \\mid x, ct_1) = \\sum_{t_2} p(t_2 \\mid x, ct_1) p(y \\mid x, ct_1, t_2) \\approx p(y \\mid x, ct_1, ct_2), \\tag{11}\n$$\n\nwhere $ct_2 = \\sum_{t_2} p(t_2 \\mid x, ct_1) t_2$.  \n\nRepeating this process for all $m$ steps yields the continuous expansion:  \n\n$$\np(y \\mid x) \\approx p(y \\mid x, ct_1, ct_2, \\ldots, ct_m). \\tag{12}\n$$\n\n\nWe demonstrate that **the multi-step concept tokens ($ct_1, \\ldots, ct_m$) are linear approximations of all possible path combinations of discrete tokens ($t_1, \\ldots, t_m$) at each step**, weighted by their respective path probabilities. As a result, **a single reasoning path composed of concept tokens can approximate all possible discrete token reasoning paths, enabling the exploration of multiple reasoning paths in parallel within a single reasoning trajectory**.  \n\n### **3. Specific Examples**\nAdditionally, as illustrated in Figure 4 of the paper, we observe examples of parallel reasoning in practice. For instance, in steps 13–14, combinations such as \"I’ll start,\" \"I can use,\" and \"I’ll break\" emerge; similarly, in steps 18–20, combinations like \"break down the number into\" and \"break down the multiplication to\" appear. These practical examples further demonstrate the ability of Soft Thinking to conduct parallel reasoning.  \n\nThus, the paper already provides qualitative evidence, theoretical proof, and case studies to explain how continuous space reasoning enables token diversity and facilitates the exploration of multiple reasoning paths in parallel.  \n\nNevertheless, we acknowledge the need for further clarity in this explanation. We will rewrite and improve this section in the revision to address the concerns raised by the reviewer.  \n\n## **Acknowledgment**\nWe genuinely appreciate the diligent and professional nature of your review. Your insightful comments have been invaluable in refining our paper, and we trust that the clarifications offered above adequately address your concerns. We eagerly await your response to our updated submission and thank you once more for your considerable time and effort."}, {"review_id": "uL8xz8qERm", "reviewer": "Reviewer_81RM", "summary": "The paper introduces \"Soft Thinking,\" a training-free method enabling LLMs to reason in a continuous concept space by replacing discrete tokens with probability-weighted concept tokens. This approach allows models to explore multiple reasoning paths implicitly, improving accuracy and reducing token usage. Experiments on math and coding benchmarks show up to 2.48% higher pass@1 accuracy and 22.4% shorter generation lengths compared to standard CoT. The Cold Stop mechanism mitigates generation collapse, and theoretical analysis frames Soft Thinking as a linear approximation of path-summation in CoT.", "questions": "see weakness.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- The text in the image is too small, which is unfriendly to readers.\n- Does Soft Thinking struggle with extremely long reasoning tasks where out-of-distribution (OOD) inputs accumulate?\n- The entropy threshold (τ) and consecutive steps (k) are fixed hyperparameters. For example, using τ=0.01 and k=128 (Section 4.2, lines 242–243) may not generalize to diverse problem distributions, potentially leading to premature termination or redundant steps. How to mitigate and discuss this issue?\n- On LiveCodeBench, Cold Stop reduces generation length but increases the correct solution length (Table 3, lines 325–326), indicating that the stopping conditions for complex code tasks may be suboptimal. Is my understanding correct?\n- The paper compares against standard and greedy Chain of Thought (CoT) but lacks benchmarks against state-of-the-art methods like Tree of Thoughts or self-consistency, which also explore multiple reasoning paths.", "was_revised": true, "final_justification": "The author's reply has basically addressed my concern, and I tend to keep my postive score (4:Borderline accept).", "rebuttal": "We sincerely appreciate your thoughtful and detailed feedback. Below, we provide thorough responses to your questions and concerns.\n\n## **1. The text in the image is too small.**  \nThank you for your suggestion. We will adjust the proportions between the blank space and the text in the images, enlarge the text, and enhance the contrast between the text and the background to improve readability.\n\n## **2. Extremely Long OOD Inputs**  \nSoft Thinking is capable of handling extremely long OOD inputs effectively. For instance, on the AIME2024 dataset, the average reasoning length exceeds 10k tokens, while on LiveCodeBench, it surpasses 7k tokens, with the maximum length reaching over 30k tokens. These results demonstrate that the model performs well on extremely long reasoning tasks.\n\n## **3. Hyperparameter Sensitivity Analysis**\nWe conducted hyperparameter sensitivity analysis on QwQ-32B and DeepSeek-Distill-Qwen-32B across AIME2024 and LiveCodeBench datasets. \n### **3.1 QwQ32b (aime2024)**\n\n#### **Length = 256, Varying Entropy**\n\n| Entropy      | 0     | 0.01  | 0.05  | 0.075 | 0.1   | 0.2   | 0.3   | 0.5   |\n| ------------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 73.33 | 80.00    | 76.66 | 83.33 | 83.33 | 83.33 | 83.33 | 80.00    |\n| Tokens       | 9457  | 11428 | 9910  | 10537 | 10514 | 11045 | 11873 | 10611 |\n\n#### **Entropy = 0.1, Varying Length**\n\n| Length       | 64   | 128   | 256   | 512   | 1024  |\n| ------------ | ---- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 80.00   | 80.00    | 83.33 | 83.33 | 73.33 |\n| Tokens       | 9933 | 10433 | 10514 | 11228 | 9027  |\n\n---\n\n### **3.2 QwQ-32B (livecodebench)**\n\n#### **Length = 512, Varying Entropy**\n\n| Entropy      | 0     | 0.01  | 0.05  | 0.075 | 0.1   | 0.2   | 0.3   | 0.5   |\n| ------------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 60.93 | 62.72 | 62.72 | 62.72 | 62.72 | 62.36 | 60.93 | 60.93 |\n| Tokens       | 7378  | 7564  | 7535  | 7463  | 7808  | 7591  | 7630  | 7295  |\n\n#### **Entropy = 0.05, Varying Length**\n\n| Length       | 64    | 128   | 256   | 512   | 1024  |\n| ------------ | ----- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 60.21 | 60.21 | 62.36 | 62.72 | 62.72 |\n| Tokens       | 7521  | 7652  | 7711  | 7535  | 7820  |\n\n---\n\n### **3.3 DeepSeek-R1-Distill-Qwen-32B (aime2024)**\n\n#### **Length=256, Varying Entropy**\n\n| Entropy      | 0    | 0.01  | 0.05  | 0.075 | 0.1   | 0.2   | 0.3   | 0.5  |\n| ------------ | ---- | ----- | ----- | ----- | ----- | ----- | ----- | ---- |\n| Accuracy (%) | 70   | 76.66 | 73.33 | 76.66 | 76.66 | 73.33 | 73.33 | 70   |\n| Tokens       | 5923 | 6497  | 6275  | 6533  | 6620  | 6377  | 6462  | 6531 |\n\n#### **Entropy = 0.1, Varying Length**\n\n| Length       | 64   | 128   | 256   | 512   | 1024  |\n| ------------ | ---- | ----- | ----- | ----- | ----- |\n| Accuracy (%) | 70   | 76.66 | 76.66 | 76.66 | 73.33 |\n| Tokens       | 5890 | 6543  | 6620  | 6888  | 6457  |\n\n---\n\n### **3.4 DeepSeek-R1-Distill-Qwen-32B (livecodebench)**\n\n#### **Length=512, Varying Entropy**\n\n| Entropy      | 0     | 0.01 | 0.05 | 0.075 | 0.1  | 0.2   | 0.3   | 0.5   |\n| ------------ | ----- | ---- | ---- | ----- | ---- | ----- | ----- | ----- |\n| Accuracy (%) | 56.98 | 59.5 | 59.5 | 59.5  | 59.5 | 58.78 | 58.42 | 56.63 |\n| Tokens       | 6877  | 6351 | 6275 | 6199  | 6335 | 6419  | 6253  | 6127  |\n\n#### **Entropy = 0.05, Varying Length**\n\n| Length       | 64    | 128   | 256   | 512  | 1024 |\n| ------------ | ----- | ----- | ----- | ---- | ---- |\n| Accuracy (%) | 53.76 | 56.63 | 58.78 | 59.5 | 59.5 |\n| Tokens       | 6935  | 6139  | 6213  | 6255 | 6402 |\n\n### **3.5 Analysis**\nThe results demonstrate that our method is robust to hyperparameter variations across a large spectrum:\n\n- **Entropy Sensitivity**:  \n  For QwQ-32B on AIME2024, entropy threshold values ranging from 0.075 to 0.3 consistently achieve optimal results, exceeding the baseline. Similarly, for LiveCodeBench, entropy threshold values between 0.01 and 0.2 outperform the baseline. For DeepSeek-32B, the trends are similar, showing robustness across varying entropy thresholds.\n\n- **Length Sensitivity**:  \n  On AIME2024, lengths threshold from 64 to 512 work effectively, while on LiveCodeBench, lengths threshold from 256 to 1024 exceed baseline performance.  \n\nOptimal hyperparameter ranges vary slightly across tasks but share a large overlap and are independent of the models used. Coding tasks require lower entropy and longer lengths threthod to generate longer code blocks, as overly aggressive stopping can truncate meaningful outputs.\n\n**For general settings**, entropy = 0.1 and length = 512 perform well across all datasets and models, consistently improving upon the baseline. In our paper, for optimal results, we selected entropy = 0.1 and length = 256 for math tasks, and entropy = 0.05 and length = 512 for code tasks.\n\n**Therefore, our method has good robustness to parameters and does not require heavy parameter tuning.**\n\nWe will add these important sensitive analysis to camera ready version.\n\n\n\n## **4. Cold Stop and generation length on LiveCodeBench:**  \n\nThe increase in average generation length for correct solutions with Cold Stop is a result of the model solving more problems, many of which require longer reasoning chains. Specifically, without Cold Stop, the model achieves an accuracy of 56.98%, with an average generation length of 6,877 tokens for correct solutions. When Cold Stop is applied, the accuracy improves significantly to 62.72%, allowing the model to correctly solve more problems, including many that demand reasoning chains exceeding 10k tokens.  \n\nAs a result, the inclusion of these more challenging problems naturally increases the average generation length for correct solutions to 7,535 tokens. This does not indicate that Cold Stop's stopping conditions are suboptimal for complex code tasks. Instead, it highlights Cold Stop helps to handle more difficult problems, which require longer reasoning chains.\n\n## **5. Comparison with Tree of Thoughts and self-consistency:**  \nYour suggestion regarding the lack of comparison with these methods is very valid. The primary reason is that our approach targets both effectiveness and efficiency, while methods like Tree of Thoughts or self-consistency require extensive sampling or searches. For example, self-consistency often requires sampling 16 or 32 times, leading to token usage that is 16 or 32 times higher than the baseline CoT. Moreover, integrating ToT and self-consistency into SGLang demands significant development time. Given the constraints of the rebuttal period and computational resources, we will conduct experiments comparing these methods in the camera-ready version to emphasize the trade-offs between effectiveness and efficiency.\n\n## **Acknowledgment**\nWe genuinely appreciate the diligent and professional nature of your review. Your insightful comments have been invaluable in refining our paper, and we trust that the clarifications above adequately resolve your concerns. We eagerly await your response to our revisions and extend our gratitude once more for your valuable time and effort."}, {"review_id": "RE3ZTCSTRi", "reviewer": "Reviewer_txtk", "summary": "This paper introduces Soft Thinking, a training-free method that enhances the reasoning capabilities of LLMs by operating in a continuous concept space. Instead of discrete token sampling (as in Chain-of-Thought), Soft Thinking generates concept tokens-probability-weighted mixtures of token embeddings- enabling implicit parallel exploration of reasoning paths. Key innovations include:\n1. Concept tokens: Replace discrete tokens with continuous embeddings aggregated from the full vocabulary distribution.\n2. Cold Stop: An entropy-based early termination mechanism to prevent generation collapse under out-of-distribution inputs.\nThe method improves Pass@1 accuracy by up to 2.48% and reduces token usage by up to 22.4% on math/coding benchmarks (e.g., MATH500, HumanEval) across models like Qwen-32B and DeepSeek-70B, while maintaining interpretability.", "questions": "1. OOD Robustness: The paper notes concept tokens place models in OOD regimes (Appendix A.1). How does performance degrade on highly OOD tasks (e.g., non-STEM)? Could lightweight finetuning on concept embeddings mitigate this?\n2. Hyperparameter Sensitivity: Cold Stop uses thresholds τandk . How do results vary with these values (e.g., lowτcausing premature termination)? A sensitivity analysis would clarify robustness.\n3. Scalability to Smaller Models: The method assumes decoupled input/output embeddings (Section 2.2), which is typical for >7B models. Would Soft Thinking work on sub-7B models (e.g., via weight tying)?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "• Strengths:\n1. Model novelty\n(1) Train a free continuous reasoning paradigm: Implement Probabilistic Embedding Propagation (PEP) by directly reusing pre trained word embedding layers (without adding new parameters) to transform discrete inference into a continuous space optimization problem.\n(2) Lightweight Implementation of Implicit Multipath Exploration: By implicitly integrating multiple inference paths through the concept token e (k) in a single forward propagation, the computational complexity is only O (n⋅d) O (n⋅ d) (n≪∣V∣n≪∣V∣).\n2. Rigorous evaluation across 8 benchmarks (4 math, 3 coding) and 3 model architectures (32B–70B).\n3. Ablation studies validate design choices (e.g., Table 3: Soft Thinking vs. average embedding/COCONUT).\n4. Theoretical grounding: Section 3.3 formally links concept tokens to path-summation approximation.\n\n• Weaknesses:\n1. Rationality analysis of model modeling\n(1) Problem 1: Semantic collapse risk in conceptual space\nThe concept token ct (k) is essentially a convex combination of word embeddings, but linear interpolation in high-dimensional embedding spaces may not maintain semantic consistency (such as \"king\"+\"woman\" - \"man\" ≈ \"queen\" only holds in specific directions). Will weighted embedding lead to semantic ambiguity when p is evenly distributed across multiple unrelated concepts (such as activating \"integral\" and \"loop\" simultaneously in mathematical reasoning)?\n(2) Problem 2: The boundary between probability decoding methods is blurred\nThe core operation of Soft Thinking - weighted embedding input - is highly similar to the following methods: COCONUT [26]: Use the hidden state h(t) as the next time input (but require training adapter); DoLa (Chuang et al. 2023, Decoding by Contrasting Layers): Weighted different layer representations generate concept vectors. This paper claims that 'no training is required', but does not prove that its mechanism is fundamentally different from the probability driven decoding mentioned above.\n(3) The incremental innovation of Cold Stop\nEntropy threshold cold stop is essentially a direct application of classical uncertainty control (such as [28]), and Not more efficient than Speculative Decoding (Leviathan et al. 2023) (the latter can skip multiple steps); The connection with Adaptive Computation Time (ACT) has not been discussed.\n2. Limited analysis of failure cases (e.g., when Cold Stop truncates useful reasoning prematurely).\n3. Benchmarks focus on STEM; generality to commonsense reasoning is untested.\n4. Related Works to Discuss\n(1) Comparative discussion: Clearly distinguish the difference in computational efficiency between ToT/GoT/Soft Prompt (e.g. ToT requires 10-100x LM calls, this article only makes a single call).\n[1] Yao et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. NeurIPS.\n[2] Besta et al. (2024). Graph of Thoughts: Solving Elaborate Problems with Large Language Models. ICLR.\n[3] He et al. (2024) Decoding by Contrasting Soft Prompts. NeurIPS.\n(2) Enhance Theoretical analysis: The theoretical foundation supporting entropy threshold.\n[4] Xin et al. (2021). BERxiT: Early Exiting for BERT with Better Fine-Tuning and Extension to Regression. EACL.\n[5] Lin et al. (2024). Calibrating LLM Confidence via Bayesian Inference. ICML.\n(3) Extended experiment: Test Soft Thinking on LeanDojo or PoT tasks to verify its generalization in formal reasoning.\n[6] Chen et al. (2022). Program of Thoughts Prompting: Disentangling Computation from Reasoning. NeurIPS.\n[7] Yang et al. (2023). LeanDojo: Theorem Proving with Retrieval-Augmented Language Models. NeurIPS.", "was_revised": true, "final_justification": "The authors paritially addressed my concerns. Since the original score I gave is positive, It is appropriate, so I will keep the score of 4.", "rebuttal": "We sincerely appreciate your thoughtful and detailed feedback. Below, we provide thorough responses to your questions and concerns.\n\n## **1. Question**\n### **1.1 OOD Robustness**\nSoft Thinking does face OOD issues. **However, the OOD problem here does not refer to tasks or problems. Instead, it specifically refers to the fact that the weighted sum embeddings of concept tokens were not seen during training.**\nWe believe lightweight fine-tuning could mitigate this issue. However, this work focuses on introducing the concept and algorithm itself, leaving robustness optimization as future work.\n\n## **1.2 Sensitivity Analysis**\nWe conducted hyperparameter sensitivity analysis on QwQ-32B and DeepSeek-Distill-Qwen-32B across AIME2024 and LiveCodeBench datasets.\n### **1.2.1 QwQ32b (aime2024)**\n#### **Length = 256**\n| Entropy | 0 | 0.01  | 0.05  | 0.075 | 0.1 | 0.2 | 0.3 | 0.5 |\n| - | - | - | - | - | - | - | - | - |\n| Accuracy (%) | 73.33 | 80.00    | 76.66 | 83.33 | 83.33 | 83.33 | 83.33 | 80.00    |\n| Tokens | 9457  | 11428 | 9910  | 10537 | 10514 | 11045 | 11873 | 10611 |\n\n#### **Entropy = 0.1**\n| Length | 64   | 128   | 256   | 512   | 1024  |\n| - | - | - | - | - | - |\n| Accuracy (%) | 80.00   | 80.00    | 83.33 | 83.33 | 73.33 |\n| Tokens | 9933 | 10433 | 10514 | 11228 | 9027  |\n---\n### **1.2.2 QwQ-32B (livecodebench)**\n#### **Length = 512**\n| Entropy | 0 | 0.01  | 0.05  | 0.075 | 0.1 | 0.2 | 0.3 | 0.5 |\n|-|-|-|-|-|-|-|-|-|\n| Accuracy (%) | 60.93 | 62.72 | 62.72 | 62.72 | 62.72 | 62.36 | 60.93 | 60.93 |\n| Tokens | 7378  | 7564  | 7535  | 7463  | 7808  | 7591  | 7630  | 7295  |\n\n#### **Entropy = 0.05**\n| Length | 64 | 128 | 256 | 512 | 1024 |\n|-|-|-|-|-|-|\n| Accuracy (%) | 60.21 | 60.21 | 62.36 | 62.72 | 62.72 |\n| Tokens | 7521  | 7652  | 7711  | 7535  | 7820  |\n---\n### **1.2.3 DeepSeek-R1-Distill-Qwen-32B (aime2024)**\n#### **Length=256**\n| Entropy | 0 | 0.01  | 0.05  | 0.075 | 0.1 | 0.2 | 0.3 | 0.5  |\n|-|-|-|-|-|-|-|-|-|\n| Accuracy (%) | 70 | 76.66 | 73.33 | 76.66 | 76.66 | 73.33 | 73.33 | 70   |\n| Tokens | 5923 | 6497  | 6275  | 6533  | 6620  | 6377  | 6462  | 6531 |\n\n#### **Entropy = 0.1**\n| Length | 64 | 128 | 256 | 512 | 1024  |\n|-|-|-|-|-|-|\n| Accuracy (%) | 70 | 76.66 | 76.66 | 76.66 | 73.33 |\n| Tokens | 5890 | 6543  | 6620  | 6888  | 6457  |\n---\n### **1.2.4 DeepSeek-R1-Distill-Qwen-32B (livecodebench)**\n#### **Length=512**\n| Entropy      | 0     | 0.01 | 0.05 | 0.075 | 0.1  | 0.2   | 0.3   | 0.5   |\n|-|-|-|-|-|-|-|-|-|\n| Accuracy (%) | 56.98 | 59.5 | 59.5 | 59.5  | 59.5 | 58.78 | 58.42 | 56.63 |\n| Tokens | 6877  | 6351 | 6275 | 6199  | 6335 | 6419  | 6253  | 6127  |\n\n#### **Entropy = 0.05**\n| Length | 64 | 128 | 256 | 512  | 1024 |\n|-|-|-|-|-|-|\n| Accuracy (%) | 53.76 | 56.63 | 58.78 | 59.5 | 59.5 |\n| Tokens | 6935  | 6139  | 6213  | 6255 | 6402 |\n\n### **1.2.5 Analysis**\nThe results demonstrate that our method is robust to hyperparameter variations across a large spectrum:\n- **Entropy Sensitivity**:  \n  For QwQ-32B on AIME2024, entropy threshold ranging from 0.075 to 0.3 consistently achieve optimal results, exceeding the baseline. Similarly, for LiveCodeBench, entropy threshold values between 0.01 and 0.2 outperform the baseline. For DeepSeek-32B, the trends are similar, showing robustness across varying entropy thresholds.\n- **Length Sensitivity**:  \n  On AIME2024, lengths threshold from 64 to 512 work effectively, while on LiveCodeBench, lengths threshold from 256 to 1024 exceed baseline performance.  \n\nOptimal hyperparameter ranges vary slightly across tasks but share a large overlap and are independent of the models used. Coding tasks require lower entropy and longer lengths threthod to generate longer code blocks.\n\n**For general settings**, entropy = 0.1 and length = 512 perform well across all datasets and models, consistently improving upon the baseline. In our paper, for optimal results, we selected entropy = 0.1 and length = 256 for math tasks, and entropy = 0.05 and length = 512 for code tasks.\n\n**Therefore, our method has good robustness to parameters and does not require heavy parameter tuning.**\n\nWe will add these important sensitive analysis to camera ready version.\n\n## **1.3. Smaller Models**\nAdditional experiments on Deepseek-distill-qwen series models:\n| Model Size | Method | Pass@1 Accuracy (%) | Token Usage |\n|-|-|-|-|\n| 7B| CoT | 55.00 | 10,502 |\n|  | Soft Thinking  | 49.17 | 15,479 |\n| 14B  | CoT | 63.33 | 9,562 |\n|  | Soft Thinking  | 63.33 | 7,705 |\n| 32B | CoT | 72.08 | 9,347 |\n| | Soft Thinking  | 76.66 | 6,620 |\n\nOur experiments reveal that Soft Thinking does not significantly outperform the baseline on smaller models. However, **as model size increases, Soft Thinking gradually surpasses the baseline.** This is due to the following reasons:  \n1. In smaller models, even embeddings of semantically distant tokens may not form angles close to 90°, resulting in a higher proportion of irrelevant tokens in the probability distribution and increased noise.  \n2. Concept tokens are OOD, requiring the model to possess strong generalization abilities to understand embeddings derived from linear interpolation.  \n\nMoreover, our work addresses the issue faced by methods like COCONUT, which rely heavily on weight tying and therefore fail to work effectively on larger models.\n\n## **2. Weakness**\n### **2.1 Problem 1: Semantic Collapse Risk**  \nThank you for your insightful observation. To clarify, Soft Thinking does not aggregate embeddings of tokens with vastly different semantics. The weighted probabilities are derived from the hidden state multiplied by the LM head and processed through a softmax function. This ensures that tokens with higher weights are semantically closer to the hidden state, while tokens with distant or opposite semantics have near-zero probabilities in the distribution.  \n\nFor instance, combinations like \"I'll start\" and \"can use\" at positions 13 and 14 (as shown in Figure 4) represent semantically consistent reasoning paths. In contrast, unrelated concepts such as \"integral\" and \"loop\" would not appear simultaneously in the same token. We hope this explanation alleviates concerns about semantic collapse risk. We will include this clarification in the revised paper. Thank you for pointing it out.\n\n### **2.2 Problem 2: Similarity to Other Methods**  \nWe want to clarify that our approach is fundamentally different from these works. Soft Thinking innovatively extends LLM reasoning into continuous space without requiring additional modules or training. Below are specific distinctions:  \n\n- **COCONUT**: Our method overcomes several limitations of COCONUT, including reliance on weight tying, the need for training, and incompatibility with large models. These differences were discussed in the related work section (line 92-100).  \n\n- **DoLA**: The statement **Weighted different layer representations generate concept vectors** is not very accurate. DoLA does not use weighted representations. Instead, it contrasts the probability distributions of premature layers and mature layers. In addition, DoLA amplifies factual knowledge encoded in higher layers by contrasting probabilities but does not generate abstract \"concept vectors and still use discrete tokens. \n\nNevertheless, thank your for pointing out these methods. We will discuss them in camera ready version for clarification.\n\n---\n\n### **2.3 Cold Stop**\nCold Stop is a straightforward and effective approach to reducing token counts and addressing OOD issues. We primarily validate the effectiveness of Cold Stop through experiments detailed in the ablation section. Regarding the theoretical aspect, due to space constraints, we will provide a more comprehensive theoretical derivation in the camera-ready version.\n\n---\n\n### **2.4 Comparison to Speculative Decoding and ACT**  \nThank you for sharing these methods. Speculative Decoding focuses on accelerating decoding speed by using smaller models, not skipping several tokens. ACT determines the number of computational steps or layers needed based on input complexity. While ACT operates vertically (layer-wise), Soft Thinking operates horizontally (reasoning steps). These are orthogonal optimization directions, and Soft Thinking could be combined with these methods, which we consider a promising future direction. These possible research directions don't impact the innovation of Soft Thinking.\n\n---\n\n### **2.5 Limited Analysis of Failure Cases**  \nOur experiments reveal that tasks requiring lengthy code generation demand higher thresholds, as lower ones may cause premature termination. We will include examples illustrating this issue in the revised paper.\n\n---\n\n### **2.6 Benchmarks Focus on STEM**  \nThe seven benchmarks (four math, three coding) we selected are widely recognized and commonly used, providing a comprehensive comparison of different methods. However, we will expand our experiments to include additional benchmarks in the revised paper to further validate generalization.  \n\n---\n### **2.7 Related Works Comparison**\nMethods like Tree of Thoughts (ToT) and Graph of Thoughts (GoT) require extensive sampling (e.g., 16-64 samples for self-consistency), making their token usage significantly higher (up to 64x CoT). We will add experiments in the revised paper to highlight the trade-offs between effectiveness and efficiency, due to the constraints of the rebuttal period and computational resources.\n\nRegarding Soft Prompt, while both methods contain \"soft\" in their names, they are fundamentally different. Soft Prompt focuses on parameter-efficient fine-tuning, requiring additional training and fixed soft prompts during inference. In contrast, Soft Thinking expands reasoning tokens into continuous space, addressing a different problem altogether.  \n\n# **Acknowledgment**\nWe sincerely respect and appreciate the hard work and professionalism demonstrated in your review. We look forward to your feedback on our revisions and thank you again for your valuable time and effort."}]

Author rebuttal:

Meta-review: This paper introduces Soft Thinking, a training-free reasoning framework for large language models that shifts reasoning from the discrete token space (as in traditional Chain-of-Thought) to a continuous concept space. Instead of sampling a single token and embedding it, the method constructs concept tokens—probability-weighted mixtures of embeddings drawn from the output distribution (with top-k and top-p constraints). This enables the model to implicitly explore multiple reasoning paths in parallel, improving efficiency and robustness. After rebuttal and discussion stages, all reviewers gives positive comments. Thus I also recommend to accept this paper.
Decision: Accept (poster)