LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking
Authors: Junhong Wu
Jinliang Lu
Zixuan Ren
Gangqiang Hu
Zhi Wu
Dai Dai
Hua Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model
latent chain-of-thought
reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We show that Soft Thinking are reduced to single-path reasoning and propose randomness-based strategies, with Gumbel-Softmax proving most effective for enhancing reasoning performance.

Abstract: Human cognition naturally engages with abstract and fluid concepts, whereas existing reasoning models often rely on generating discrete tokens, potentially constraining their expressive capabilities. Recent advancements aim to address this limitation by enabling large language models (LLMs) to generate soft, abstract tokens, thus facilitating reasoning within a continuous concept space. In this paper, we investigate the $\textit{Soft Thinking}$ capabilities of various LLMs through a systematic analysis of their internal behavior using a suite of probing techniques. Contrary to the prevailing belief that Soft Thinking supports parallel exploration of diverse reasoning paths, our findings reveal that $\textbf{LLMs behave as single-threaded reasoners}$—they predominantly rely on the token with the highest probability in the soft input to predict the next step. This behavior induces a greedy feedback loop that suppresses alternative reasoning paths and undermines the benefits of transmitting richer information via Soft Tokens. To address this $\textit{Greedy Pitfall}$, we propose $\textbf{Stochastic Soft Thinking}$, which introduces stochasticity to break free from the greedy tendency. Our experiments demonstrate that incorporating $\textit{randomness}$—particularly with the $\textbf{Gumbel-Softmax trick}$—can alleviate the limitations of vanilla approaches and unleash the potential of Soft Thinking, resulting in superior performance across eight reasoning benchmarks.

Published as a conference paper at ICLR 2026
LLMs ARE SINGLE-THREADED REASONERS: DEMYS- TIFYING THE WORKING MECHANISM OF SOFT THINK- ING
Junhong Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai! Hua Wu
Baidu Inc., Beijing, China
ABSTRACT
Human cognition naturally engages with abstract and fluid concepts, whereas ex- isting reasoning models often rely generating discrete tokens, potentially con- straining their expressive capabilities. Recent advancements aim to address this limitation by enabling large language models (LLMs) to generate soft, abstract to- kens, thus facilitating reasoning within a continuous concept space. In this paper, we investigate the Soft Thinking capabilities of various LLMs through systematic analysis of their internal behavior using a suite of probing techniques. Contrary the prevailing belief that Soft Thinking supports parallel exploration of diverse reasoning paths, our findings reveal that LLMs behave as single-threaded rea- soners-they predominantly rely the token with the highest probability in the soft input to predict the next step. This behavior induces a greedy feedback loop that suppresses alternative reasoning paths and undermines the benefits of trans- mitting richer information via Soft Tokens To address this Greedy Pitfall, we propose Stochastic Soft Thinking, which introduces stochasticity to break free from this Greedy Pitfall. Our experiments demonstrate that incorporating random- pess-ppaticullrryy with the Gumbel-Softmn of vanilla approaches and unleash the potential of Soft Thinking, resulting in su- perior performance across eight reasoning benchmarks. We further demonstrate that Stochastic Soft Thinking exhibits stronger exploration potential compared conventional COT. Our findings deepen the understanding of continuous reason- ing and establish the foundation for future work on improving Soft Thinking with Reinforcement Learning.
trick can alleviate the limitations
INTRODUCTION
Large Language Models (LLMs) have achieved remarkable progress across a wide range of tasks (Jaech al., 2024: DeepSeek-Al al., 2025). A key driver of this success Mhain-of-Thhugih (CoT) prompting, which enables models to solve complex problems by generating intermediate rea- soning steps in natural language (Kojima reasoning process to sequences of discrete tokens, which may limit the model's ability to explore alternative solutions and reason beyond the bounds of natural language (Yao et al., 2023).
al., 2022). However, CoT inherently constrains the
In contrast, neuroscientific studies suggest that many aspects of human reasoning operate indepen- dently of language, engaging distinct brain regions (Fedorenko Varley, 2016; Benn et al., 2023; Fedorenko et al., 2024). Inspired by this insight, recent work has proposed replaceing discrete to- kens with continuous representations such as hidden states or output distributions for decoding (Hao et al., 2024; Shen et al., 2025; Zhang et al., 2025). These approaches aim enable LLMs to reason with abstract, high-bandwidth signals and explore multiple reasoning paths in parallel (Hao et al., 2024; Zhang et al., 2025; Gozeten al., 2025; Zhu al., 2025).
Despite promising results, the underlying mechanisms of Soft Thinking remain poorly understood. In this paper, we conduct systematic investigation into the behavior of Soft Thinking in modern
"Contact: 1600013522 B0t0005222@kk.eduu... pku.edu.cn, daidai @ baidu.com 'Corrsponding author

---

Published as a conference paper at ICLR 2026
LLMs with strong reasoning capabilitiess (DeepSeek-AI et al., 2025; Team, 2025; He et al., 2025). Surprisingly, our preliminary experiments reveal that the vanilla Soft Thinking approach performs significantly worse than conventional token sampling-basee based decoding methods. To understand this discrepancy, we analyze the model's s internal dynamics using a suite of probing techniques. Our findings challenge the prevailing assumption that Soft Thinking facilitates parailel exploration of multiple reasoning paths. As illustrated in Fig 1, LLMs tend to rely predominantly on the single token with the highest probability in the soft input predict the next step. This behavior induces a feedback loop that reinforces the most confident reasoning trajectory, thereby suppressing the ex- ploration of alternative paths. While Soft Tokens do transmit richer information and can potentially unlock novel reasoning trajectories, this Greedy Pitfall prevents the LLM from fully capitalizing on this advantage.
To address this issue, we introduce Stochastic Soft Thinking, framework that restores controlled randomness into the reasoning process using techniques such as Dirichlet sampling and the Gumbel- Softmax trick (Gumbel, 1954; Maddison et al., 2014; Kool et al., 2019). Our empirical evaluations on challenging reasoning benchmarks across three mainstream LLMs show that these methods mit- igate greedy behavior and unlock the potential of Soft Thinking. In particular, Gumbel-Softme offers a flexible trade-off between randomness and smoothness, leading to consistent performance improvements over both vanilla Soft Thinking and traditional COT. With controllable stochasticity, we enable Test-time Scaling for Soft Thinking and observed stronger exploration potential com- pared to standard Token COT. Our findings deepen the understanding of continuous reasoning and establish the foundation for future work on improving Soft Thinking with Reinforcement Learning.
Our contributions are threefold:
We present the first in-depth analysis of Soft Thinking in LLMs, revealing that it does not inherently support parallel reasoning and is dominated by top-1 I token signals.
We propose Stochastic Soft Thinking, demonstrating that controlled via setee-Sooomaa-iiinfiiiiaae
cedoomness-sspcccnne
improves reasoning performance.
We demonstrate that Stochastic Soft Thinking exhibits stronger exploration potential compared to conventional COT, establishing the foundation for future work improving Soft Thinking with Reinforcement Learning.
Soft Thinking
Output Takers
demce can of cuts 55525
isoder' dirm damileeee anvil for
Probability
lall
Large Language Model
Gorn Tokos
Embeidingg
CSM
Tokons
st,
SGg
Teminal Tokee
Soft Thinking replaces the discrete token ł with the Soft Token st (defined as the probability distribution over vocabulary). Right: Soft Thinking predominantly explores branches associated with the top-1 token. In contrast, paths stemming from non-top-1 tokens are typically terminated in the next step.
Figure 1: Left:
BACKGROUNDS: DISCRETE THINKING AND SOFT THINKING
2.1 DISCRETE THINKING
We begin by outlining the discrete token generation process in LLMs. Given a query x, the model generates a sequence y, which includes both the intermediate reasoning path t and the final answer a. Initially, the model generates the intermediate reasoning path t. Upon decoding special end- of-tthinking token, <EOT>, the model transitions to generating the final answer a. During each decoding step, the L.M uses the input and all previously generated tokens y<e context to
2

---

Published as a conference paper at ICLR 2026
predict the next token y::
LLM(x, E A
(I)
Here, V represents the vocabulary of the LLM, and AI is the probability simplex over the vocabulary, representing all possible probability distributions.
2.2 SOFT THINKING
In the discrete thinking process, the selection of only one token inherently restricts the expressive capacity of LLMs to natural language and may result in potential information loss. To address this limitation, more intuitive approach is to bypass the final token selection process entirely, instead feeding the entire probability distribution into the next step of the model (Zhang et al., 2025). To formally define this vanilla Soft Thinking process, we introduce the following terms.
Definition (Soft Token). The soft token is defined as the generated probability distribution over the LLM's vocabulary:
st = LLM(XX E A
(2)
Definition 2 (Soft Impur). To feed a Soft Token into the model, we employ the embedding matrix of the model E RlV|xd Here, represents the embedding size of the model. Let e; denote the embedding vector of the token t,, The embedding of a Soft Token, denoted as E(s(), is calculated as the weighted sum of the individual token embeddings:
M
E(st)
(3)
This construction ensures E(s() is a convex combination of all embedding vectors. As a result, it remains within the manifold of the original LLM input space, which helps alleviate the issue of out-of-distribution (OOD) inputs (Yuan et al., 2023). Preserving all tokens in the vocabulary can introduce noise and significantly increase computational overhead. To address this, we employ top- k or top-p truncation followed by renormalization, effectively truncating the unreliable tail of the probability distribution (Holtzman al., 2020).
Soft Thinking Process. In practice, Soft Thinking is applied only during the intermediate reasoning process, replacing the discrete token CoT. each step, the model outputs a Soft Token and constructs its corresponding Soft Input for the next generation step. When the top-1 token of the generated Soft Token is <EOT>, the reasoning process is terminated, and the model switches to discrete token decoding to generate readable final answer.
2.3 POSSIBLE WORKING MECHANISM OF SOFT THINKING
One of the most intriguing properties of Soft Thinking is its potential to inherently preserve mul- tiple reasoning paths, effectively forming latent search tree. COCONUT suggested that models utilizing Soft Thinking can maintain a diverse set of possibilities within the continuous reason- ing process (Hao et al., 2024). Many works (Zhang et al., 2025; Zhu et al., 2025; Gozeten et al., 2025) propose theoretical frameworks demonstrating how Soft Thinking can perform implicit par- allel search. However, the empirical evidence supporting these hypotheses is still absent.
PRELIMINARY EXPERIMENTS: Is SOFT THINKING EFFECTIVE?
In this section, we present our initial experiments with a vanilla implementation of the Soft Thinking approach. We begin by detailing the tasks and LLMs employed in our experiments. Subsequently, we demonstrate that vanilla Soft Thinking generally underperforms compared to token CoT.
3.1 EXPERIMENT SETTING
Models. We utilize range of mainstream LLMs known for their reasoning capabilities. These include Ri-bsscck-R--iiiil------ (DeepSeek-AL et al., 2025), QwQ-32B (Team, 2025), and Skywork-OR1--2B (He et al., 2025).
3

---

Published as a conference paper at ICLR 2026
Benchmarks. Our evaluation is conducted on eight diverse benchmarks. These include ALME'24/" (Mathematical Association of America, 2024), MATH-500 (Cobbe et al., 2021), and AMC'23 23 mathematics competitions. (2023) in the mathematical domain, GPQA-Diamond (Rein et al., 2023) for knowledge-based question-and-answer. and HumanEval (Chen et al., 2021), MBPP (Austin al., 2021), and LiveCodeBench (Jain et al., 2025) for code generation. This com- prehensive suite allows us to thoroughly assess the performance across varied domains and tasks. Please refer Appendix A for details.
Implementation Details. We utilize the SGLang (Zheng et al., 2024) Soft Thinking implementa- tion provided in Zhang et al. (2025). For both vanilla Soft Thinking and discrete Token Thinking, set the temperature to 0.6, top-p 0.95, and top-k to 30. Across all experiments, the maximum generation length is capped at 32,768 tokens. For the AIME' 24/ 25 and AMC'23 benchmarks, we report Average @ 32 mitigate the effects of randomness in smaller test sets. For the remaining benchmarks, we report @ass@1. I.
Table I: Detailed results on math, knowledge Q&A, and code benchmarks, Results highlighted in green indicate an improvement or performance comparable to Token COT. Results highlighted in red signal decline in performance relative to Token COT.
Thinking
AIME24
AIME25
MATHSOD
AMC23
GPQA-DDamend
HamanEval
МЯPР
LiveCrdeBench
Deepseck RI-Distll Qwen: 326
(Greedy)
92.30
85
Token (Sampling)
72.08
94.30
95.46
80.60
97.25
95.13
57.35
78,50
Sodl (Vmillaa
62.00
49.17
91.60
90.00
60.10
85.41
87.93
44.80
71.50
QwQQ:326
Tokes (Greedy)
$0.00
70.00
97.00
100.00
64.14
95.12
96.
58.78
62.64
Token (Sangling)
96.20
62.03
(anillla)
Skywork OR
Token (Greedy)
76.67
95.80
90.00
81.71
Token (Samgling)
78.75
71.25
96.40
98.
62.62
95.95
97.28
62.37
82.99
Solt (Namilla)
79.16
69.38
96.00
97.97
59.50
85.37
90.56
55.56
79.21
3.2 EXPERIMENTAL RESULTS
As shown in Table across all tested large language models (LLMs), vanilla Soft Thinking gener- ally underperforms compared to discrete Token Thinking. Overall, we find that vanilla Soft Thinking does not lead improvements compared discrete Token thinking in accuracy. Instead, it appears to achieve similar performance with greedy decoding.
303
Figure 2: An example illustrating the probability distribution of the vanilla Soft Thinking method.
4 ANALYSIS OF SOFT THINKING BEHAVIOR
Our preliminary experiments revealed that its performance did not surpass that of discrete token decoding methods. This unexpected result prompted to question the actual behavior of Soft Thinking in practice: Why does incorporating vanilla Soft Thinking not lead to the anticipated performance improvements?

---

Published as a conference paper at ICLR 2026
4.1
CASE STUDY
Figure 2 presents a case study of the vanilla Soft Thinking process applied to a simple question. Through this example, we observe that, in consecutive decoding steps, the components of the subse- quent token consistently and exclusively exhibit semantic coherence with the dominant token from the preceding step. For instance, in steps 2 and 3, the word 'I' not semantically appropriate successor to 'let'. Based on these observations, we propose the following hypothesis:
Hypothesis: L. Signle- Threeedd Reasoners
LLMs lack the ability process multiple different semantic trajectories in parallel. When Soft Token is fed into an LLM, the generation process is typically dominated by the majority component of the Soft Token.
To verify this hypothesis, we employ QwQ-32B to generate responses for AIME'24 and AIME' 25. We collect prediction distributions for nearly 10t steps and only preserve the intermediate reasoning steps. Subsequently, we analyze the decoding behavior of these generated responses.
4.2
OUTPUT PROBABILITY OF SOFT THINKING
1st token
token
Vonila Soft Thinknn
3.6
Similartty (Rouge-1 Score)
0.9
Discrete token lhinkig
3.8
0.8
3.
Propertion
0.7
0.6
2.4
0.6
2.4
0.4
9.5g
30
60
Tokeni Entropy
Layer
Prefix Length
Entropy JS Divergence
(b) Logit Lens
(c) Prefix Similarity
Output entropy/token probability JS- IS-divergence between next token prediction proba- bilities yield from different inputs. The prediction of soft input is nearly identical to the prediction of the 1st token input, but completely different from the prediction of the 2nd token input.
Figure 3:
We begin by examining the model's output probabilities during identified soft steps by conducting three separate forward passes and observing their outcomes: pass uses the entire Soft Token, denoted as Pst. while the other two use the highest and second-highest probability tokens within the Soft Token, denoted as P1 and Pz, respectively. We employ Censen-Shannon (JS) Divergence to measure the variation in prediction behavior resulting from these distinct inputs.
As shown in Figure 3(a), the JS divergence between the Soft Thinking result, Pst. and the highest probability token thinking result, Pı, shows a highly concentrated distribution around O. This sug- gests that the model's prediction during Soft Thinking step closely matches the outcome with just the specific token, implicating a significant influence of the top-1 token on the model's reasoning process. In contrast, the JS divergence between the Soft Thinking result, Pst, and the second-highest probability token thinking result, Pz, frequently approaches the maximum value of IS divergence, indicating that the second- -ighest probability token has limited influence on the thinking process.
4.3 DECODING HIDDEN STATES BY LOGIT LENS
To deepen our understanding of the decoding process, we apply the Logit Lens technique (nostal- gebraist, 2020) to track the reasoning paths of various components of the Soft Token. This method involves normalizing the output of intermediate hidden states and projecting them through the LM head. Previous studies have shown that these projections often yield interpretable, top-ranked tokens that align closely with the model's intermediate representations (Dar et al., 2023; Geva al., 2023).
5

---

Published as a conference paper at ICLR 2026
We start by pinpointing branching points where the model generates Soft Token consisting of least two semantically diverse tokens'. To construct a balanced Soft Token, we manually assign probabilities of 0.6 to the first token and 0.4 to the second. Following the previous section, perform three separate forward passes and apply the Logit Lens. For the Soft Token forward pass, extract the top 10 tokens, while for the two single-token forward passes, we extract the top tokens each. We then plot the size of the intersection between the Logit Lens results from different forward passes illustrate the extent of each token's 's reasoning path within the Soft Token process.
As depicted in Figure 3(b,, the representation of both single tokens' reasoning paths gradually rises within the first 2-3 layers. This suggests that the model initially considers both reasoning paths in parallel. However, as processing continues through additional layers, the prominence of the first token's 's path steadily increases 1.0, while the second token's 's path decreases". This pattern indicates that the forward process inherently acts as pruner, progressively favoring the reasoning path of the first token while diminishing that of the second.
4.4 GREEDY PITFALL
As demonstrated in previous sections, L.MSs predominantly rely on the top-1 token for forward computation. This tendency creates a positive feedback loop where the model becomes entrenched in its most self-assured reasoning path-a phenomenon we term the "Greedy Pitfall." To further substantiate this, we evaluate the sequence similarity between vanilla Soft Thinking, discrete Token Thinking. and Greedy Token Thinking. Specifically, we concatenate the top-1 I token from each step in vanilla Soft Thinking to form coherent reasoning trace and compute the ROUGE-L metric using the Greedy Token Thinking trace as the reference. As shown in Figure 3(c), the ROUGE-L score of vanilla Soft Thinking is significantly higher than that of discrete Token Thinking, indicating that vanilla Soft Thinking inherently exhibits greedy behavior. This explains why vanilla Soft Thinking underperforms, the reasoning path that maximizes likelihood is proven to be generic, repetitive, and awkward (Holtzman et al., 2020).
5 UNLEASHING THE POTENTIAL OF SOFT THINKING WITH RANDOMNESS
Our experiments and analysis reveal that vanilla Soft Thinking tends to be inherently greedy and does not nec- essarily enhance performance. However, this does not en- tirely negate the potential of Soft Thinking. Specifically, as illustrated in Figure 3(a), there are instances where the richer information in Soft Token leads to reasoning branches that diverge from the 1st token's is decoding path. As the generation process continues, these branches cause the model to gradually deviate from the greedy trace, as evidenced by the decreasing ROUGE-L scores shown in Figure 3(c). Despite this potential for stronger expressive capabilities, the overall tendency towards greediness ob- scures the benefits of the soft input, leading to suboptimal performance. To uncover the true potential of Soft Think- ing, exploring approaches to break free from the
Sotchestic Thinking
Output Tokans
Sompling
Probabiitty
Large Language Model
Embedding
Tokens
Xi. Xmn
St
Figure 4: An illustration for Stochastic Soft Thinking, which incorporates ran- dom sampling techniques to construct a Stochastic Soft Token.
"Greedy Pitfall."
5.1 STOCHASTIC SOFT THINKING
Previous works have established that the top-quality generations obtained from the model rely on randomness in the decoding method (Holtzman et al., 2020). We believe that such randomness is similarly crucial for effective Soft Thinking and further RL training (Appendix ??). Therefore, we explore strategies to introduce randomness into the Soft Thinking process. Intuitively, our focus
We select semantically diverse Soft Thinking steps according to the IS divergence hetween the first and the second token's 's forward results. "The prominence of the second token does not converge zero, because it shares some tokens with the reasoning paths of the first token.

---

Published as a conference paper at ICLR 2026
is generating Stochastic Soft Tokens with controllable randomness, st', shown in Fig 4. We
begin by outlining several key properties that a Stochastic Soft Token st' should possess:
Validness: should still be a valid probability distribution over
Randomness: should be unbiased and reflect the original predictive information in
Softness: It should be soft and do not collapse into one-hot vector.
Guided by these principles, explore two different approaches, namely, Dirichlet Sampling and the Gumbel- Gumbel-Softmax trick.
5.1.1 DIRICHLET SAMPLING
Perhaps the most common distribution over the probability simplex A= is the Dirichlet distri- bution. Intuitively, could set the model's output distribution p the concentration parameters and construct a corresponding Dirichlet distribution Dir(p). However, directly using p as concentra- tion parameters will cause the probability nearly one-hot vectors. To this end, we introduce a scaling parameter "y and sample from Dir(yp).
to concentrate near the simplex corners, resulting in
5.1.2 GUMBEL-SOFTMAN TRICK
The Gumbel-Max trick is an algorithm for sampling from categorical distribution over classes i I,..., n with probability The algorithm first samples independent noises gi from the Gumbel distribution G(O, 1). Subsequently, the sampled noise is applied to the logits log(T;) of probability æ, followed by an argmax operation. The Gumbel Softmax distribution utilizes the softmax function continuous, differentiable approximation to argmax, deriving n-dimensional
Gumbel-Softmax
with temperature sample vectors y E n-11 where:
exp ((g. log((:))/T) y/ E exp ((9x + log (Tx))/T)
Empirically, the Gumbel-Softman distribution interpolates between discrete me-hott-nncoded cate- gorical distributions and continuous categorical densities (Maddison et al., 2014; Kool et al., 2019). Therefore, applying this trick to the original Soft Token yields a valid Stochastic Soft Token.
5.2 EXPERIMENTAL RESULTS
Table Detailed results on math, knowledge Q&A, and code benchmarks. Results highlighted in green indicate an improvement or performance comparable to Token COT. Results highlighted in red signal a decline performance relative to Token CoT
ATME34
AIME35
MATH500
AMC23
GPQA-Daamodd
HumunFxol
MRPP
LiveCrdeBennc
Avg
Deqpseak RI-DRRDiD---gee
Token (Sansplnng)
Soft (Vanilla)
62.00
49.17
91:40
00.00
60.10
86.41
87.93
44.80
71:50
Soft (Dirichlet)
69.79
$4.58
94.60
94.53
62.12
98.17
95.72
$7.35
78.36
Soct (Gumbel))
96.00
QwiQ-320
Token (Sampling)
77.92
67.5
96.20
97.5
62.63
98.17
96 89
62.00
$2.35
Soft (Namille)
76.67
62.29
20
98
59.60
93.90
95.33
57.71
80.06
Soll (Gumbel)
78.96
97.20
96.
67.67
97.56
62.72
83.63
Skywork-OR (OII--RR--300
(Sungling)
62.62
Seft (Dirichlee)
78.06
71.25
96
97.
66. 16
96. 34
97.25
61.20
83.1 12
Soft (Gummhel) 79.79
73.75
97.40
98.59
67.67
97.56
98.05
64.16
$4.62
We implemented two randomization approaches within the Soft Thinking framework introduced in Section and conducted experiments adhering to the protocol outlined Section 3.1. For Dirichlet Sampling, the scaling parameter was tested in the range of [1.0, 10.0] with increments of 1.0. For the Gumbel-Softmax trick, the temperature hyperparameter 7 was tested in the range of [0.3. 0.9) with increments of 0.1. We found that setting 4.0 and 0.5 yielded good performance, and these values were used as the default hyperparameters. The results across eight benchmarks are presented in Table 2.

---

Published as a conference paper at ICLR 2026
2.
(a) Dinichlet y = 0.3
(b) Dirichlet 7 = 1.0
(c) Dirichlet 7 5.0
(d) Dirichlet y = 10.0
(c) Gumbel t = 0.3
(f) Gumbel 7 0.6
g Gumbel 7 0.9
(h) Gumbel T 1.2
Figure 5: Softness vs. randomness for Stochastic Soft Tokens.
Both Stochastic Soft Thinking approaches demonstrated improved performance compared to vanilla Soft Thinking. This suggests that incorporating randomness effectively mitigates the "Greedy Pit- fall." Notably, only the approach utilizing the Gumbel-Softmax trick achieved performance gains beyond those of discrete Token Thinking. We conducted further experiments to understand this difference.
5.3 BALANCING RANDOMNESS AND SOFTNESS
To understand the performance differences between various Stochastic Soft Thinking approaches, quantified the randomness and softness of each method. Specifically, we used the normalized entropy of generated Soft Tokens as a proxy for softness. For randomness, we calculated the JS divergence between the original LLM output probability and the Stochastic Soft Token st'.
As illustrated in Figure 5, the Gumbel-Softmax trick allows for easy adjustment of the softness by varying the temperature hyperparameter т, while consistently maintaining adequate randomness, indicated by high JS divergence. In contrast, Dirichlet sampling struggles to balance randomness and softness. When the scaling parameter 7 -> the Stochastic Soft Token exhibits sufficient randomness but collapses into near one-hot vector, as indicated by the low entropy. Conversely, when the scaling parameter yy increases, the Stochastic Soft Token gradually converges to the original model's 's output probability distribution, displaying higher softness but limited randomness. This difference highlights that the Gumbel-Softmax trick can effectively leverage both the advantages of randomization and Soft Tokens, whereas the Dirichlet approach cannot.
5.4 THFORETICAL JUSTIFICATION GUMBEL-SOFTMAX TRICK AND LUCE'S CHOICE AXTOM
The Gumbel--Softaa trick is not merely an experimentally convenient method for sampling from categorical distributions; is theoretically well-motivated for constructing Stochastic Soft Tokens because uniquely satisfies Luce's choice axiom (Luce et al., 1959). Luce's choice axiom asserts that the probability of choosing an item from a set depends solely on its utility relative to the sum of all utilities in the set, independent of other alternatives. This property is crucial for ensuring that selection probabilities reflect the true relative preferences among items. The Gumbel-Softmax distribution naturally satisfies Luce's axiom (Maddison et al., 2014; Kool et al., 2019). By adding Gumbel noise the logarithm of the utilities (or probabilities). the Gumbel-Max trick ensures that
8

---

Published as a conference paper at ICLR 2026
5
(a) Qwen2.5-0.5B
(b) Qwen2.5-1.5B
(c) Qwen2.5-3B
(d) Qwen2.5-7B
Figure 6: @ass@k Comparison.
the selection probabilities are proportional to the original utilities:
exp(Iog(TT))
argmax g + log((T))
(5)
Σ; exp(Iog(T:))
We generalize the argmax operation argtopk operation, which returns the indices of the largest values, in order of decreasing value. Kool et al. (2019) proved the following theorem:
Theorem 1. For k < n, let I' ...., I be argtopk, [g. + log(=,)). Then I' I'.... is an (ordered) sample without replacement from the Categorical Distribution x on a pool N. This means, for realization i!, ....,"" 1", the following holds:
P(I = ) ΣN, ne
T
(6)
where N₂ N (i!....... is the domain (without replacement) for the j-th sampled element.
This theorem shows that the Gumbel-Man trick naturally extends to sampling multiple items without replacement while preserving proportional probabilities. Such a mechanism closely mirrors the ideal process of constructing Stochastic Soft Tokens from LLM output distributions, where tokens are sequentially selected according to their relative utilities among the remaining options. Replacing the argtopk with Softmax and top-k renormalization provides a relaxation: the discrete ranking is converted into continuous distribution, whose probabilities preserve the ranking information and can be directly used to weight token embeddings when constructing the input for the next generation step.
5.5 STRONGER EXPLORATION POTENTIAL
To further demonstrate the potential of Stochastic Soft Thinking in reinforcement learning (RL), evaluate the @ass@k metric introduced by Brown et al. (2024) across various base models. @ass@k k measures the proportion of problems a model can potentially solve within k attempts, and commonly used to assess the capability boundaries large language models (LLMs) (Yue al., 2025; Chen et al., 2025). A higher Pass@k score indicates stronger exploration ability, which is critical for effective RL training.
Figure 6 presents the @ass@k performance (A: = 1, 2,2,4,8, 16,32) the MATH500 benchmark for the Qwen2.5 base model, ranging from 0.5B to 7B parameters. As shown, Stochastic Soft Thinking consistently outperforms Discrete Token Thinking across all settings, suggesting that incorporating Stochastic Soft Thinking into rollouts and RL training may be a highly promising direction. Due to time and resource constraints, leave full RL integration for future work.
LIMITATION AND FUTURE WORKS
While research primarily focuses training-free Soft Thinking approach, our findings also shed light on fine-tuning LLMs to support this method of reasoning. Notably, we discovered that LLMs typically function single-threaded reasoners, lacking the ability for parallel reasoning. This limitation appears to stem from an inductive bias inherent in both the Transformer architecture and the next-token prediction objective used in pretraining and fine-tuning. Due to this strong inductive

---

Published as a conference paper at ICLR 2026
bias, enabling LLMs to engage in parallel reasoning through Soft Thinking presents significant challenges.
Supervised fine-tuning can lead to substantial distributional shift, resulting in catastrophic forget- ting of the knowledge acquired during pre-training. This issue may explain why approaches like COCONUT (Hao et al., 2024) and its variants struggle match the performance of token-based COT, as they require fine-tuning the model to compress multiple tokens into one embedding. Al- though recent studies show that reinforcement learning (RL) can enhance reasoning abilities without forgetting (Lai et al., 2025), it is also constrained by the capabilities of the base model (Yue et al., 2025).
As we have shown in the paper, existing LLMs are unable to generate Soft Thinking traces with par- allel reasoning characteristics. Therefore, relying solely on RL facilitate parallel reasoning may not be a feasible approach. Valuable directions include exploring multi-toker prediction pre-training and novel architectures build the ability of parallel thinking. Nonetheless, our experiments indi- cate that Soft Thinking demonstrates certain advantages beyond parallel thinking, attributed to the enriched information conveyed through Soft Tokens. Future research could concentrate on harness- ing this potential to improve the reasoning capabilities of LLMs.
7 RELATED WORK
Chain-of-thoughh (CoT) prompting enhances language model performance by facilitating step-by- step reasoning through natural language (Kojima et al., 2022). However, this approach can be ineffi- cient due to its reliance discrete text tokens. To address this, various previous studies have inves- tigated the potential of reasoning in a continuous space. Yang et al. (2024) and Shalev et al. (2024) investigate the implicit reasoning capabilities of transformers with multi-hop reasoning tasks. Other works have sought to use the language model's internal hidden states to perform implicit reason- ing, instead of explicitly producing the chain of thought reasoning steps (Deng et al., 2023; 2024). Geiping et al. (2025). Another line of work sought to fine-tune LLMs, enabling them to reason with explicit continuous tokens. COCONUT (Chain of Continuous Thought) (Hao al., 2024) operates within the model's hidden state space, eliminating the need for explicit text generation. CODI (Shen et al., 2025) frames the problem as learning to align recurrent hidden states through self-distillation. While these works are promising theoretically, they struggle to generalize to larger models and more challenging benchmarks.
Previous research has established that Large Language Models (LLMs) can generate coherent re- sponses when presented with mixed embeddings as input(Shen et al., 2024; Marro et al., 2025). Building this, recent frameworks, such as Soft Thinking (Zhang al.. 2025) and Mixture-of- Inputs (Mol) (Zhuang et al., 2025), propose training- free Continuous COT methods for reasoning LLMs. These methods leverage the distribution over the vocabulary at each step to bridge the hidden state output space with the input embedding space. Despite their innovation, these approaches lack behavioral analysis of the soft decoding procedure. In contrast, our work identifies the Greedy Pit- fall in Soft Thinking through comprehensive analysis and introduces random sampling techniques effectively leverage soft inputs, presenting the first effective Soft Thinking decoding approach.
CONCLUSION
In this paper, we investigate the Soft Thinking ability of modern LLMs. Contrary the prevailing assumption that Soft Thinking facilitates the exploration of diverse reasoning paths, we demonstrate that LLMs cannot track multiple reasoning paths simultaneously. Instead, they predominantly rely on the most influential component of the soft inputs during subsequent decoding steps. This re- liance hinders the exploration of different reasoning paths and leads to a feedback loop that turns vanilla Soft Thinking into process resembling greedy decoding, obscuring the advantage of trans- mitting more information through Soft Tokens. To disrupt this cycle and unleash the potential of Soft Thinking, we propose Stochastic Soft Thinking by introducing controlled randomness into the Soft Thinking Process. Our findings indicate that the Gumbel-Softme approach, evidenced by both theoretical proof and experimental results. Our paper deepens the understanding of LLMs' latent reasoning abilities through detailed behavior analysis of the genera- tion process and establishes the foundation for further reinforcement learning (RL.) training.
is an ideal randomization
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Jacob Austin, Augustus Odena, Maxwell L. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le. and Charles Sutton. Pro- gram synthesis with large language models. CoRR, abs/2108.07732, 2021. URL https:
//arxiv. wtg.
077322.
Yael Benn, Anna A Ivanova, Oliver Clark, Zachary Mineroff, Chloe Seikus, Jack Santos Silva, Rosemary Varley, and Evelina Fedorenko. The language network is not engaged in object cate- gorization. Cerebral Cortex, 33(19)110300110400 2023.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.2178) 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavar- ian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plap- pert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Pe- ter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. URL https:tt///om /awxiv..vvvvv////// org/ abs////7 03374.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balaneing exploration and exploitation of large reasoning models. arXiv preprint 2508 00.225508.0777... 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CORR, abs/2110.14166 2021. URL
https://arxiv/ wwtrxiivvv.rrrooooooooooo alss/ 2210.111...00...11011111211112 14168.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers embedding space. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9- 9-14, 2023, pp. 16122-16170. Association for Computational Linguistics, 2023. doi: 10. 11533//1122233.A1111 18653/vl /2023 acl 11l1--onggg ong.. 893.
URL https:: //ds:///oiii org/10. 00.
DeepSeck-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu. Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501. 12948, 2025, doi: 10. 48550/ARX ARXIV.2000 12948. URL https: //o/f..././///// .rrg/10 48550 / a rx 2501 12948.
11

---

Published as a conference paper at ICLR 2026
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stuart M. Shieber. Implicit chain of thought reasoning via knowledge distillation. CoRR, abs/2311.01400, 2023. doi: 1044850/ARXXV.311.00444 URL htttps://ao..org/11000 2.11.00400
Yuntian Deng, Yejin Choi, and Stuart M. Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. CoRR, abs/2405. 14838, 2024, doi: 10.48550/RXXV.2405..
URL httphtts/////lt httg///ooooooooooooooooo.ooooe.o ott./.....500000........... 48550/ w.t.00/arxv.... 14448....8..
Evelina Fedorenko and Rosemary Varley. Language and thought are not the same thing: evidence from neuroimaging and neurological patients. Annals of the New York Academy of Sciences, 1369 (1):132-153, 2016.
Evelina Fedorenko, Steven T Piantadosi, and Edward AF Gibson. Language primarily a tool for communication rather than thought. Nature, 630(8017):555-58,, 2024.
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. CoRR, abs/2502.05171, 2025, doi: 10.48550/ARXIV. 2502.05171. URL https: //t::///oooo wttpooooo.........///////
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations auto-regressive language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 12216- Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023 Bttp:///2222.EELL-MPM.MMM EMNLP-MAIN.755 URL https:
//doi.org/10 8653 v12////v1222333/ /2023. emn mammmmlmmmmn.mmm
Halil Alperen Gozeten, Muhammed Emrullah Ildiz, Xuechen Zhang, Hrayr Harutyunyan, Ankit Singh Rawat, and Samet Oymak. Continuous chain of thought enables parallel explo- ration and reasoning. CoRR, abs/2505.23344 2025. doi: 10:48550/RXIV.2555.2222
URL
https:/ / /ooii 01tp///..8550//// a arxiv. 2505 23648.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. CoRR, abs/2412.06669, 2024. doi: 10.48550/ARXII.2422.0777 2412. 06769.
URL httpt:////// /ttps://///i/.///// wrw.orgg///./ 411.0..8555/aaxii..
Jujie He, Jiacai Liu, Chris Yuhao Liu, Rui Yan, Chaojie Wang, Peng Cheng, Xiaoyu Zhang, Fuxiang Zhang, Jiacheng Xu, Wei Shen, Siyuan Li, Liang Zeng, Tianwen Wei, Cheng Cheng, Bo An, Yang Liu, and Yahui Zhou. Skywork open reasoner technical report. CoRR, abs/2505.22312, 2025. doi: (0.4850/ARXI..555222311 2505. 22312.
URL https: : /wt.idooooooooooooo 11t/1o..0.000////.......
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL pettps://openzeee http://oormmmdryyy0000
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie. Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, An- drey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghor- bani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton. Brandon McKinzie, Brydon Eastman, Camillo Lu- garesi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely,
12

---

Published as a conference paper at ICLR 2026
David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Ed- mund Wong, Elizabeth Prochl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O' O'Connell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya. URL
Openai ol system card. CoRR, abs/2412.16720, 2024. doi: 10448500/RXIV.221222 https: doi. org/ 10. 48550/ arxiv 2412. 16720.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirleenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
URL bettps://opeereeeee
net/ Www.moiddcccheee.... ww-.nddhhchhhh......
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large language models zero-shot reasoners. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference
Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, htlp/pppr_ffeese/ wtt.lss/pppee/222//11...
2022, 2022. URL http: / pwwppsppsss.ppp//pppp pappers.nips... 8bb0d29 www.aaccccoffff1120000 09f29999211666666 Abst--cttaatttt... Coefe-Coooeeeeteteeeeee html.
Wouter Kool, Herke van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In Kamalika Chaud- huri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML, 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3499-3508. PMLR, 2019. URL http: / /pproceedings mlr pww.spresss....e
1ww.l99......................... html.
Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Min Xie, Qingfu Zhang, al. Reinforcement fine-tuning naturally mitigates forgetting in continual post-training. arXiv preprint arXiv:2507.05386, 2025.
R Duncan Luce al. Individual choice behavior, volume 4. Wiley New York, 1959.
Chris J. Maddison, Daniel Tarlow, and Tom Minka. A sampling. mani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27: Annual Conference on Neural In- formation Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pp. 3086-3094, 2014. URL htttpp::/pprcceedings http://rcceednnn...eeieeissesppp newrips........ wtt.:nnee.......peer/eeeee htte//pppr/211///asstee 300fe44451ee51222221222bbbbbbbb.
Zoubin Ghahra-
html.
Samuele Marro, Davide Evangelista, X. Angelo Huang, Emanuele La Malfa, Michele Lombardi, and Michael J. Wooldridge. Language models are implicitly continuous. In The Thirteenth In- ternational Conference on Learning Representations, ICLR 2025, Singapore. April 24-28, 2025. OpenReview.net, 2025. URL https: /pw.//eeeeeeeeeeee/eee.e.e...eee net www.:nrtttoooeoooo.omm fwrmmm d=w.dd
Mathematical Association of America. American invitational mathematics examination (aime), 2024. URL wwtpt:tt///// //aaa///ma.../ .oreaaorrggeee wwwoor//maa... http://aaa.ormmaaaaaaaaaa wtt.iiiittatt.oooooooooetttettte itioo//
American mathematics competitions. American mathematics competitions, 2023.
nostalgebraist. interpreting gpt: the logit lens, 2020. URL hwwph:ttp://// pptpt///Acctkaspppddddd///aeeeep
wewswwong....................... com/
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. CoRR, abs/2311.12022. 2023. doi: 10.48550/ARXXI2311..222 URL https: //doi //dbi.
12022.
13

---

Published as a conference paper at ICLR 2026
Yuval Shalev, Amir Feder, and Ariel Goldstein. Distributional reasoning in Ilms: Parallel reasoning processes in multi-hop reasoning. CoRR, abs/2406. 13858, 2024. doi: 10.48550/ARXIV24466
13858. URL https:/ /oi. wrw.loorrooo..o 48550/ (01/55//rXX0 2406.13858
Ethan Shen, Alan Fan, Sarah M. Pratt, Jae Sung Park, Matthew Wallingford. Sham M. Kakade, Ari Holtzman, Ranjay Krishna, Ali Farhadi, and Aditya Kusupati. Superposed decoding: Multiple generations from a single autoregressive inference pass. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http: /papers//pppppss.ppeepp https.cc/ppppee. d S7
wwtppppee.illlee//peeeeeeeeeeeee wwtpse/iieesssppeeeeeeeeeeeeeeee 2ttp///sh//////////////// /hash/ Abstract- CobstrrattCooffeennee.htttt
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. CODI: compress- ing chain-of-thought into continuous space via self-distillation. CoRR, abs/2502.21074, 2025. doi: 10448550/RRX22250220077 URL https: //t:::///d....// 21074.
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https:/ http///wwwmmm.mmm//////....mm..a
htoblll//////oooo...ooo....ooo/t h////wg332000 32b/.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? In Lun-We Ku, Andre Martins, and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Pupers), ACL 2024, Bangkok, Thailand, August 1 11-16, 2024, pp. 10210- 10229, Association for Computational Linguistics, 2024. doi: 10.18655/11/2024AACLLL 550. URL https: /ww.i org/ 10. 18653 /vll 2024 acl long 550.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023. URL http: pwwp:://ppppppppppppppeeeeeesse nips wttpcccpppppppeeeeeeeeeeeee wttpetieeeeeppeee///ee/eee 271 (el.cb9922bddcc --t.aaaeee-daaaeeeeeeeeee6------
Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng. Heng Ji, Zhiyuan Liu, and Maosong Sun. Revisiting cittrf--distiibution robustness in NLP: benchmarks, analysis, and Iims evaluations. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neurul Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA. December 10 16, 2023, 2023. URL http:/ http://papers. wwwpaprrrs.iippp.pmm oc haee/appr_fftte fwwes/lleppppp patkee//pppr//22333 hash/b6b5f50a200 wwte2220cccccccc6eeee6e4eeee -atp:eeccaao---bbbbaree Abstract Benchmarks. html.
Datasets -asttract-attsstss.....
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does inforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint ar arXiv:2504.138.
2025.
Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of Ilms in continuous concept space. CoRR, abs/2505.15778. 2025. doi: 10.48550AARXI1255555577 URL https:
//ldoi.org/// 485.444555//./ 15778
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrIPS 2024, Vancouver, BC, Canada, December 10-
14

---

Published as a conference paper at ICLR 2026
15, 2024, 2024. URL
/hash/
html.
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reason- ing by superposition: A theoretical perspective chain of continuous thought. arXiv preprint arXiv:2505.12514 2025.
Yufan Zhuang, Liyuan Liu, Chandan Singh, Jingbo Shang, and Jianfeng Gao. Text generation beyond discrete token sampling. CoRR, hbbs22005.148277 2025. doi: 10.48555/ARXIV2550. 14827. URL /ttp:///00o.o111008500000 ott.::/:......../...............
15

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
A4ME'24&'25 The AIME'24 and AIME'25 (Mathematical Association of America, 2024) datasets consist of problems from the 2024 and 2025 American Invitational Mathematics Exami- nation (AIME), a highly regarded high school mathematics competition known for its challenging questions. Each dataset includes 30 problems designed to test deep mathematical insight and cre- ative problem-solving
The MATH500 benchmark (Cobbe et al., 2021) evaluates the mathematical reasoning and problem-solving capabilities of language models, addressing the growing need for more rigorous assessments as model performance improves. It comprises 500 problems spanning five fundamental mathematical domains: algebra, combinatorics, geometry. number theory, and precalculus. Each problem is crafted to require multi-step reasoning and complex problem-solving, moving beyond basic computation factual recall.
MATH500
AMC'23 The AMC'23 (mathematics competitions., 2023) dataset contains 40 problems from the 2023 American Mathematics Competitions (AMC), a widely recognized mathematics contest aimed at middle and high school students. The problems emphasize logical reasoning, mathematical cre- ativity, and conceptual understanding, making the dataset a valuable resource evaluating models on moderately challenging mathematical tasks.
GPQA-Diamond GPQA-Diamond (Rein et al., 2023) a curated subset the GPQA benchmark, containing 198 multiple-choice questions across biology, chemistry, and physics. The questions range in difficulty from advanced undergraduate to postgraduate level. This subset includes only those items where both domain experts answered correctly and the majority non-experts answered incorrectly, ensuring high standard of quality and discriminative power.
HumanEval HvmanEval (Chen et al., 2021) is a benchmark designed to assess the functional correctness of code generated by language models. It consists of hand-written Python programming problems, each paired with a unit test. The benchmark evaluates a model's ability to synthesize correct and executable code from natural language prompts, making it a standard for measuring code generation performance.
MBPP The MBPP (Mostly Basic Python Problems) dataset (Austin al., 2021) includes 974 crowd-sourced Python programming tasks that cover basic algorithmic and data manipulation skills. Each problem is accompanied by a natural language description and test cases, MBPP is particularly useful for evaluating models on beginner to intermediate-level programming tasks.
LiveCodeBench LiveCodeBench (Jain et al., 2025) is a recent benchmark that evaluates real-time code generation and editing capabilities of language models. It includes diverse set of program- ming tasks across multiple languages and domains, emphasizing interactive coding scenarios such incremental edits, debugging, and code completion. This benchmark reflects practical coding workflows and is designed to test models in dynamic development environments.
16


Reviews:
[{"review_id": "N734xRCmS3", "reviewer": "Reviewer_PScP", "summary": "This paper tackles the problem of learning from human feedback when that feedback contains systematic noise and biases. The authors propose OPEN (Objective Preference Elicitation from Noisy feedback), which uses an auxiliary objective inference model to recover true preferences from noisy observations.", "questions": "How sensitive is the method to the choice of prior over objective functions?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The paper addresses a genuinely important problem since human feedback in practice is often inconsistent and biased, especially when dealing with complex tasks or subjective preferences. The theoretical framework connecting noisy observations to latent objectives through a probabilistic model is well motivated, and I appreciate that the authors provide both convergence guarantees and empirical validation. The experiments on LLM summarization and dialogue tasks show meaningful improvements over baseline RLHF methods, with particularly strong results when feedback quality degrades.", "weaknesses": "you can improve your writing, authors.\nThe auxiliary model for objective inference adds considerable complexity to the training pipeline, and I'm concerned about the computational overhead this introduces compared to standard RLHF. While the synthetic experiments are convincing, the real world experiments could benefit from more diverse evaluation settings beyond just text generation tasks. The paper also doesn't fully address how to set the hyperparameters for balancing between the inferred objectives and raw feedback, which seems crucial for practical deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jvpdEkKN6I", "reviewer": "Reviewer_YsqS", "summary": "This paper investigates the underlying mechanisms of Soft Thinking and challenge the prevailing belief that it would allow LLM to explore multiple reasoning paths in parallel. The primary experiments show that the standard Soft Thinking approach achieves similar results to greedy decoding and their analysis reveals that the subsequent generation is dominated by single token with highest probability. Numerical metrics such as JS Divergence and ROUGE-L similarity are applied to verify the findings. The paper then proposes Stochastic Soft Thinking to address above issues. They evaluate Dirichlet Sampling and Gumbel-Softmax Trick on Deepseek, QwQ, Skywork models in mathematical reasoning benchmark and Gumbel-Softmax method shows consistent improvement.\n\nIn short, the paper discover the Greedy Pitfall of Soft Thinking and address the problem by injecting stochastacy.", "questions": "1. Could the authors please provide the mean and variance (or confidence intervals) over multiple runs with different seeds to validate that these improvements are statistically significant and not an artifact of randomness? Could the authors provide analysis on the stability of the method at different randomness level?\n2. Could the authors repeat the behavior probes under Dirichlet/Gumbel soft tokens and report whether top-1 dominance is addressed?\n3. Could the authors provide an ablation study showing the end-to-end performance of both vanilla Soft Thinking baseline at higher temperatures to confirm this is an inherent flaw and not just a result of the low-temperature setup?\n4. Could the authors please revise the figures mentioned in weakness 4?\n\nI will raise my score once above issues are properly addressed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Clear Motivation and Problem Framing: The central claim that LLMs are \"single-threaded reasoners\" and that Soft Thinking defaults to a greedy process is an important observation on the drawback of previous method.\n2. Sufficient Analysis to support the Greedy Pitfalls: The evidence from output probability (JS Divergence) , hidden state representations (Logit Lens) , and sequence-level output (ROUGE-L) effectively support the hypothesis.\n3. Practical and Effective Solution: The paper address the issue by Stochastic Soft Thinking, particularly using Gumbel-Softmax which is practical and shows consistent improvement against vanilla Soft Thinking. The method is evaluated on various math, coding and QA benchmarks. This paper also conduct ablation study to demonstrate the necessity of randomness and softness.", "weaknesses": "1. The paper's modest performance gains are not shown to be statistically significant. The proposed \"Stochastic Soft Thinking\" method involves randomness. However, the reported average improvements are small (ranging from +0.42 to +1.05 points). For five of the eight benchmarks, the authors report Pass@1 scores, which may be sensitive to run-to-run variance. The reproducibility statement confirms these experiments were run with a single \"fixed random seed\". This is insufficient to demonstrate the stability and hyperparameter sensitivity of proposed method.\n\n2. Causal evidence for “mitigating the Greedy Pitfall” is incomplete. The paper shows accuracy gains and an input-level softness–randomness analysis (Fig. 5), but it does not re-run the core behavioral probes used to diagnose the pitfall—JS Divergence, logit-lens intersection trajectories, or ROUGE-L similarity to the greedy trace—after introducing randomness. Without those, it’s hard to conclude that improvements come from reducing top-1 dominance rather than other effects of stochasticization. Please repeat the diagnostic analyses under the stochastic methods to substantiate the claim.  \n\n3. The \"Greedy Pitfall\" is only demonstrated at a fixed temperature, lacking a crucial ablation study. The paper's entire premise that vanilla Soft Thinking fails is based on its performance at a fixed temperature of 0.6. The \"Greedy Pitfall\" is defined as the model's reliance on the top-1 token. The numerical dominance of the top-1 token is directly controlled by the temperature setting. It is plausible that simply increasing the temperature (e.g., to 1.0 or even higher) would flatten the probability distribution, make the soft tokens inherently less greedy, and potentially solve the pitfall without requiring the proposed stochastic methods. (Logit Lens analysis probes a single manually-balanced token with probability 0.6/0.4, this is not a substitute for a full sequential generation where a high temperature would be applied at every step of the generation.)\n\n4. Formatting Issue: The authors should consider adjusting the font size in Figure 5 and legend in Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8OdstNwCFL", "reviewer": "Reviewer_6x6V", "summary": "Prior work (soft thinking (Zhang et al., 2025)) processes the chain-of-thought context as a convex combination of token embeddings according to each position's model output distribution. This paper improves soft thinking by incorporating random sampling in the CoT process, where sampling is modelled by Dirichlet distribution and Gumbel-softmax distribution.", "questions": "- It has been mentioned that vanilla Soft Thinking fails to explore different reasoning paths. How does Stochastic Soft Thinking, e.g., Dirichlet sampling and Gumbel-softmax sampling, allows the model to explore different reasoning paths? Is there any evidence to support this?\n- How many samples are drawn (from Dirichlet distribution and Gumbel-softmax distribution) to form each soft thinking token? An ablation study on the effect of number of samples to the performance would give a better picture on whether soft token thinking benefits from the mixture of embeddings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The detailed description in Appendix D is very helpful in understanding soft token thinking. I highly recommend the authors to incorporate the idea of Appendix D into the main text if space allows.\n- The benefits of soft token thinking incorporated in RL training is promising, especially on larger models, according to the evidence in Appendix E.", "weaknesses": "- The Dirichlet scaling parameter is described by $\\gamma$ in Line 335 but $\\alpha$ is used in Line 370.\n- The term \"Greedy\" has been mentioned multiple times, but I find it difficult to understand without its precise definition on what it refers to. For example, \"Greedy Token Thinking\" is mentioned Section 4.4 along with \"discrete Token Thinking\", but the difference of the two methods is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DyZJ7FKkPR", "reviewer": "Reviewer_kcpe", "summary": "This paper studies Soft Thinking, a decoding method where the next input embedding is a weighted average of token embeddings according to the model’s predicted probability distribution. The community has suggested that Soft Thinking enables parallel reasoning, where multiple reasoning paths are explored simultaneously.\nThe authors perform a careful empirical investigation across multiple large reasoning models (e.g., DeepSeek-R1-Distill-Qwen-32B, QwQ-32B, Skywork-OR1-32B) and several math, QA, and coding benchmarks. Using logit and layerwise analyses, they find that LLMs behave as single-threaded reasoners: even when given continuous inputs, they collapse almost deterministically to the top-1 trajectory. This “Greedy Pitfall” shows that Soft Thinking, as commonly implemented, is functionally equivalent to greedy decoding.\nTo mitigate this, the authors propose Stochastic Soft Thinking: injecting randomness through Dirichlet or Gumbel-Softmax sampling to restore diversity. The Gumbel-Softmax variant improves reasoning accuracy and stability, with theoretical justification via Luce’s choice axiom.", "questions": "1. Soft token off-distribution effects: You claim that mixing token embeddings does not take the model off-distribution, yet the model was never trained on combinations of embeddings. Did you observe any qualitative or quantitative side effects from this?\n2. Depth of single-threadedness: You show that Soft Thinking quickly collapses to the top-1 trajectory, but is this collapse uniform across layers or token positions? Do early layers ever represent meaningful multi-path structure, or is the single-threadedness present from the start?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well-motivated study: The paper addresses a timely and widely misunderstood topic. Many groups have speculated about “continuous reasoning,” and this work decisively clarifies what actually happens.\n- Empirically clean and thorough: Multiple strong models and diverse benchmarks are used. The behavioral evidence for “single-threaded reasoning” is compelling and well visualized.\n- Clear diagnosis and fix: The “Greedy Pitfall” is an elegant conceptual summary of why Soft Thinking fails. The Gumbel-Softmax correction is simple but principled.\n- Theoretical framing: The link to Luce’s choice axiom provides a clean justification for their stochastic sampling choice, grounding the fix in existing probabilistic decision theory.", "weaknesses": "The main limitation of this paper is that its contributions concern only decoding behavior and not any internal mechanisms of reasoning. Despite its title, the paper does not truly “demystify” the model’s reasoning process. The finding that Soft Thinking collapses into top-1 decoding is empirically strong but does not explain why the model’s internal computation prefers that collapse or how it represents/sorts out alternate hypotheses. The work therefore improves our understanding of sampling dynamics, not of reasoning per se.\nA second issue concerns the claim that Soft Thinking remains “on distribution.\", this assumption is doubtful, as it was never trained on such mixed representations. The paper does not analyze whether this produces distortions in internal representations, altered activation norms, or other side effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)