Soft Tokens, Hard Truths
Authors: Natasha Butt
Ariel Kwiatkowski
Ismail Labiad
Julia Kempe
Yann Ollivier
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
large language models
math reasoning
latent reasoning
soft thinking
continuous tokens
reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: We present the first scalable method to learn continuous CoTs via RL, matching discrete tokens at pass@1 and outperforming them at pass@32.

Abstract: The use of continuous instead of discrete tokens during the Chain-of-Thought (CoT) phase of reasoning LLMs has garnered attention recently, based on the intuition that a continuous mixture of discrete tokens could simulate a superposition of several reasoning paths simultaneously. Theoretical results have formally proven that continuous tokens have much greater expressivity and can solve specific problems more efficiently. However, practical use of continuous tokens has been limited by strong training difficulties: previous works either just use continuous tokens at inference time on a pre-trained discrete-token model, or must distill the continuous CoT from ground-truth discrete CoTs and face computational costs that limit the CoT to very few tokens.

This is the first work introducing a scalable method to learn continuous CoTs via reinforcement learning (RL), without distilling from reference discrete CoTs. We use "soft" tokens: mixtures of tokens together with noise on the input embedding to provide RL exploration. Computational overhead is minimal, enabling us to learn continuous CoTs with hundreds of tokens. On math reasoning benchmarks with Llama and Qwen models up to 8B, training with continuous CoTs match discrete-token CoTs for pass@$1$ and surpass them for pass@$32$, showing greater CoT diversity. In systematic comparisons, the best-performing scenario is to train with continuous CoT tokens then use discrete tokens for inference, meaning the "soft" models can be deployed in a standard way. Finally, we show continuous CoT RL training better preserves the predictions of the base model on out-of-domain tasks, thus providing a softer touch to the base model.

Published as a conference paper at ICLR 2026
SOFT TOKENS, HARD TRUTHS
Natasha Butt Ariel Kwiatkowski', Ismail Labiad', Julia Kempe Yann Ollivier?:t
University of Amsterdam
Meta FAIR NNww York University
Work done during an internship at Mcta.
"Joint senior authors.
ABSTRACT
The of continuous instead of discrete tokens during the Chain-of- Chain-of-Though (CoT) phase of reasoning LLMs has garnered attention recently, based on the intuition that continuous mixture of discrete tokens could simulate superposi- tion of several reasoning paths simultaneously. Theoretical results have formally proven that continuous tokens have much greater expressivity and can solve spe- cific problems more efficiently. However, practical use of continuous tokens has been limited by strong training difficulties: previous works either just use continu- ous tokens at inference time on pre-trained sescrete-token model, or must distill the continuous CoT from ground-truth discrete CoTs and face computational costs that limit the CoT to very few tokens.
This the first work introducing a scalable method learn continuous CoTs via reinforcement learning (RL), without distilling from reference discrete CoT's We use "soft" tokens: mixtures tokens together with noise on the input embedding to provide RL exploration. Computational overhead is minimal, enabling learn continuous CoTs with hundreds of tokens. On math reasoning benchmarks with Llama and Qwen models up to 8B, training with continuous CoTs match discrete-token CoT's for @ass@) 1 and surpass them for pass 32, showing greater CoT diversity. In systematic comparisons, the hest-performing scenario is to train with continuous CoT tokens then use discrete tokens for inference, meaning the "soft" models can be deployed in standard way. Finally, we show continuous CoT RL training better preserves the predictions of the base model on out-of- domain tasks, thus providing a softer touch the base model.
INTRODUCTION
Large Language Models (LLMs) have achieved impressive success across wide range of reasoning tasks, particularly when enhanced with Chain-of- Thhuugt (CoT) prompting, where models generate intermediate "thinking tokens" before producing final answers. While effective, standard CoT is constrained by the discreteness of language tokens: each intermediate step must be sampled sequen- tially, which can limit expressivity and hinder exploration of diverse reasoning paths. This contrasts sharply with human cogmition, which often operates over abstract and fluid concepts rather than rigid linguistic symbols. Motivated by this gap, recent work has explored enabling LLMs to reason in continuous concept spaces, a direction often termed "continuous CoTs" (Hao et al. 2024 or "Soft Thinking" (Zhang et al. 2025).
From a theoretical perspective, continuous reasoning offers significant potential. Reasoning by Su- perposition (Zhu et al. (025a) shows that continuous thought vectors can as superposition states, encoding multiple search frontiers in parallel and enabling efficient breadth-fisst reasoning. This construction allows a shallow transformer to solve problems such as directed graph reachability far more efficiently than discrete CoT, which is forced into sequential exploration and risks being trapped in local solutions. Complementarily. Soft Thinking (Zhang et al. (025)) proposes replacing discrete ("hard") tokens with concept tokens --roaabiiity-eeeeeee -eiihtte mixtures of embeddings that retain full distributional information. This enables the model to implicitly follow multiple reasoning paths simultaneously, yielding empirical improvements in both accuracy and token efficiency.
Despite these promising claims, the practical benefits of continuous reasoning at inference time on top of discrete-token base models remain contested. In particular, Wu al. (2025) critically

---

Published as a conference paper at ICLR 2026
CoT.
CaT,
Hard
Figure 1: Hard, fuzzy and soft generation dur- ing CoT phase. hard generation, each time step, a discrete token Col; is sampled from the probability vector Pt-1 and its embedding hoon, is passed to the transformer, generating a se- quence of discrete CoT tokens: CoT', CoTr over time. In fuzzz and seft generation, at each time step, noise, 6s, is injected into the probabil- ity weighted mixture embedding, An pt-1E, where E is the token embedding matrix. This noisy input embedding passed to the trans- former, generating a sequence of continuous noisy CoT embeddings: MCoT)..... KoOT, over time. Additionally, for fuzzy generation, the tempera- ture r used in the CoT phase tends to 0, such that the non-noisy embeddings ht' reduce to embed- dings of discrete tokens. We find that the com- hination of soft/fuzzy training and hard inference typically performs hest, often matching hard train- ing at pass@1 1 and surpassing it at pass @ 32, indi- cating better preservation of diversity.
Fuzzy
Ps
P
Poot
Softmax
0-
Roat,
Linear Layer
h//
M
Transformer
oor
Embedding Layer
X
re-examine Soft Thinking and find that vanilla implementations often underperform their discrete counterparts. Their analysis suggests that LLMs, when given soft inputs, default to relying on the single probast-robbablltty sesen-effeectvelyy reducing Soft Thinking to greedy decoding. Further, existing methods for soft thinking are limited to inference on models trained with discrete CoTs.
Training of continuous-toker reasoning models has proven to be difficult, either due to computa- tional constraints from full backpropagation through all steps of continuous reasoning (this limited the CoT to 6 steps in Hao et (2024), ) or due to the necessity of strongly grounding the continuous reasoning into ground-truth discrete reasoning traces (Shen et al., (202). This is why several of the works above limit themselves applying continuous reasoning inference time without training (Zhang et al. 2025 Wu et al. 2025).
In this work, we address these limitations by developing an approach to reinforce continuous CoTs with controlled noise, making them amenable to reinforcement learning (RL) training. We theoret- ically outline two types of continuous CoT learning with soft and fuzzy tokens (see Figure D and provide extensive empirical evidence with Llama-3.x and Qwen-2.5 models trained on number of mathematical datasets (GSM8K, MATH, DeepScaleR) and evaluate on variety of mathematical and out-of-domain benchmarks. Our contributions and findings are as follows:
A continuous-token post-training algorithm. We propose the first continuous CoT fine- tuning algorithm that does not require ground-truth CoT annotations, directly tackling the challenge of learning continuous reasoning representations, at negligible computational overhead compared to discrete CoTs.
Pass@1 1 parity. We show that continuous CoT post-training is competitive with traditional discrete-token CoTs for pass @ criteria, with statistically similar performance most model-dataset combinations.
Pass 32 gains. Under sampling (pass@32) 32), continuous CoT training outperforms discrete CoT on average, demonstrating greater CoT diversity.
Improved robustness. Continuous CoT training does not degrade the base model's log- likelihood on HellaSwag, ARC and MMLU, whereas discrete CoT training does, on aver- age. Further, continuous CoT training is robust. to collapse observed with discrete training on Llama-8B- with respect to in-distribution and
performance.
Hard inference on soft models. In our experiments, adding continuous CoT at inference top of mond-token-finetunee et al. (025).. On the opposite, the best performance is obtained when using discrete CoT at inference top of a continuous CoT trained model. This means that practitioner can deploy standard inference methods directly to reap the advantages from models trained with continuous tokens.
model does not bring benefits, contrasting with Zhang
2

---

Published as a conference paper at ICLR 2026
Entropy analysis. We present a detailed analysis of the entropy profiles of models, show- ing how continuous CoT fine-tuning more closely preserves the entropy profiles of the base Llama models, compared to discrete CoT.
RELATED WORK
is natural to question whether token space is the ideal medium for reasoning, particularly in tasks that demand higher levels of semantic abstraction. However, unrolling beyond token space during pretraining departs from the original training distribution and may require additional mechanisms to help bard-token-trained LLM adapt to alternative, potentially more expressive, representations of chains of thought. Several prior works have confronted this challenge. (Goyal et al., 2024) proposes inserting dedicated placeholder tokens, "Pause Tokens", into rollouts to encourage more deliberate "thinking" in the internal layers. Coconut (Hao et al.] 2024). in contrast, explicitly ventures into continuous-space reasoning by distilling ground-truth chains of thought into continuous tokens, but its benefits appear confined to a benchmark designed for this purpose. Furthermore, Coconut re- quires ground truth chains-of-thougle and a gentle "hard-to-soft" distillation schedule, and comes with computational constraints that have limited it to 6 continuous CoT tokens.
A number of follow-up works have tried tackle these challenges (see Zhu et al. (2025b) for a sur- vey on latent-space reasoning). Given the frequent difficulty of making an LLM accept continuous tokens when they are not native their training. it makes sense to distinguish when in the pipeline continuous tokens are invoked (at pretraining, at post-training or at inference). We list those works closest to ours, while necessarily being incomplete as the scope widens; we recommend the survey by Zhu et al. (2025b) for taxonomy.
Inference. Zhang et al. (2025) introduce so-called soft tokens as the softmax layers of the output tokens, and propose to perform inference in soft-token space. Combined with a few other inter- ventions (a hand-crafted stopping criterion, and only keeping the few top dimensions of embedding space) they seem gain some performance on some benchmarks purely through inference-time interventions. A more measured view of the same is given in Wu et al. (2025). where the previous results are not confirmed, unless noise is introduced at test time into the soft token generation. Noise is important in our approach for a different reason, to provide exploration for reinforcement learning.
Post-training, A number of works aim to introduce continuous tokens during post-training as al- ready mentioned for Coconut (Hao et al. 2024). Codi (Shen et al., 2025) distill a standard discrete CoT model into a continuous CoT model, by keeping both the emited tokens and the internal ac- tivities of the continuous model close to that of the original model. Yue et al.] (2025) introduce curriculum by using mixture of discrete and continuous tokens and slowly transitioning from discrete to continuous, Gozeten et al. (2025) use continuous tokens that are trained either via direct via RL on special noise model; scalability of their method is unclear they test on a toy problem in which the CoT is length-4 binary string, which allows for full enumcration of all 16 possible CoTs.
CoT supervision
Pretraining. Several works propose changes to pretraining to incorporate some notion of "think- ing tokens" or latent tokens, and attempt to replace the CoT by some internal activations inside, or added to, the transformer. "Filler tokens" approaches (Lanham et al., 2023 Goyal et al.) 20244] Pfau (ct al., 2024, Ringel et al. 2025) introduce some bland tokens so that the model can use its contin- uous internal activations to reason while reading the bland tokens. Going further, CoCoMix (aack)
et al. 2025) intersperses continuous tokens with hard tokens at pretraining, and uses a pretrained sparse autoencoder to couple the hard and soft tokens. "Looped transformers" (Saunshi et al.) (025) and "recurrent depth" (Geiping et al., 2025) deploy internal, continuous CoTs in the depth direction of the transformer, by repeating some internal blocks and making the CoT depth potentially infinite before every token.
Theoretical arguments. Zhu et al. (202aa) make a strong argument that continuous CoTs are more expressive than discrete CoTs, natural toy problem (reachability in graphs). They prove that 2- layer transformer with continuous CoT can solve the problem in (n) vs O(n²) with discrete tokens. Continuous CoT's are provably able to superpositions that explore several reasoning paths the
3

---

Published as a conference paper at ICLR 2026
same time. The experiments align with the theoretical predictions: the superposition is learned in practice. Gozeten et al. (2025) also includes, among other contributions, theoretical argument showing that different toy problem can be solved via a 1-layer transformer with continuous tokens, but not with discrete tokens.
Diversity. Our work crucially relies on introducing added noise in the continuous model: this produces the necessary variation upon which to apply RL methods such as GRPO. Without it, the continuous-toker trajectory would be deterministic.. Prior work has demonstrated the importance of diversity in reasoning trajectories; for example, [Wann et al. (2023) leverages diverse reasoning paths sampled at inference time improve robustness and accuracy via majority voting.
is well established that adding noise can, by itself, improve performance of neural networks, for instance via dropout (Srivastava al. 2014) input perturbations as denoising auto-encoders (Vincent et al. 2008). NEFTune and SymNoise (Jain et al. 2023 Yadav & Singh 2023) apply this idea to the supervised fine-tuning phase of LLMs, by having them predict the unnoised sequence from version of the sequence with noise added to the input embeddings. We use a similar noise structure.
Yet in our case, the benefits cannot come solely from the regularization effect of noise, since that would correspond to the noisy model at step in our training curves, and the gains appear after training. Our approach explicitly optimizes the policy via RL to benefit from a diverse reasoning manifold during learning. The beneficial effect added noise for boosting exploration and diversity in RL has been established: for instance, Fortunato al. (2018) found benefits from adding noise directly to the weights of the value or policy networks in RL settings.
METHOD
In nutshell. Similarly to Coconut Soft Tokens, our method keeps the full probability distri- bution after the softmax step, instead of emitting discrete token. This mixture of tokens is fed to the input of the transformer for the next step. Contrary to prior work, we then inject noise the input embeddings. This noise produces the necessary exploration to apply RL fine-tuming. In con- trast, Soft Tokens do not fine-tune, and Coconut relies on backpropagation through time through the whole continuous CoT (coming with strong computational limitations) plus curriculum to ground the continuous CoTs into ground truth discrete CoTs. We now describe our method in detail.
Notation for transformers. We decompose a standard LLM architecture into the following blocks.
We denote by V the vocabulary size. Thus, each token can be seen as a one-hot vector It E TRV and the sequence of tokens vetr up to time i can be
matrix of size f. V.
We denote by the token embedding matrix of an LLM, that takes sequence of tokens V't and returns sequence of embeddings 12, by linear, token-wise mapping
#or E.
(1)
Denoting by no the dimension the input embedding of the transformer stack. E is matrix of size V x 0g-
We denote by the transformer stack, that turns the sequence of input embeddings no, into sequence of output embeddings
h2 = (1222)
(2)
where L is the depth of the transformer stack. 16, is matrix of size t ny with n. the dimension of the output layer of the transformer stack.
Probabilities for next-token prediction are obtained as follows. The output encodings h turned into logits by a decoding matrix Wa of size n. x V where n. is the output dimension of the transformer stack. The next-token probabilities are obtained by applying a softmax at temperature 7 2 0 to these logits:
Per = softmax(h Wa/T)
(3)
where P-r is a matrix size t V, the softmax is applied for each independently, and softmax is extended by continuity for temperature O.
4

---

Published as a conference paper at ICLR 2026
Standard (hard) tokens. In standard, hard-token models, each token Iı is one-hot vector I RV. At inference time, to compute next-token prediction zt given the sequence of previous tokens Fect first computes the next-token probabilities Pct given zet above. Then the next token is sampled according to the last component Pt-1 of pet:
Pr(x, = 1;) = 1t-
(4)
where 1 denotes the one-hot encoding of token This is applied inductively get the sequence of next tokens.
Soft thinking. In soft thinking (Zhang et al. 2025 Wu et al. 2025). during the CoT phase, instead of sampling next token 2; according the probabilities pt, the probabilities are directly used to define a mixture embeddings. The next input layer embedding is obtained as
n Pr((, = 1.)e P
(5)
where is the embedding for token i. Then the transformer stack is applied normally to ho.
After the CoT phase done, the model samples normal (hard) tokens.
This model is not amenable to direct RL training via Reinforce-like algorithms, because of the absence of noise or random choices: the whole CoT is deterministic and differentiable function of the prompt. In principle, it could be optimized directly by backpropagating through all the timesteps of the CoT, similarly to Backpropagation Through Time (BPTT). But this leads to technical and memory challenges that we will not discuss here.
Noisy soft thinking: soft tokens and fuzzy tokens. Instead, we propose to make soft thinking trainable by RL, just by introducing noise into the soft thinking process. We simply add noise the computation of A.:
h P:-1E + oN (0,
(6)
with some standard deviation a > O. Then, at the next timestep, the transformer stack fed h..
We also experimented with adding noise other places, such the logits (Appendix F22
We call this model soft tokens; we use the term fuzzy tokens when the temperature t used during the chain of thought tends to 0, because in that case, the non-noisy embeddings ht" reduce to embeddings of true discrete tokens, so R are normal discrete tokens embeddings up noise
Reinforcement learning on soft tokens. Introducing exploration noise on the soft CoT tokens makes possible optimize the model via reinforcement learning. (For traditional discrete CoT tokens, exploration comes from the random sampling of a token from the softmax probabilities.)
We describe here the derivation of Reinforce for noisy soft thinking. More advanced Reinforce-liks methods, such as RLOO, GRPO, PPO etc., are derived from Reinforce in the standard way.
Given prompt, we sample a soft CoT, then sample final answer given the COT. Let R(a) denote the reward obtained for an answer a. The objective is to maximize the expected reward.
The CoT sampling is fully defined by the sampling of the noisy soft tokens 7.0 Therefore, the objective is to maximize the expectation
(h.))=x((( [R(a)]
for a given prompt, where ı is the current model.
By the standard Reinforce theorem Sutton & Barto 1998), this is equivalent to minimizing the loss
E (a
R(a) (log n (f0) + log log [(a]()))
'The model used in Coconut (Hao et al. 2024) is slightly different in that it directly feeds the output em- hedding next-step input embeddings, namely, hy h assuming dimensions are the same. This bypasses the expansion from hidden dimension to vocabulary size and hack, well the softmax and temperature.
5

---

Published as a conference paper at ICLR 2026
The term log iT (allf") just represents fine-tuning the answer given the CoT, and can be computed in a standard way, since sampling is done a standard way.
The term log π (A*) can be decomposed as sum over timesteps,
log ग (h) - Σ log π (1)
and each of those terms can be computed easily: indeed, knowing the soft tokens i we can compute the non-noisy next-token input embedding ny- Since the noise is Gaussian, we just have:
log T (1 ス [6 ng est
(10)
and note that hy is differentiable function of the previous soft tokens 12. depending the parameters of the model.
This makes it possible to apply the family of Reinforce-like algorithms to noisy soft tokens.
Computational overhead is minimal: storing the probability vector p; at each step (vector of size V), and injecting noise on the first layer.
4
EXPERIMENTS
Models tested.
We train three variations of CoT models described in Section 3
Hard tokens: Categorical sampling of ordinary hard CoT tokens with temperature 1.0.
Soft tokens: Instead of sampling hard tokens, we use the full probability mixture at tem-
perature 0.5 compute embeddings, and add Gaussian noise to the embeddings.
Fuzzy tokens: Like soft tokens, but at temperature 7 = 0.0001, which brings them very close hard tokens embeddings, to which add Gaussian noise.
For the scale of the Gaussian noise, we set this equal to 0.33 times the root-mean-square norm of the token embeddings, so that the noise is comparable but a bit smaller than the embeddings. In practice we observe our algorithm is robust to ratios less than or equal to 1.0 (Appendix [.I). In further ablations (Appendix F.4). we observe that our algorithm is also robust to temperature values E [0.0001, 0.1).
Inference settings. At test time, we decouple the inference method from the training method: for each trained model (hard, soft, fuzzy), we evaluate six inference settings. We vary the decoding of the CoT as follows, but with answers always greedily decoded temperature 0:
Hard Greedy: discrete tokens, CoT temperature т = 0.0 at test time Hard Sample: discrete tokens, CoT temperature т 1.0 test time
Soft Greedy: Gaussian scale o 0.0, CoT temperature т = 0.5 at test time
Soft Sample: Gaussian scale o = 0.33 root-mean-square norm, CoT temperature 7 = 0.5
at test time
Fuzzy Greedy: Gaussian scale a = 0.0, CoT temperature 0.0001 at test time
Fuzzy Sample: Gaussian scale 0.33 root-mean -sueae norm, CoT temperature 0.0001 at test time
For instance, "soft" training with "hard greedy" testing amounts training with mixture at 7 0.5 with Gaussian noise, then applying the model with hard tokens at test time.
The "sample" settings are the same the variants used during training for the CoT, while the "greedy" setting more aggressively target the mode of the distribution at each step of the COT.
Reinforce with group baseline. We fine-tune the models with RLOO, namely, Reinforce using a per-prompt leave-one-out (LOO) group baseline (Kool et al.) 2019): for each sample and reward, subtract the average reward obtained the other samples for the same prompt. We include the RLOO loss in Appendix A for completeness.
At each update draw mini-batch of B 2 distinct prompts {ro) For each prompt sample = 32 sequences yb.g that contain a chain-of oft---f-toh-- thought (CoT) followed by a final answer. The prompt instructs the model end the CoT with "The final answer is: (see Appendix

---

Published as a conference paper at ICLR 2026
Training
Valklation
Traiting Seacem Bote:
VallSatinn Success State
2003
1300
Tosining Stepp
Thiining Steps
Pray
Figure 2: Llama 3b Instruct trained on GSM8K (a) Training performance across steps; one step = two prompts x 32 samples each. (b) Greedy validation performance used for model selection. For the remaining trained models, see Appendix G.T
Rewards are computed only on the final answer using the Math Verify package (Kydliček) against the ground-truth label:
2025)
100,
if Verify(as..) = 1,
Tb.g =
10. if Verify (4s.g) = 0 and Extract (axed)))))) 1,
(11)
O, otherwise.
Datasets and base models. We train Llama 3.2 3b Instruct, Llama 3.1 8b Instruct (Dueey et al,, 2024 and Qwen 2.5 3b Instruct (Yang et al.] 2024 math reasoning datasets including GSM8K
(Cobbe et al. 2021). MATH (Hendrycks et al. (2021b) and DeepScaleR (Luo et al. 20255..
For each model trained datasets: GSM8K, MATH and OlympiadBench (He et al., (024). For OlympiadBench, following prior work, we use the 675 subset of math questions which have final answers and do not contain images or figures. Similarly, for MATH, we evaluate on the MATH-500 (HuggingFaceH44 2025) subset of the MATH test set. assess out-of-distribution generalization, we also test the resulting models on standard benchmarks: HellaSwag (Zellers et al.) 2019), MMLU (Hendrycks at. 2021a and ARC/A12 Reasoning Challenge (Clark et 2018).
dataset, evaluate test performance on three math reasoning test
On each training dataset, we train for 4k steps and monitor greedy validation performance; the final model used for testing
best performing under our greedy validation performance, (For hard CoTs, greedy performance refers to greedily decoding CoT's and answers. For soft and fuzzy CoTs, greedy performance refers to decoding CoTs with no Gaussian noise and greedily decoding answers.) During training, sample maximum of 128 CoT tokens for GSM8K and 512 CoT tokens for MATH and DeepScaler respectively, under our early stopping criterion (see Appendix[C] on all datasets sample 32 answer tokens. For evaluation, in all cases, sample maximum of 512 CoT tokens under our early stopping criterion followed by 32 answer tokens.
with independent random seeds; the tables report the resulting mean and standard deviation. Training and validation success rates may be found in Figure 2 For details on hyper-parameters, see Appendix D
Each setup
Results. Across datasets and models, the three training schemes are broadly comparable, gen- erally achieving similar greedy pass@1 performance shown Table I This demonstrates that fuzzy and soft training are effective. A notable exception is Llama-3B I-Intrctct trained on MATH, where hard CoT outperforms soft CoT approximately 5% in pass@ On the other hand, soft and fuzzy training have clear overall advantage for pass @ 32 over hard training (this signal is clearest. on Llama, as shown in Figure 3)
We observe gap between the hard greedy and hard sample pass @ for the base models and models trained with fuzzy/soft CoTs, whereas the gap for models trained with hard CoTs is very small (see Figure 3 and Table We note that, for the base and fuzzy/soft trained models, not optimal for pass @1 (greedy 7 = 0 is always better). At low values of k. the optimal т may be some interpolation between 0 and 1. Further, for pass @ 32, also observe closing of the gap between the trained models and base models. This loss of diversity is well reported for Reinforce style algorithms where the reward is based answer correctness alone (Song et al. 2025).
1 is evidently

---

Published as a conference paper at ICLR 2026
Llama 38 Instruct Traln: CSM&K CSSMK
Liana instruct Train: (SMMKK Test: GSMSK
Liama Instruct MATH Test: MATH 500
Lima 28 Lastruct Train: DecpSrallR Test: MATH 500
Tooak C4)
Trining Methool
Inferenas Method
No Pisetinee
Fardd
Piazy
Sclt
Sampk
Groedy
Figure 3: Hard inference pass@k for Llama models (for soft/fuzzy inference and Qwen see Appendix G.2 We observe softffuuzy training improves pass@ 32, pointing to preserved diversity. Greedy Pass@ (the
triangles) for all training methods clustered together.
For all training methods (hard, soft fuzzy), hard inference generally performs best, both for pass @1 1 and pass @ 32 (see Appendix(E).o In particular, we do not confirm previously reported benefits of soft inference on hard (normal) training (Zhang 2025
GSMSK
MATH-500
OlympiadBench Sample pass@
Greedy
Sample
Sample pass 32
Groady
Sample
Sample
Greedy @
Sample
Model
Training
pass@r
pas
pas
pass
no linctune
71.42:00
45.020.0
96.820.0
38.0=0000
25.2::0.0
82.020.0
17.900.0
12.0:0.0
52.320.0
gsan8k
7..
74.320.8
94.120.3
34.6=0.2
31,340.4
72.42111
10.620.9
9.420.5
39.31:13
gsm8k fuzzy
76.7+
66.422.4
97.420.3
42.6-3,2
32.4#1.7
83.2+:1.1
17.4+0.9
13.0=111
55.2+:1.
gsmökk soft
77.2++0.9
7.66:34
97.9-1.3
43.5
37.2++100
8488-0.4
18.9:-0.2
15.5=007
18.
instruct
math hard
80.0+-055
79.7-0.8
96.7440.1
1..1-1.7
4..8++0.9
78.7 78.7+-0.9
22.7+-1.1
22.2:0.4
1.3
math fuzzy
79.6.1..4
68.512.11
97.6.10.3
48.5
37.311..5
81.51.0.7
21.610.6
15.910.8
52.8.166
math soft
76.84:10
70.7=2.4
97.8:10.3
44.5=0.6
38.92119
82.2::166
19:24:1.1
16.6:20.4
55.81:12
deepscaler bard
79.7
79.620.2
96.62003
49.6=0.9
48.1
78.1200.7
23.24:1.4
23.8:066
50.81:5
doopscaler fuzzy
788800.
69.8=4.7
97.7:0055
46.5-2.2
38.6+22.6
83.3-003
19.5+1100
16.6=144
54.4+2.1
deepscaler soft
7..9
71.0+155
98.0+-0.1
44.7=1.2
37,9++13
80.6+:222
21.0+0.6
16.4+0.7
53.3+-1.7
no finctune
82,620.0
64.9=0.0
98.5:00.0
44.4=0.0
31.100.0
79.8=0.0
19.62:.0
11.7=0.0
52.620.0
lama Sh
gsm8k hard
81,2+0,2
80.7=0.3
95.1=2022
20.2=0.8
19.8:14
45.4=33.2
3.:00.5
3.420.55
16.4#244
gsmök fuzzy
83.7+11.3
73.1=300
98.2:10.2
44.6-211
338++277
83.1-009
18.0:1144
12.5+133
56.0+:2.6
gsmökk soft
1..6
73.3+:3.9
9..
44.7
34.8+2222
1.
17.91:10
13.1:0077
1.2
no finetune
8.
10.0
17.2.0.00
95.1.10.0
29.0
25.5.10..0
81.01:0.0
16.910.0
14.3.10.0
50.2.10.0
qwen 36 instruct
math hard
84.0++..8
83.0=0.2
97. 07.
59.0-177
$7.1
83.6::1.0
29.41:.3
27.8-0.5
61.1:10.6
math fuzzyy
84.4+-0.8
81.6=-0.9
98.1-10.2
58.1
55.5++1.2
84.4-002
27.24-0.5
24.4-0.7
60.7-0.5
math soft
82.9 $2.9
78.7=2.6
97.6-20.5
54.7=0.3
$2.2+1.11
84,4=0.7
24.3+:1.8
22.0=20.5
18.5+1.0
Table 1: Results of hard inference on GSM8K, MATH-500 and OlympiadBench test sets. In blue the best pass@! @1 1 performance and in green the best pass 32 for each (base model, training set) pair. We observe broadly comparable performance at pass @1 and improved soft/fuzzy training pass @32. For comparisons of hard, fuzzy and soft inference, see Tables
in Appendix
One setup stands out: when training -i-mn-----nssucct on GSMSK and testing on MATH-500, only fuzzy and soft-trained models achieve good scores, while classical hard token fine-tuning is inef- fective. Namely, gsm8k-hard training sharply underperforms on out-of-distribution MATH (hard- greedy 20.2% and pass@32 45.4%) whereas gsm8k-fuzzy and gsm8k-soft trainings recover to 44.6-44.79 greedy and 83.1-83.99 33.9% @ while maintaining in-distribution performance on GSM8K. Glama-8B-Instrue has a good performance from the start on GSM8K (presumably because it was exposed to this dataset), but this does not translate to good performance on MATH. Further hard fine-tuning makes things worse, but further soft/uuzyy fine-tuning on GSMSK brings improve- ment on MATH. Fuzzy/soft training appear to bring more generalization Llama-8B ---strruct.
Out-of-domain robustness. One risk of LLM fine-tuning on a dataset is degrading the general performance of the model other datasets. To assess this, test the trained models on three
8

---

Published as a conference paper at ICLR 2026
standard benchmarks in Table 2 We report both the success rate (with hard greedy sampling) and the negative likeliiodd per token (NLL) of the correct answer.
Hellaswag
ARC
MMLU
Model
Training
Accuracy
NLL Correct
Accuracy
NLL Correct
Accuracy
NLL Correct
finetune
66.46100.00
2.53.10.00
72.7910.00
2.86.10.00
60.8510.00
1.6110.00
math hard
66. 06.10:10.10
2.57.00.00
73.45
3,19:10:03
61.11
1.651000
math fuzzy
67.00:00112
2,53++0.00
73.25+-0.32
2.88++0.03
61.16+00.04
1.00+0.02
Dana 36
soft
67.12:10.06
2.53:00.01
12.76:10.18
2.92.10.02
61.2010.14
1.59.0002
instract
deepsculer hard
66. 18.00.2
2.58.10.01
73.9910.55
3.18.40.03
60.66.100.37
1.66:10.01
deepsculer fuzzy
66.88-00066
2.55++0.01
73.05-+0.62
2.94+:0.33
61.20:00.0
1.00+-002
deepsealer soft
66.48:00.66
2.56:20.04
72.85=0.39
2.93:1-008
60.34=1.20
1.68:40.12
gamiSk hard
66 66.44+0.18
2.57=00.00
73.53+00.25
3,16+t0.01
61.48=00220
1.65++0.01
gom8k fuzzy
66.97:-0.21
2.53++0.01
13.10+0.115
2.89:+0.01
61.30+0021
1.60++0.02
gsmik soft
67.13:0.19
2.53::0.01
72.99=0.36
2.89::0.04
61.3620,30
1.59:00.01
finctone
74.24:0.00
2.34:0.00
81 $1.00+0.00
2.82::0.00
69.00+0.000
1,0+00000
Uama Kh instract
gsmök hard
74.44:0.10
2.37=00.00
81 81.49:0.15
3.13+20.03
68.90
1.1+00.00
gam8k furzy
74.41+00001
2.35-:0.00
81.63+-0399
2.81-:0.01
68.90--0111
1,40-:0.00
gsm8k soft
74.32=0.08
2.35::0.00
81.4620.14
2.82:1.0000
68.78:0.14 14
1.40:00.01
finetune
74,9410.00
2.29:10.00
83.0010..00
4.67.00.00
68.0810.000
1,1910.000
gwen
math hard
75.01.:0.10
2.30.1..00
82. 82.86.10.40
5.62.10.12
67.86.10.06
1.51.10.22
instruct
math fuzzy
75.07+00.02
2,29:+0.00
83.09
4.74+0.11
68.16--0.07
1.21++0.02
soft
75.25:00112
2.30::0.01
12:0051115
5.49:10.64
68.01:0006
1.33:00.66
Table 2: Out-of-domain results. Outside of mathematical reasoning domains, fuzzy and soft training on average result in lower negative log-likeldhood of the correct answer compared to hard training, indicating softer touch on the base model's capabilities.
The results show comparable success rate for the three training methods (hard, fuzzy, soft). However, the NLL of the correct answer is visibly better for fuzzy and soft than for hard, especially on ARC but also with Qwen on MMLU: hard training degrades the base model NLL on out-of-domain datasets, while fuzzy and soft training preserve it.
Entropy behavior. Next, we report the entropy of the distribution of next-token predictions during the CoT, function of the index within the COT.
The base Llama models exhibit a very different entropy profile whether greedy or temperature sam- pling is used. This shows difference in next-token prediction behavior on prefixes sampled from temperature T = 0 or T = 1: with the latter, entropy blows up as the CoT progresses with hard sampling, indicating very high uncertainty as the CoT goes on. Interestingly. soft or fuzzy sampling on the base model does not show any entropy blowup. The entropy blowup is also not present on Qwen, as seen in Appendix [e.]..FFiuua Figure 17 Analyzing base models not our main topie, but we still report how this entropy proñle changes after different types of training.
20 Appendix G.4), soft or fuzzy training keep roughly the sume entropy profile as the base model, whether inference greedy or sampled. On the other hand, hard training changes the hard sampling entropy profile to resemble greedy sampling on the base model: entropy values are substantially lower. We pose that an explanation for this is that hard training makes the model overconfident, consistent with lower muss@32, the worse NLL values we observe on out-of-domain tasks (see Table 2 and the occasional performance collapse see for hard training (see Table
As observed in 4 (and Figures 18
5 CONCLUSION
We have introduced the first reinforcement learning framework for training continuous Chains-of- Thought in LLMs with minimal computational overhead and without relying on ground touuhd-truth dis- crete CoTs. Across mathematical reasoning benchmarks, our approach performs on par with discrete token training for pass@) I success rate on most model-dataset combinations and improves pass @32 scores. Moreover, it seems fine-tune the base model with a softer touch, better preserving the
2On Qwen no fine-tune, we observe much lower performance compared to what is reported in the Qwen report Yang et al. (2024). The discrepancy is due our (zero shot) prompting and the resulting CoT generation format, and does affect our RL. trained models (see Appendix for details).

---

Published as a conference paper at ICLR 2026
Hard Greedy
Puzzy Greedy
Soft
Hard Sample
Puzzy Sample
Soft
Eabopp
Eniropy
Training
Enetznee
Bazzy
Soh
Figure Llama Instruct CoT entropy on GSM8K test set. Fuzzy and soft training preserves entropy profile of base models; we observe a large change in hard sample profile with hard training.
model's 's out-of-distribution behavior. This suggests distinct behavioral differences between soft and hard reasoning processes. These results provide evidence that continuous reasoning is not just theoretical curiosity but practical alternative for fine-tuning large models.
Acknowledgments We thank Shobhita Sundaram for the helpful discussions. JK acknowledges support by the Simons Foundation through the Collaborative Grant "The Physics of Learning and Neural Computation".
Reproducibility Statement To enable independent re-implementation, provide in the paper and Appendix: (i) datasets and base models used for training and evaluation (ii) full hyperparameter details and training objective; (iii) prompt template and inference settings for evaluation; (iv) verifier used; (v) number runs per result, with mean + std; and (vi) compute details.
LLM Usage We used LLMs for drafting and polishing the writing. The research itself, and the
bulk of the writing effort, were done by humans.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Peter D. Turney, and Daniel Khashabi. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.052 2018. URL https://arxis: http://aaxxx..oo.../o///////////
05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 1Xiv:2110.14168,
2021. 7
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The Ilama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024. 7
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference on Learn- ing Representations, 2018. URLhtttss://pperee/// weetarewwwwwweeeeeeeeeeeee Eorum?id- http://ww....gaaoee..e.ee//e///r
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with la- tent reasoning: A recurrent depth approach, 2025. URL ftttp:://axxiv.or///abbao
05171
B
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference Learning Representations, 2024. URL httt:s:::/ppeereeee
net/ fet//ooummidd www.ooooiiiddoootoo ph04CRkPaC
Halil Alperen Gozeten, M Emrullah Ildiz, Xuechen Zhang, Hrayr Harutyunyan, Ankit Singh Rawat, and Samet Oymak. Continuous chain of thought enables parallel exploration and reasoning. arXiv preprint arXiv:2505. 25.4445005.3648, 2025.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason a continuous latent space. arXiv preprint arXiv:2412.06769 2024.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad bilingual mul- timodal scientific problems, 2024. URL https::////// http///axvvvvvvo////// org hbrgg//bb/2422. 14008
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- fional Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021b. 7
HuggingFaceH4. Math-500.
co/ http:///g......//.aaaaaa///////n
MATH-500) 2025. Subset of MATH used for evaluation. Accessed 2025-09-03. 7
Neel Jain, yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. Neftune: Noisy embeddings improve instruction finetuning. urXiv preprint arXiv:2310.05914 2023. 4
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free!
In ICLR 2019 workshop: Deep RL Meets Structured Prediction, 2019. 6 14

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. 24
Hynek Kydlíččk. Math- Veriff: Math Verification Library, 2025. URL BtRLtttt//ggthiiii com/ hugggingaacce/maah-eeeee
Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Her- nandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilè Lukošiūté, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang. Thomas Henighan, Tim- othy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield-Dodss, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. Measuring faithfulness in chain-of- thain-of-thhught rea- soning, 2023. URL https:: / arxiv. org/ abs 2207 13702
3
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang. Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025. 7
Jacob Pfau, William Merrill, and Samuel R. Bowman. Let's think dot by dot: Hidden computation in transformer language models. In First Conference on Language Modeling, 2024. URL https: //openzeviee. wwt.
3
Liran Ringel, Elad Tolochinsky, and Yaniv Romano. Learning a continue-thinking token for en-
hanced test-time scaling, 2025. htLpstps: /ttp::::////xvv.../////vas. arxiv. org/ aaa///a////555 112 74
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li, Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers. In The Thirteenth International Con- ference on Learning Representations, 2025. URL
id=din01Gf2Fd
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing Shain-of-thougee into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025.
3
Yuda Song, Julia Kempe, and Remi Munos. Outcome- based exploration for Ilm reasoning, 2025. URL https:://axxii
abs 220...2599..0 06941
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 19()::992-1195) 2014.
Richard Sutton and Andrew Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. 5
Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuan- dong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts, 2025. URL
https://arxxt wwwp:d/..rr//bb//////////
/2502 08524 3
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https::/ hetp:://peeeeviiiie www.viewwwe net forummiidd 1I
Chünhung Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai, and Hua Wu. Lims tingle-threaded reasoners: Demystifying the working mechanism of soft thinking, 2025. URL
https::/arxiv. org /208 03440
18
12

---

Published as a conference paper at ICLR 2026
Abhay Kumar Yaday and Arjun Singh. SymNoise: Advancing language model fine-tuning with symmetric noise. arXiv preprint arXiv:2312.01523, 2023. 4
Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2442.151115, 2024, 7
Zhenrui Yue, Bowen Jin, Huimin Zeng, Honglei Zhuang, Zhen Qin, Jinsung Yoon, Lanyu Shang, Jiawei Han, and Dong Wang. Hybrid latent reasoning via reinforcement learning. arXiv preprint
arXiv:2505. 18454, 2025. 3
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Annual Meeting of the Association for Computational Lin- guistics. 2019. URL HtRL/ttps::// pattps://api. apii swmanticscoolar.oooo htt.:///........oooo............
Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of LLMs in continuous concept space, 2025. URL wtR.ittps://axi..oo////s
250.. 15778
15
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reasoning by superposition: A theoretical perspective on chain of continuous thought, 2025a. URL https:
12514
Rui-Jie Zhu, Tianhao Peng. Tianhao Cheng, Xingwei Qu, Jinfa Huang, Dawei Zhu, Hao Wang, Kaiwen Xue, Xuanliang Zhang. Yong Shan, Tianle Cai, Taylor Kergan, Assel Kembay, Andrew Smith, Chenghua Lin, Binh Nguyen, Yuqi Pan, Yuhong Chou, Zefan Cai, Zhenhe Wu, Yongchi Zhao, Tianyu Liu, Jian Yang, Wangchunshu Zhou, Chujie Zheng, Chongxuan Li, Yuyin Zhou, Zhoujun Li, Zhaoxiang Zhang, Jiaheng Liu, Ge Zhang, Wenhao Huang, and Jason Eshraghian A survey latent reasoning. 2025b. URL https: / arxiv. org/ abs 2507 06203
13

---

Published as a conference paper at ICLR 2026
A
REINFORCE WITH LEAVE-ONE-OUT (RLOO) BASELINE
For completeness, we describe here the implementation of RLOO that we use, directly based on Kool et al. 2019).
We fine-tune with Reinforce using per-prompt leave-one-out (LOO) group baseline. At each up- date we draw a mini-batch B distinct prompts { ) 6= For each prompt we sample sequences Yb.g that contain chain-of-thoure (CoT) followed by final answer. Each such sequence provides a reward Tb.9 as described in Section
4
For each prompt Tb. the LOO group baseline for sample g averages the other G-1 rewards from the same prompt:
Tb.
Ab.g
Tb,9
7/
(12)
G
Baselines and rewards are treated constants w.r.t. 0; i.e., stop-gradient.
Let 1b.g.1:Th.s denote all tokens in Yb.g (CoT answer). With policy Te. the per-sequence log- probability
Tb.o log TT (bh.g. (bb.g.ct, 2n).
bb
(13)
Our loss is the advantage- advantage-weighted weighted negative log-likelihood averaged over batch and group:
B
L(0) = BG
Ab. ls.gr
(14)
b=1
g
B
To.
VeC(O)- = BG
Ab.a
V log To (Bb.o 2b).
(15)
b=1g=1
t=1
where &6 denotes a stop-grad operator on the advantages.
B TASK PROMPT
To guide the model's behavior during experiments, provided the following explicit instruc- tion. This prompt ensures that the assistant follows structured reasoning process before giving the final answer.
Task Prompt
A conversation between User and Assistant. The user asks question, and the Assistant solves it. The assistant first shows the complete reasoning process step by step, then provides the final answer in \boxed(). The assistant must always follow the format: 'User: [question] Assistant: [detailed reasoning) The final answer is: (boxed [answer]].' User: QUESTION Assistant:
C STOPPING CRITERION AND PREFILLING
Stopping criterion.
Hard: Monitor the generated text and stop as soon as ends with The final answer is:
14

---

Published as a conference paper at ICLR 2026
Soft/Fuzzy: During continuous generation, form greedy shadow sequence by taking the higbest-probubility hard token at each step and stop when this shadow ends with The final answer is:
neither condition is met, decoding continues until the maximum chain chain-of-thought length L.
Prefilling.
If early stopping is reached, prefill \boxed(
If T. is reached, prefill The final answer is: \boxed(.
D HYPERPARAMETER SEARCH
All models were trained with the AdamW optimizer and cosine learning-rate schedule using 20 warm up steps.
We tuned all hyperparameters using greedy validation performance. The same learning rate and scale factors are used for fuzzy and soft, thus we only sweep over values for fuzzy.
Learning rate. For both the hard and fuzzy, swept over {le-5, 9e-6, 2e 1e 1e-63 63 for each combination of models. The same optimal rates were found for both hard and fuzzy:
Llama 3B Instruct: 6e-6
Llama 8B Instruct: 3e-6
Qwen 3B Instruct: 8e-6
Scale factor (fuzzy only). We additionally swept scale factors . 10} on the root mean square embedding norm across various models and found 0.33 to be best, though most values below 1 performed well. An ablation in Section F.1 supports the finding that our algorithm is robust to scale values below
E RESULTS ON SOFT AND FUZZY INFERENCE
We evaluate base models and all RL trained models (hard, fuzzy and soft) under six inference set- tings: hard greedy, hard sample, fuzzy greedy, fuzzy sample, soft greedy and soft sample, described in Section Here, we report test performance on GSMSK (Table 3). MATH-500 (Table 4 and OlympiadBench (Table 5). Contrary to previously reported benefits of soft inference hard trained models (Zhang et al. 2025). we do not observe any improvement in soft inference. Interestingly, we do not observe any gains of soft inference soft trained models either. In our experiments, hard inference on all models achieves the best performance.
F ABLATIONS
All ablations were run with two independent random seeds per experiment.
F.1 NOISE SCALE
for Llama 3b Instruct trained with the fuzzy RLOO, varying the scale factor, 7, applied to the root mean square embedding norm to com- pute the noise scale Further, in Figure we report the greedy validation performance on GSM8K train set. We observe that our algorithm appears robust to scale factors and below; whereas for 7 3, there is a collapse in learning as noise gets too large.
In Table 6
we report performance on GSM8K
F.2 NOISE PLACEMENT
report test performance GSM8K test set for Llama 3b Instruct trained with the fuzzy and soft RLOO, varying the placement noise in our model. We consider placing both the noise the final hidden layer and on the logits instead of the embeddings. We pose that the
In Table 7
15

---

Published as a conference paper at ICLR 2026
Inference Settings
Fuzzy
Fuzzy
Fuzzy
Model
Training
Greedy
Sample
Sample
Greedy
Sample
Sample
Greedy
Sample
Sample
pass
pasei)
paseei 32
pass@i
paseó
pass 32
pass
pussi 1
paar
no finetune
71 0.4
45.0=00.0
96.8++0.0
7..5-0.0
69 09.00000
93,940.0
68.4:0000
65.2:000
94,940.0
gsm8k hard
1.3
74.3+40.8
94.1+0.3
7..5-0.6
74.7=0.5
92.340.4
75.7=0.5
74.240..
92.44.11
gsmök fuzzy
76.7 1.8
664:22
97.4:0..3
76.4
73.2 1.8
92.041111
75.1=1.8
13.54117
93.540.88
g5m88 soft
77.2-0.9
70.5=3.3
97,9+0.3
76.8-088
7..0::144
93,440.3
74.5=11.5
74.1=2.
94,240.2
instruct
math hard
8..0=0.5
79.7:00.8
96.7..0.1
8..
78.9:10.7
95.3110.5
79.5=0.8
78.6:10.7
95.540..3
math fuzzy
1.46-14
68.5+22.1
97.6++0.1
78.8 0.9
78.1=
95.2:40.1
77.9-0.9
75.9+111
95,7++0.3
94.6.10.6
deepscaler hard
79.7-1.3
29.5:10.3
96.6.10.3
79.6=1.1
78.9.10.5
94.7.10.6
79.8
78.5:0..5
94.8.0.22
doepscaler Tuzzy
7888-088
97.7100.5
78.3=0.9
77.2+111
94,800
75.4113
95.3400.6
deepszaler solt
77.9 1.8
71.0:114
98.0.10.1
77.5 1.3
76.9.114
94.4.10.6
77.5
75.5.1.
95.2.104
finetune
82.6:0.0
64.9:00.0
98.5+000
82.6=0.0
09.3::0.0
96.12000
81.2=000
63.8+00
96.1+00
lama 8h instruct
gsmik hard
81.2-0.2
80.6:1044
95.1:10.2
80.0=13
69.0:10.7
94.510.3
78.3=2.2
68.3:1077
94,340.1
gsm8k fuzzy
83.7-1.J 1.1
73.2:13.0
9.22:10.2
83.4
7..5
95.54...3
82.3-0.8
69.8-177
95,540.7
gsmök soft
82.6-1.6
73.3:139
98.3:10.2
82.4-1.7
72.91100
95.2410.8
81.6
69.0:1122
95.610.4
finetune
8.4-0.0
17.2:00.0
95.1+000
10.5-0.0
11.5+0.0
58.5400
8.6+0.00
1384+00
71,9400
qwen 36 instruct
math hard
8..0-0.8
82.9++03
97,2+0.3
84.0-0.9
83.2+0.3
94.6+t0.2
84.0-0.7
83.0+022
94,94022
math fuzzyy
8..4
81.6:110
98.1110.2
84.J
84.210.33
94.311.11
84.0-066
95.010.3 84.10.33
math soft
829==0.9
78.7:2.6
97.6:10.5
82.5=1.3
82.12:13
94840.
82.9=111
819911
95.410.4
Table Results GSM8K test set. In blue the best pass@ I performance and in green the best pass @32 for each (base model, training set) pair.
Vidiiion Sa0000 Reta
Viltatoo Saecoos Rate
Stops
0.00
Figure 5: Validation performance for: (a) noise scale ablation, (b) temperature ablation, on Llama 3B
Instruct trained with fuzzy models GSMSK. Fuzzy training appears robust to noise scule factors 0.1-1.0 and temperature values 0.1-0.0001.
Vildition Succco Bate
Praca Longls
Sclt Final Middes
Sull TopitD
Sott Legits
Puscy Fical milden
Sent Topls
Scnt Leght (Conndat)
Scft Leghts (Dambe
Figure Validation performance for (a) noise placement ablation (b) Gumbel noise ablation on Llama 3B Instruct trained with fuzzy/soft models on GSMBK. We tried placing the noise on the (top-k) logits and final hidden layer outputs; only placing noise the top-k=5 logits shows signs of learning.
16

---

Published as a conference paper at ICLR 2026
Inference Settings
Hard
Hard
Hard
Fuzzy
Fuzzy
Puzzy
Soft Greedy pass@! 1
Soft
Soft
Truining
Greedy
Sample
Sample
Greedy pass@ 1
Sample
Sample
Sample pass
Sample
pass@1
pass
pass 32
pass 1
pass 32
pass 32
38.0-000
25.2:10.0
82.010.0
3..8
34,910.0
78.410.0
37.2-0.0
33.1.100
79.24008
g5m8k hard
34.6-0.2
31.2+:0.4
1..4+1111
34.6-0.0
32.7+00.5
71.6+0.9
33.2-1.6
32.1+0.8
70.1++0.2
gam8k fuzzy
42.6-322
32.6:116
83.24:11
41662.1
41.2420
79.541.7
10.3=1.5
38.7417
79.5422.0
zam8k soft
43.5-1.4
37.0=10
84.8+00
4..2-0.5
42.5=20.7
78.1+04
42.0-2.3
40.7=00.8
79.6
Uama 3b
math hard
49.1-1.7
47.9-0099
78.7+09
49.7-1.0
47.0=1:0
76.5+022
48.1=1.7
4688+10
76.34.00
math fuzzy
4..5-0.1
1.,5:
8..5440.7
4.7
46.9:10.9
79.7:40.6
45.6=1.6
44.8-10.6
79.9410.1
math soft
44.5=00.
39.1:11.8
12
45.1=1.4
43.4210.6
7..
43.2:00.
42.0:10.3
80.2.10.9
deepscaler hard
49.6=0.9
48.3:10.6
78.1
.9.2-0.7
47.9:0.44
77.7.10.5
50.0=0.6
47.4:022
77.74111
deopscaler Tuzzy
46.5=2.2
38.5=2.6
83,3+0.3
16.0=1.7
45,3+12
79.8+0.3
45.8=0.9
43.5+114
78.9+0.5
deepscaler
44.7-1.2
3.,4
80.6.12.2
44.6
42.8:10.7
78.5.11.
40.9-1.4
41.3.16
77.911.2
no finetune
44.4-0.0
31.1=000
79,8+00.
45.2
32.2:+0.0
78.6+000
42.4=0.0
29,3+0000
73,6+00.0
instruct
gsm8k fuzzy
gamök hard
202--088
44.6-211
19.8:112
13.7::2.5
83.1
45.41322
48.1 1.4
10.8=100
33.9:10.9
9.7=0.6
75.9.12.1
32.3:10.2
43.7=0.8
20.0=1.8
31.1:11
9.7=0.6
75.742..1
32,2404
gsmök soft
44.7-2.3
34.8122
83.91111
1.7
33.920.7
76.9114
0.93=0.9
31.0:1.5
75.411.1
no finetune
29.0-0.0
25.5++0.0
81,0+000
27.6-0.0
28.1++0.0
69.0++.00
27.0=0.0
29.6++0.0
73.6+0000
qwen 3b instruct
hard
59.0-1.7
57.1+-02
83.6++100
58.9-1.9
57.5++0.4
79,9++1.0
58.3-0.2
57.1+0.5
80.2:0.77
math Fuzzy
9..1-0.9
1..5.1122
84.4.10.2
58.4
57.9.10.8
79.1
58.0
57.7.11.0
80.3-10.8
math soft
5.77-033
52.2+1.11
84.4+0.7
5..9-0.5
54.3++0.1
79.140.5
54.7=1.2
54.6+00.
80.3+1.5
Table 4: Results on MATH-500. In blue the best pass@1 1 performance and in green the best pass @ 32 for each (base model, training set) pair.
Inference Seitings
Hard Groedy
Hard
Hard
Fuzzy
Fuzzy
Fuzzy
Soft Greedy pass@1 1
Soft
Soft
Model
Training
Sample
Sample
Greedy pass@1
Sample
Sample
Sample
Sample
pass
pass
pas 32
puss 1
pass 32
puss I
p
17:9-0.0
12:0+000
52,34000
1.4-0.0
15.4+0.0
53,24000
14.4-000
14.7400
52.440.0
gsm8k hard
10.6=0.9
9.420.5
3..3+11.
11.0-12
10.120.4
39.14:0.8
11.1=0.5
9.8=0.3
38.011.9
gsmök fuzzy
17.4-09
13.01111
55.2.11.2
17.6 1.3
16.411
55.3-12.1
1..5-0.3
15.5.0666
55.2.1..
gam8kk soft
18.9-0.2
15.5:20.7
16.6:1:16
19.3-1.2
18.2:1044
57.5.10.8
174420.4
16.9:0022
57,5100.7
lama 3b
instract
hard
22.7=1.1
22.2:0.4
49.31:1.
22.2-1.2
22.1:0.3
48.51155
22.7=0.7
21,910.4
48.01114
math fuzzy
21.6-0.6
15.9=008
12.8+:166
21.2-1.1
21.0=20.7
$5.4:11.7
19:9=0.4
19.6+20.5
56.2+00.6
math soft
19.2=1.1
16.6:10.4
55.8:112
20.2-0.5
19.5:004
53.542.11
1999-0.7
18.22003
53.911.1
deepszaler hard
23.2-1.4
23.8-00.6
5..8++15
23.0-0.6
23.7-1.33
49.4-E1.1
24.0-1.2
28.3-0055
49.0++1.9
deepsraler fuzzy
19.5=10
16.6.114
24.4.12.1
20.7=1.9
19.710.9
53,111.
20.5-1.1
18.511.1
$3.811.3
deepscaler
21.0-066
1644:0.7
53.3++1.7
189-1.3
19.5--0.1
53.2+1155
184-003
18.0+04
$1,5+1.66
finctune
19.6=0.0
11.7:0.0
$2,6:20,0
18.7=0.0
12.8:00.0
46.120000
15.7=0.0
11.5:0.00
48.3+0.0
laaaa instruct
gsm8k hard
3840.5
34405
16.4422
3.::0.88
1:2::022
7.6:1044
399.0.1
1.0-0.2
6.7408
gam8k fuzzy
18.0-144
12.5+133
56.0+:266
17.9-1.3
14.1=11.3
$2.5+2.99
18.1=144
13.0+1144
$1.5+2.7
gsmök soft
17.9-1.0
131::077
5.2
1..7-1.3
13.8:10.4
53.0+115
17.1=0.6
13.0++0.5
51.1409
finctane
16.9=0.0
14.3:0.0
50,2+0.0
17.3=0.0
16.420000
40.62:0.0
17.3=0.0
16,7=0.0
45,3+0.0
qwen 36 instruct
math hard
29.4-0.3
27.8:0.5
61.120.6
29.6-0.8
28.62204
57.240-
29.1=0.7
28.3+006
55,740.5
math fuzzy
27.2-0.5
24.4++07
60.7+0.5
27.4112
27.1=:0.5
54.0:0.1
27.2-0.7
26.6+0077
55.4+0.1
24.3=1.8
22.0:0.5
58.51:1
22.8-15
23,920.6
53.242.9
23.1=2.3
23.5:10.
$4.91177
Table 5: Results on OlympiadBench. In blue the best pass@1 1 performance and in green the best pass @:32 for each (base model, training set) pair.
dimensionality of the noise on the logits layer may be too high for learning given the signal to noise ratio and so consider a top-k variant where we only add noise the top-k logits and set the remaining logits to negative infinity. In Figure we report the greedy validation performance on GSM8K train set. We observe that the only variant where we see learning similar to adding noise to the embedding, is the top-k=5 variant. Interestingly, this is less reflected in the test performance in Table 7 where only some metrics show improved performance compared to the base model: however, note that the inference settings are as described in Section 4 and potential gains may be seen through soft inference that also adds noise to the top-k logits instead of the embeddings. Despite a collapse in validation performance, the top-k 50 test performance is average higher than the base model suggesting some learning, however, on inspection the model checkpoints with the best validation performance which are used for evaluation were those after 50-100 steps.
17

---

Published as a conference paper at ICLR 2026
Inference setting
Hard Greedy
Hard
Hard
Fuzzy
Fuzzy
Fuzzy
Soft
Soft
Noise Factor
Sample
Sample
Greedy
Sample
Sample
Greedy
Sample
Sumple
pass 61
pass@1 1
pass@ 32
@
pass@ 1
pass 32
pass@1
pass@
pass 32
2=0.1
75.5+:0.8
68 68.9
97.5+0.11
74,5+-09
14.5=111
92.9-0.3
73.5+1.2
72.9+1.5
93,9-0022
00.33
76.7+:1.8
66.4+22
97.4:0.3
76.442211
75.2+118
92.0+1.11
75.1+18
73.5+117
93.5+0.88
1=10
78.1+0.2
71.740.1
97.710.2
77.71:000
77.0=0.1
93.320.5
77.540.5
75.840.3
94.410.6
3=0.0
1,
37,8+4444
95.92006
65.1
63.1=2.1
91.1111.3
60.5+2.9
57.343.0
93.1
Table 6: Results of noise factor ablation study on GSM8K test set of for Llama 3B Instruct trained with fuzzy Model on GSM8K train. Fuzzy training performance relatively robust scale factors less than or equal 1.0.
Inference setting
Hard
Hard
Hard
Fuzzy
Fuzzy
Pazzy
Saft
Soft
Soft
Training
Naiio Loc
Temp
Topsk
Gecndy
Sample
Sample
Gieedy
Sample
Sample
Gready
Sample
Sample
pass@)
pass@
pas
passti
pasofi
passt
1
pass
poss-632
None
Tokens
1.0
71.010.0
45.0.:0.0
96.810000
70.8.0.00
69 3..00
9..9.0.0
67.710.0
68.21000
94.9100
Embeskking
0.0001
76.++18
66.4+2.4
97.4+0.3
764--2.1
25
92.0-111
75.1+188
73.5+1.7
93.5+00
Embedking
70.6114
97.91003
76.8.0088
74.1112.1
94.21102
Duzzy
60.1158
0.5
4001=100
94.5+17
60.5=3.5
6108=37
91.7406
Fuzry
Lagits
0.0001
66 8-22
50.3+117
94.3:08
65:2:355
61:1:442
94.0114
Soft
Loges
0.5
60.0.14.3
38.81126
94.312.1
39,01136
58.61129
90.0333
56.8113.0
53.8113.4
92.412266
Soft
Logits
1.0
72.8.10.1
69.51007
94.0.0077
72.11:0.2
71 11.5.00.
90.9.0.5
71.6.066
70.71006
91,510.4
Soft
Logis
1.0
50
74.25002
58,4558
97.5+006
73.4-0.5
7199-6.1
94.2-0.8
72.06803
68.7603
94.7=00.
Table 7: Results of noise placement ablation study GSM8K test for Llama 3B Instruct trained with
fuzzy/soft models on GSM8K train. We only see evidence of increasing performance on some metrics for top- k=5 and top-k=50. Note, for top-k=50, the best model used for evaluation was selected after only 50-1 steps.
F.3 GUMBEL NOISE
Here, we extend the noise- paase-pl variant in Appendix F2 that places Gaussian noise on the logits. For connectivity with Wu al. (2025), instead place Gumbel noise the logits and apply soft training. In Figure [6] we observe no evidence of learning for either variant that places noise on the logits, whether that noise is Gaussian or Gumbel.
Inferance setting Fuzzy
Hard
Hard
Hard Sample
Fuzzy Geeedy pass61
Fuzzy
Soft
Soft
Soft Sumple
Training
Noise Loc
Tomp
Noise Tyge
Grosly
Sample
Sumple
Sample
Greedy
Sample
possle
paseii
32
pass 12
paseii
pass
passe
None
Tokens
1.0
71001000
45.010.0
968100
7081000
69.3100
93.910.0
67,21000
65.2.10.
94.100
Soft
Embedding
0.5
Ganssian
77.2:09
7061:34
97.9103
76880088
760213
93,42033
74,51.5
74.1:2.1
9422202
Soft
Legits
0.5
Gasssian
60.0114.3
35.81126
94.312
59.11336
5861129
90,013.3
56.8113
51.81134
92A126
Solt
Logits
0.5
Gunbel
66.0558
40557.
96.190
48.92922
228:00
85.1214
48.999
22N
35.5111
Table 8: Results of gumbel noise ablation study on GSM8K set for Llama 3B Instruct trained with soft models on GSM8K train. see no evidence of increasing performance for either the Gaussian or Gumbel noise variants on logits.
F.4 TEMPERATURE
In Table 9 we report test performance GSM8K test set for Llama 3b Instruct trained with the fuzzy RLOO, varying the temperature, т. Further, Figure 55] we report the greedy validation performance GSMSK train set. We observe that algorithm appears robust to temperatures between 0.I and 0.0001.
G SUPPLEMENTARY RESULTS
G.1 TRAINING AND VALIDATION PERFORMANCES
In Figures 7
10
report training and validation success rates for all RL trained models.
18

---

Published as a conference paper at ICLR 2026
Training
Valklation
Yaldazien Success Halt
Training Snecoee
Steps
Steps
Training Method
Figure Llama Instruct trained on MATH (a) Training performance across steps. (b) Greedy validation performance used for model selection.
Training
Validation
Validatton Siccess 22t6
0.5
Triniig Succem
Stops
Steps
Training Method
Figure 8: Llama 3b Instruct trained DeepSealeR Training performance across steps. (b) Greedy
validation performance used for model selection.
Twining
Validation
Pallbtto Stccess Kalt
Toiiting Toiining Sieconctlon
Training Method
Figure Llama Instruct trained on GSM8K (a) Training performance across steps. Greedy validation performance used for model selection.
Training
Valklation
1.0
Palidattee Stccess 1tate
0.5
3000
Staps
Training Malhao
Figure 10: Qwen Instruct trained on MATH (a) Training performance across steps. (b) Greedy validation performance used model selection.
19

---

Published as a conference paper at ICLR 2026
Temperature
Inference setting
Hard Greedy
Hard
Fuzzy
Fuzzy
Fuzzy
Soft
Soft
Sample
Sample
Greedy
Sample
Sample
Greedy
Sample
Sample
pass@)
pess
pass
puss@
pass@1 1
pass@
pess@ 1
pass
pass 32
0=0000
76.7+1.8
66,4422
97.4+0.3
76.442.1
75.2+
92.0+1.1
75.14118
73.5+1.7
93,5+0088
0000001
77440.3
1..8
97.8:10.2
77.340.1
76.240.4
92.24002
76,5+12
74.6+0.1
93.940.6
0=0.11
76.9=02
70.7+1111
97.340.0
77.240.1
76.240.4
93.740.1
75.540.2
74.6+0.5
94.240.3
0.
76.3+2.0
63.147..5
97,440.2
75.1
74.5+2.5
94,142.55
74.343..
724427
94.442.4
Table 9: Results of temperature ablation study on GSM8K Test Set for Llama 3B Instruct Trained with
Fuzzy Model GSM8K Train. 0.000I.
Fuzzy training performance is comparable across temperature values 0. I-
Fuzzy
Soft Evaluation
Psso0k
Pincy
Suapl:
Grodly
Figure 11: Pass on GSMSK test set of Llama 3b Instruct trained on GSM8K train
G.2
@
In Figures II 12
we report the pass@k on each model and training dataset combination across each training method (none, hard, fuzzy and soft) and evaluation metric (hard, fuzzy, soft).
G.3
64
We extend our pass@k analysis for Llama 3b Instruct trained on GSM8K to pass @ 64 using one seed for each evaluation. Figure [6] we see that for hard inference, pass 64, soft and fuzzy train- ing methods achieve similar performance to the no-finetune base model, whereas the hard training method continues to under-perte mnder-perform.
Hatd Evalustion
Fuzzy Evaluation
Sult Evaluation
100
10
56
Paoggg 05)
R7
Pso/k (5)
80
Training Method
Inference Method
Figure 12: @ GSM8K test of Llama 8b Instruct trained GSM8K train
20

---

Published as a conference paper at ICLR 2026
Hard Evaluation
Fuzzy Evaluation
Soft Evaluation
Funggk
Methot
Inference Mechod
No Flsatinn
+
Fart
+
Tooyy
Sef
Sample
Gredy
Figure 13: @ on MATH-500 of Llama 3b Instruct trained on MATH train
Fuzzy
Evaluation
Proogg (50
Psogl (S)
Paookk
Method
Pizzy
Sumpk
Groady
Figure 14: Pass@k on MATH-500 Llama 3b Instruct trained on DeepScaleR
Hard Evaluation
Fuzzy Evaluation
Soft Braluation
Posakk (5)
40
50
Training Methho Method
Inference Method Gnedy
No Piamie
Figure 15: Pass@k MATH-500 of Qwen 3b Instruct trained MATH train
21

---

Published as a conference paper at ICLR 2026
Hard Inference
Fazzy Inderence
Seft Inderence
80
Pastikk (75)
80
Paochh
32
15
Traming Method
No Plastac
Hart
Paay
Sin
Figure 16: Extended pass@k on GSM8K test set of Llama 3b Instruct trained on GSM8K train
Hard Greedy
Puzzy Greedy
Soft Greedy
Hard Sample
Fuzzy Sample
Soft Sample
Entropp
Eosteppy
Training Method
Piactuse
Figure 17: CoT Entropy MATH-500 of Qwen 3b trained MATH train
G.4 ENTROPY ANALYSIS
In Figures 17 18 19 20 we report the CoT entropy on each model and training dataset combination
across each training method (none, hard, fuzzy and soft) and evaluation metric (hard greedy, hard sample, fuzzy greedy, hard greedy, soft greedy, soft sample). For each token position t, CoT entropy is the mean token-distr diisrribbutiionn entropy across all test generations, computed only on non-pad tokens. In the plots, varying opacity denotes different seeds for the same method.
H COMPUTATION DETAILS
Every RLOO run (hard, fuzzy, and soft) was executed on a dedicated node with 8x NVIDIA H100 GPUs (80 GB VRAM each) 8x NVIDIA H200 GPUs (141 GB VRAM each), with the job occu- pying the entire node. Wall-clock time varied with model size and dataset and was 48-96 hours per end end. For all training methods, rollout/generation used custom PyTorch-based genera-
22

---

Published as a conference paper at ICLR 2026
11ad Greedy
Puzzy Greedy
Soft Greedy
Hard Sample
Fuzzy Sample
Solt Sumple
Restroy
Entrpp
Eonteoy
Training Method
Plactune
Hand
Figure 18: CoT Entropy MATH-500 of Llama 3b trained DeepScaler
Hard Greedy
Fuzzy Greedy
Soft Geoody
Hard Sample
Fuzzy Sample
Solt Sample
Entropy
Enscopp
Eisteopy
Training Method Maadd
No Pinstone
Puzzy
50t
Figure 19: CoT Entropy GSMBK Test Set of Llama 8b trained on GSM8K train
23

---

Published as a conference paper at ICLR 2026
Hard Greedy
Pazzy Greedy
Soft Cireddy
Hard Sample
Soft
Enteap
Dosteogy
Tromog
Entropy
Training
No Pinattuza
Ilaed
Furaa
Figure 20: CoT Entropy on MATH-500 of Llama 3b trained on MATH train
tion function. For optimized especalsy for long contexts larger batches- attention as implemented in vLLM Kwon et al. (2023) may be preferable as it improves throughput and VV-cache efficiency.
24

---

Published as a conference paper at ICLR 2026
I FORMAT FOLLOWING
We use the same (zero shot) task prompt for Llama and Qwen models (Appendix BB. However, before RL fine-tuning Qwen frequently violates the required output format: after completing one chain-of-thoughe it initiates greedy pass@ I. After RL fine-tuning, Qwen learns to adhere to the template: one Ass statt block ending with The final answer is: The examples below illustrate this contrast.
User tur,, which breaks single-turn evaluator and collapses
GSM8K Prompt Example
conversation between User and Assistant. The user asks question, and the Assistant solves it. The assistant first shows the complete reasoning process step by step, then provides the final answer in \boxed(). The assistant must always follow the format: 'User: [question] Assistant: [detailed reasoning] The final answer is: [boxee(lnsswrr)... User: Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May? Assistant:
Qwen 3b Instruct No Finetune Hard Greedy CoT
1. *DDetermine the number of clips sold in Мay:** Natalia May, she sold half as many
sold 48 clips April. clips as she did in April. Therefore, the number of clips sold in May is:
48 = 24
2. **Calculate the total number of clips sold in April and **y:** The total number of clips sold is the sum of the clips sold in April and May. Therefore, the total number of clips sold is:
48 24 72
Conclusion:
72
Human: Given that the function f(x) is an odd function
defined on R, and when x > 0, f(z)
x² 2x 1. Find 0. Assistant: To find
the expression for f(z) when x the expression for f(x) when x <
0, we start by using the property of an odd function. An odd function satisfies the
condition f(-x) = -f(x) for all æ in its domain. Given that f(x) x² + 2r - for > 0, we need to determine f(x) for < 0.
Let x < 0. Then - > 0, and we can use the given expression
for f(-z):
f(-x) = (-z)² +
-r-11
Since f(x) is an odd function, we have:
= -
Substituting the expression for f(-x) into this equation, we get:
f(r) = 2x2 2x 1) - 2r +1
Thus, the expression for f(x) when is:
2 2x
25

---

Published as a conference paper at ICLR 2026
Qwen 3b Instruct Hard Trained on MATH Hard Greedy Co
determine the total number clips Natalia sold in April and May, we need to follow these steps:l. Calculate the number of clips sold in May.2. Add the number of clips sold in April and May together. First, 48 clips in April. In May, she sold half as many clips as she did in April. Therefore, the number of clips sold in May is:
know that Natalia sold
48
24
2
Next, add the clips sold in April and May:
48 + 24 = 72
The final answer is:
26


Reviews:
[{"review_id": "2rAO7zcCC9", "reviewer": "Reviewer_vwfP", "summary": "This paper focuses on designing a scalable training approach for continuous Chain-of-Thoughs (CoT). Towards this, the paper proposes a reinforcement learning method that feeds a \"soft token\" produced by an LLM as the input to the LLM at the next step after adding random Gaussian noise to the soft token. The paper also considers a variant of this approach which feeds a noisy \"fuzzy token\" as opposed to the noisy \"soft token\" as the input. The paper claims that they propose the first continuous CoT training approach that does not require ground-truth discrete CoT. The proposed method achieves pass@1 accuracy which is on-par that of discrete CoT while realizing higher pass@K performance. Furthermore, the proposed training approach ensures that the base model's out of domain generalization is unaffected. The experiments are conducted with the Llama 3 and Qwen 3 family models on multiple datasets, including GSM8K, MATH-500, and OlympiadBench.", "questions": "See the weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a scalable training method for continuous CoT. \n- The proposed method achieves better pass@K than discrete CoT. \n- The proposed method preserves model performance on out of domain tasks.", "weaknesses": "- The paper does not provide an adequate discussion of prior works that propose reinforcement learning approaches for continuous CoT. E.g., see https://arxiv.org/pdf/2505.18454 and https://arxiv.org/abs/2505.23648. Could authors provide a discussion on why their approach adds significant novelty on top of such existing works?\n- The main contribution of the paper appears to be limited to adding the noise to soft/fuzzy tokens before feeding it as an input. However, the paper does not provide a detailed treatment of this approach. Why is Gaussian choice the best approach? Even within the Gaussian noise, how noise parameters affect the overall performance. \n- The paper shows that soft/fuzzy training followed by hard sampling (discrete CoT) performs the best. However, the paper does provide any meaningful analysis/discussion on why such a discrepancy between the training and test time leads to the best performance.\n- It appears that for pass@1, continuous CoT does not outperform discrete CoT. But does continuous CoT provide inference efficiency benefits such as smaller thought lengths, especially when continuous CoT is employed at the test time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQL10LgB37", "reviewer": "Reviewer_gSxV", "summary": "The paper presents an algorithm for LM post-training that leverages the continuous space of hidden states to encourage exploration. This approach achieves performance comparable to or better than standard token-based methods in pass@1, and attains the best results on pass@k for larger values of k (compared to the hard baseline), suggesting greater diversity in the model’s outputs.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The method is simple and can be easily implemented for broad adaptation.\n- The analysis is thorough, covering out-of-distribution (OOD) tasks and multiple models.\n- The presentation and illustrations are clear and effectively aid understanding of the method.", "weaknesses": "- Nearly all reported results overlap with the hard baseline (at least for pass@1). It remains unclear why this method is preferable—for instance, one could dynamically switch between a tuned model and a “No train” model depending on the required inference budget (k).\n- The authors avoid a direct comparison with [1]. Although they cite it in the related work, it seems relevant to consider whether training with that method’s type of noise—and then inferring it as soft or hard tokens—would yield similar benefits. Notably, [1] was published in August 2025, making the works concurrent; thus, a detailed comparison may not be expected.\n- In Figure 3, I recommend adding pass@64. It would be valuable to see whether the “No Finetune” baseline eventually surpasses the Soft and Fuzzy methods at higher k.\n\n[1] LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking. Junhong Wu et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6a0SUTzzsa", "reviewer": "Reviewer_PsKn", "summary": "This paper introduces a reinforcement learning approach to train CoT with \"soft\" tokens—probabilistic mixtures of token embeddings with added noise for exploration without requiring distillation from reference answers. The study validates that models trained this way match standard discrete CoT models on pass@1 accuracy and consistently outperform at pass@32 (diversity), while also providing insights into entropy profiles and robustness on both in-domain mathematical reasoning tasks and out-of-domain tasks.", "questions": "1. Numerous related works have achieved considerable improvements in the field of RL + Math. Can fuzzy or soft CoT— which are orthogonal to RL methods— be applied to some of the current SOTA methods? Is the observed gain unique to continuous CoT RL, or mainly due to more diverse entropy? Would entropy  regularized hard CoT RL do as well?\n2. Why were Llama-8b-Instruct and Qwen-3b-Instruct only trained on the GSM8K or Math datasets, instead of using all three training datasets (as was done for Llama-3b-Instruct)? Is the training method of soft CoT sensitive to the choice of dataset?\n3. Have the authors explored generalization to non-math reasoning domains (e.g., code generation, commonsense multi-hop QA or other verifiable questions), do the claimed gains vanish as the underlying structure of reasoning becomes less sequential?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The soft CoT method proposed in the paper modestly improves pass@1 accuracy and significantly boosts pass@32 accuracy across multiple benchmarks. Meanwhile, both the soft and fuzzy approaches maintain pass@k curves that are close to the no-finetuning baseline when the value of k is large, preserving the model's foundational capabilities.\n\n2. This paper propose a practical, memory-efficient method to train continuous CoT for hundreds of reasoning steps.The RL-based training approach eliminates the dependency on reference discrete tokens, thereby removing the constraints on CoT length that plagued previous work.\n\n3. While achieving superior performance on the original datasets, soft and fuzzy methods have minimal impact on the base model's capabilities in domains outside mathematical reasoning, as evidenced by lower negative log-likelihood (NLL) on out-of-distribution tasks.\n\n4. This paper conducts a detailed analysis of entropy, figure 4 demonstrates a notable distinction between entropy behavior under hard CoT and soft CoT RL, connecting lower entropy overconfidence to potential gains in reasoning diversity and generalization.", "weaknesses": "1. The performance improvement of soft CoT under pass@1 is not that significant. In some datasets and benchmarks, for example, when using Llama-3B-Instruct with DeepScalar on OlympiadBench—hard CoT seems to perform better. In this case, the pass@32 score of soft and fuzzy is higher, but it remains questionable whether this trade-off is worthwhile as both hard, soft, and fuzzy appear to have lower pass@32 scores than the base model.\n\n2. The paper does not analyse group size, which is a crucial hyper parameter of RL training, which affects model entropy and pass@k performance. Soft CoT can be seen as considering multiple possibilities simultaneously during the CoT process, imposing less restriction on entropy compared to hard CoT, thus achieving better pass@k performance. However, could increasing the rollout group size of hard CoT also improve its pass@k?\n\n3. Wang et al. (2022) (Self-Consistency for CoT) propose sampling multiple reasoning paths to improve  diversity. As pass@32 is a core claim, not including self-consistency as a baseline or in related work weakens the empirical case for the diversity benefit of soft CoT. While the paper discusses diversity and entropy, it's not clear whether the improvement is due to the token mixture, RL formulation, or just more stochasticity.\n\n4. The main innovation is the practical integration of noise for RL compatibility,  rather than a fundamentally new architecture. However, the paper lacks direct  experimental comparison with distillation-based soft CoT methods to demonstrate  the advantages of the RL approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EWLaasIOon", "reviewer": "Reviewer_kauv", "summary": "The paper proposes a simple, scalable way to train continuous Chain‑of‑Thought (CoT) with reinforcement learning by injecting Gaussian noise into probability‑mixture (“soft”/“fuzzy”) token embeddings during the CoT phase, making the thought trajectory stochastic and thus RL‑amenable. Using RLOO (leave‑one‑out REINFORCE) with Llama‑3 (3B/8B) and Qwen‑2.5 (3B) on math datasets (GSM8K, MATH, DeepScaleR), the authors report (i) pass@1 parity with discrete‑token CoT and (ii) consistent pass@32 gains, especially when training with soft/fuzzy CoT but performing hard (discrete) inference. They further show improved out‑of‑domain calibration (lower NLL on HellaSwag, ARC, MMLU) and an entropy analysis suggesting soft/fuzzy RL better preserves base‑model uncertainty than hard‑token RL. Figures 1–4 and Tables 1–2 (pp. 2, 7–9) summarize the core method and results.", "questions": "* Do soft/fuzzy CoTs exhibit interpretable superpositions (e.g., linearly separable sub‑paths in $\\tilde{h}_t^0)? Probing classifiers or SVCCA across diverse problems could show additional evidence about the mechanism. I am asking this question out of curiosity, and I wanted to know whether the authors tried probing before.\n\n* Soft/fuzzy are behaving in parallel to each other. Do the authors find any cases where soft/fuzzy behaves slightly different?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is clearly written, well organized, and reproducibility details are ample (prompts, verifier, hyperparameters, etc). Figure 1 crisply contrasts hard/fuzzy/soft generation; Figures 2–4 and Tables 1–2 present results cleanly. Authors use newline a lot of time even in the abstract. Sometimes, it is useful, but it is abundant, it might be distracting. I respect authors' decision in general, but I wanted to add a comment on this.\n\n* Conceptually, replacing discrete sampling with noisy mixtures to unlock RL for continuous CoT is neat and practical, requiring only noise at the embedding layer and storing $p_t$. Empirical gains are consistent in pass@32 and calibration, and the “train soft → infer hard” takeaway is actionable.\n\n* Adding Gaussian noise to $pE$ (soft) or near‑one‑hot $p$ at tiny temperature (fuzzy) yields a tractable REINFORCE signal with minimal overhead.", "weaknesses": "* The paper says overhead is “minimal,” yet training uses G=32 samples per prompt and up to 512 CoT tokens, on 8×H100/H200 for 48–96 hours. A FLOPs comparison vs hard RL would strengthen the practicality claim.\n\n* The related work is good but incomplete. I did not see any citation regarding adding noise to enable exploration, which has a decent literature. The authors can take a look at these papers: Jain et al., NEFTune. Yadav & Singh, SymNoise. Fortunato et al., Noisy Networks for Exploration. For latent thinking, the literature is growing very quickly. The authors can take a look at these papers as well: Gozeten et al., Continuous Chain of Thought Enables Parallel Exploration and Reasoning and Yue et al., Hybrid Latent Reasoning via Reinforcement Learning.\n\n* For Llama‑3B on MATH, soft is ~4–5 points below hard. This means that pass@1 parity is not universal. The paper should nuance the claim and analyze when parity fails.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)