LaDiR: Latent Diffusion Enhances LLMs for Text Reasoning
Authors: Haoqiang Kang
Yizhe Zhang
Nikki Lijing Kuang
Nicklas Majamaki
Navdeep Jaitly
Yian Ma
Lianhui Qin
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Reasoning
Diffusion Models
Latent Reasoning
Primary area: foundation or frontier models, including LLMs
TL;DR: LaDiR is a novel latent reasoning framework that encodes latent “thought tokens” with a VAE and predicts them via latent diffusion models, enabling adaptive test-time compute, parallel diverse generation, and better intepretability.

Abstract: Large Language Models (LLMs) demonstrate their reasoning ability through chain-of-thought (CoT) generation. However, LLM's  autoregressive decoding may limit the ability to revisit and refine earlier tokens in a holistic manner, which can also lead to inefficient exploration for diverse solutions. In this paper, we propose LaDiR (Lalent Diffusion Reasoner), a novel reasoning framework that unifies the expressiveness of continuous latent representation with the iterative refinement capabilities of latent diffusion models for an existing LLM.  We first construct a structured latent reasoning space using a Variational Autoencoder (VAE) that encodes text reasoning steps into blocks of thought tokens, preserving semantic information and interpretability while offering compact but expressive representations. Subsequently, we utilize a latent diffusion model that learns to denoise a block of latent thought tokens with a blockwise bidirectional attention mask, enabling longer horizon and iterative refinement with adaptive test-time compute. This design, combined with explicit diversity guidance during diffusion inference, enables the generation of multiple diverse reasoning trajectories that explore distinct regions of the latent space, rather than producing repetitive solutions as often occurs in standard autoregressive sampling. We conduct evaluations on a suite of mathematical reasoning and planning benchmarks. Empirical results show that LaDiR consistently improves accuracy, diversity, and interpretability over existing autoregressive, diffusion-based, and latent reasoning methods, revealing a new paradigm for text reasoning with latent diffusion.

Published as a conference paper at ICLR 2026
LADIR: LATENT DIFFUSION ENHANCES LLMs FOR TEXT REASONING
Haoqiang Kang'
Yizhe Zhang
Nikki Lijing Kuang'
Nicklas Majamaki
Navdeep Jaitly²
Yi-An Ma
Lianhui Qin-
Univeersity of California, San Diego
Apple
ABSTRACT
Large Language Models (LLMs) demonstrate their reasoning ability through chain-of-thoon thought (CoT) generation. However, LLM's autoregressive decoding may limit the ability to revisit and refine earlier tokens a holistic manner, which also lead to inefficient exploration for diverse solutions. In this paper, we propose LaDiR (Latent Diffusion Reasoner), a novel reasoning framework that unilies the expressiveness of continuous latent representation with the iterative re- finement capabilities of latent diffusion models for an existing LLM. We first con- struct a structured latent reasoning space using Variational Autoencoder (VAE) that encodes text reasoning steps into blocks of thought tokens, preserving seman- tic information and interpretability while offering compact but expressive repre- sentations. Subsequently, we utilize latent diffusion model that learns to denoise a block of latent thought tokens with a blockwise bidirectional attention mask, enabling longer horizon and iterative refinement with adaptive test-time compute. This design, combined with explicit diversity guidance during diffusion inference, enables the generation of multiple diverse reasoning trajectories that explore dis- tinct regions of the latent space, rather than producing repetitive solutions as often occurs in standard autoregressive sampling. We conduct evaluations a suite of mathematical reasoning and planning benchmarks. Empirical results show that LaDiR consistently improves accuracy, diversity, and interpretability over exist- ing autoregressive. ieffusion-based and latent reasoning methods, revealing a new paradigm for text reasoning with latent diffusion.
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable reasoning abilities through extensive pretraining human languages, yet the inherent limitations of the autoregressive (AR) paradigm are becoming increasingly difficult to overlook (Zhou et al.) 2024b Bachmann & Nagarajan 2025). As shown in Fig. [ (top left), their sequential nature prevents revising earlier tokens, making self- refinement inefficient and difficult (Chen et al. 22024] Huang et al.] (2024). Moreover, AR models
with discrete CoT generate linear chain of thought (CoT) (Dziri et al. 2023 Wei et al. (023).
which limits reasoning diversity and restricts exploration of multiple valid solutions (Naik et al.)
2024
Yu et 2024).
Diffusion models (Ho al. 2020), originally introduced for generation in continuous domains like images, have recently gained attention in text generation for their ability to maintain global coher- ence and enable iterative refinement Ye et al. 2024b Nie al. 2025 Lou al. 2023) Yu al. 2025b] Weligalle 2025 Sahoo et al.] 2024 Gulrajani & Hashimoto 2023). Moreover, prior works
explored continuous or latent diffusion for language generation
al. 2022: Lovelace al.,
2024 ZZhnn et al. 2023 Lovelace et al. 2026 Cetin et al.) 2025). operating diffusion in latent
spaces obtained from text autoencoders or loken-embedding spaces. Existing works largely empha- size the parallelization properties of diffusion models (Israel al. al.] 2025 Nie et al. [2025] Weligalle
2025) or evaluate fluency in text generation (Li al. [022] Lovelace et al. (224:]22aaa et al. 2023]
Lovelace al. 2026). Arguably, more important direction to ask: How can these approaches enhance the reasoning capabilities of LLMs? We focus on one particularly promising capability: the ability to self-correct and refine reasoning chains at semantic levels in latent space. As shown

---

Published as a conference paper at ICLR 2026
in Fig. [ (top right), such self-refinement cannot be achieved by discrete diffusion language models that merely transit into masked tokens.
Diffusion LMS (Blockwise)
AR CoT
Block
[nower
IMAGK (MSK]
[MASK]
(MASKK
AR Latent CoT (e.g. Coconut)
[MASK]
[1]
[Arowee]
Answer
LaDiR: Latent Diffusion for Reasoning
Tokan
[Answer]
Continuaus Takke of LLM
Denolsing
Latant Though: from
Gaussian Random Noize
Figure 1: Comparison of reasoning paradigms: autoregressive CoT and latent CoT generate discrete continuous tokens sequentially; diffusion LMs iteratively convert [MASK] tokens to discrete text tokens in parallel in a semi-autoreee cemi--uvoregressivve way; and our proposed method LaDiR reasons over latent thought tokens via diffusion, enabling iterative refinement at semantic level and diverse exploration.
To address this limitation, introduce LaDiR (Latent Diffusion Reasoner), flexible reasoning framework that encodes high-level semantic representations of reasoning steps into continuous latent tokens via Variational Autoencoder (VAE) as latent thought tokens, and trains a latent diffusion model over them to perform reasoning. This bridges the gap between surface-level token refinement and deeper semantic reasoning. After the reasoning process, the model generates final answer tokens conditioned on the generated latent thought tokens. Unlike prior latent diffusion works for text generation (Li et al. 2022, Lovelace et al. 2024 Zhang et al., 2023), which focus on fluent text generation, our framework is explicitly designed for latent reasoning: it learns causal dependencies across reasoning steps through blockwise diffusion, propagates answer correctness signals back to latent tokens.
Our proposed paradigm establishes new reasoning framework as post-training method, bring- ing several distinctive advantages. First, the iterative refinement ability of diffusion enables better trade-off between accuracy and test-time compute, as additional denoising steps can be flexibly allo- cated to improve performance. Second, our framework introduces that applies repulsive forces during diffusion inference, pushing latent trajectories apart within batch to explore multiple diverse reasoning paths, where as AR models tend to collapse to similar trajectories. Finally, leveraging VAE-based latent space enhances interpretability over continuous diffusion models, making the reasoning process more transparent and readable.
irversity-guidance mechanism
Experimentally, we demonstrate that diffusion-based latent reasoning is not only more accurate but also qualitatively different from prior approaches. On math reasoning benchmarks, including GSM8K (Cobbe et al. (02T) and MATH (Hendrycks et al.) 2021), where Coconut (Hao et al.) 2024) fails to surpass AR CoT supervised finetuning (SFT), LaDiR consistently outperforms it aver- age across 7 benchmarks with the LLaMA 3.1 8B model (Dubey et al.) (024). This suggests that modeling reasoning the semantic level, rather than the token level, may lead to more faithful intermediate steps that accumulate into stronger final answers. Moreover, on the Countdown plan- ning task, LaDiR shows over 30% absolute improvement both Pass @ and Pass@ 100, indicating that latent thought tokens potentially enhance global planning ability, while parallel diversity explo- ration enables the model to generate diverse reasoning paths. Together, these findings suggest that diffusion- based latent reasoning provides principled way to balance accuracy and diierritty-ker ingredients for advancing beyond sequential autoregressive reasoning.
PRELIMINARIES
This section introduces key concepts and notations in VAE and latent diffusion models (Rombach et al. 2022). Detailed formulations and background information about VAE and diffusion models are provided in Appendix B
2

---

Published as a conference paper at ICLR 2026
Attention Mask
VAE
Ippf Casation
Decoder (LLM)
Tinestop
Latent Block
sample n Mu. o ross risustees
<EOT> Takan
Tinestop
Encoder (LLM)
Latert Block
-BOT> Taken
Tokens
Embeddings
Reasoning Model (Latent Block Diffusion + AR)
Block
Latent Block
AR Text Generation
4,
4'
Large Language Model
Figure 2: Illustration of our block-wise latent reasoning framework. A question Q is first input condition generate latent blocks, each delimited by <BOT> and <EOT>, For each block, the model iteratively denoises latent tokens (()) across timesteps, with bidirectional attention inside block and causal attention across blocks. The reasoning process terminates when the model emits the <SOA> token, after which the model generates the answer text autoregressively.
2.1 VARIATIONAL AUTOENCODER
A Variational Autoencoder (VAE) (Kingma & Welling 2013) learns latent representation of data by balancing reconstruction accuracy and prior regularization. Let æ E IR RLxd, denote a sequence of token embeddings with length L and embedding dimension dz. and E R Rxdds denote latent repre-
sentations with M latent tokens of dimension de. We adopt the 3- V-V Higgins et al. (017). where scaling factor A controls this trade-off:: L.vVVEE (x
log po
To [p(=))
Here, (=(x)) is the encoder distribution parameterized by d, (=)))2) the decoder likelihood parameterized by 0, and p(2) = -O T) is the prior distribution over latents.
Larger 3 values encourage disentangled and structured latent spaces, at the cost of reconstruction fidelity. During inference, the encoder of VAE produces a mean/variang pair {(4, J)}. }, and a latent token is sampled
+ @ C,
N/ I)-
2.2 LATENT DIFFUSION AND FLOW MATCHING
Latent diffusion models (Ho et al. 2020 Rombach al. 2022) generate data by denoising latent variables from Gaussian noise the latent space of a VAE, which preserves high-level semantic structure. Diffusion also be viewed
continuous time generative flow trained via flow match- ing (Lipman et al. 2022), which we adopt as our primary framework for its superior performance. The training and inference processes
follows:
Training Let [zeJejo..j denote path interpolating between clean data 20 Pdata and noise t)20 te. This path is controlled by ordinary differential equation (ODE) u² (=), dz 20, where u* is the target velocity field. A neural network ug(2,, t) is trained to approximate u* by minimizing the flow matching loss:
I). 2
LrM = E,-2((0 1), Por-Pan (0.1) [110(E),)) u² u (21,
(1)
Inference. At generation time, the process begins from Gaussian noise = ~N /). The learned velocity field uz !) is then integrated backward in time using an ODE solver as follows: žt-At = 2, At uo (2,, t), with steps from latent representation. This procedure naturally supports iterative refinement, as each integration step progressively transforms noise into a coherent latent z.
to t 0. The final state 2g corresponds
clean

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
Our approach separates reasoning from answering. A variational autoencoder (VAE) constructs a latent space of intermediate reasoning steps, encoding each step block of thought tokens. We further utilize a reasoning model that predicts and refines thought tokens via latent diffusion, and then generates the final answer tokens conditioned on the denoised latent tokens.
3.1 ARCHITECTURE
We employ a VAE to construct the latent space of intermediate reasoning steps, and a reasoning model that predicts latent tokens via diffusion and generates the final text answer.
Blockization. We separate the chain-of-thought (CoT) reasoning and the final answer in the dataset using the prefix "The answer is' The text preceding the prefix is treated as CoT e, while the text following is treated as the final answer y. We then split into individual sentences,
each treated as a block of latent tokens with block size Tap: ( = [=1 - (b)
2Lo
J:
b= 1,.., N.
This one-sentence- -eeeteence-er-block design ensures that each reasoning step is localized in latent space, diffusion is performed at the block level as block diffusion Arriola et al. (2025).
VAE architecture. As shown in Figure 2 (left), our VAE encoder is initialized from pretrained LLM and fine-tuned with all parameters, along with Ls learnable embeddings. The encoder's last hidden state is passed through two linear projections to obtain the mean μe and variance o², from which we sample (b) ~ (²) The decoder is frozen pretrained LLM that conditions on the sampled Z to reconstruct the corresponding block of text. This design enables the encoder to compress each reasoning step into a structured latent representation aligned with the semantic space of the language model.
Reasoning model architecture. We utilize an existing LLM as reasoning model. As illus- trated in Fig. 2 consider the prediction of the second latent block. After the input question Q, we insert special token <ROT> to mark the start of block, followed by the first block tokens Z and a token <EOT> to mark its end. For the second block, since it is being predicted, we add timestep embedding between <BOT> and 2] encode the timestep information. Once the latent reasoning process is complete, we switch to text generation mode by appending a <SOA> token to indicate the start of the answer, which is then generated autoregressively. To balance lookahead and variable-length generation, we adopt hybrid attention mask M (Fig- 2 top right). Within each block, tokens attend bidirectionally, enabling the model to internally reason over a horizon defined by the block size and capture richer local dependencies. Across blocks, attention is strictly causal, so later steps depend on earlier ones in an autoregressive manner.
3.2 TRAINING
We train the two components separately: the VAE is first trained to learn latent representations of thought tokens, after which the reasoning model trained to predict these thought tokens. We describe each stage in turn, beginning with VAE training and followed by reasoning model training.
3.2.1 VAE TRAINING
We build on the standard B-VAE training and inference framework described in Section 2.1 with the following adaptations tailored to our task.
Robustness augmentations. To improve generalization and make the latent space resilient to noise and input variability, we introduce two augmentation strategies during training:
(b) Latent Gaussian noise. For each latent token zi we inject isotropic Gaussian perturbations:
(b) 2 Mhi N. ~ No0, k2 I), where find k achieve the best downstream perfor- mance. This enhances robustness by smoothing the latent space and mitigating sensitivity to small semantic variations.
Input token substitution. For the encoder input sequence, with probability p 0.3 we replace token with another randomly chosen token (sampled uniformly from the LLM vocabulary). This forces the encoder to learn invariances to paraphrasing, typos, minor corruptions in input text, ensuring that latent representations capture semantic content rather than exact lexical form.
Together, these augmentations encourage the VAE to build smoother latent space that is both robust to perturbations and expressive enough to encode thought-level reasoning steps. A more detailed diagram of the VAE can be seen in Appendix
4

---

Published as a conference paper at ICLR 2026
3.3 REASONING MODEL TRAINING
After constructing latent reasoning space with the VAE, we train a latent diffusion model fy from the same pretrained LLM as our VAE model to denoise latent blocks, gradually transforming noisy latent representations to coherent reasoning blocks. Empirically, we observe that training with the flow-matching objective yields the best performance (see Appendix B.4), and therefore adopt as our default training objective in the paper.
Answer Token Loss. While f learns to predict latent reasoning trajectories, to avoid explicitly decoding these steps through the VAE decoder for efficiency during inference, we use the same transformer backbone & with LM head to autoregressively predict answer text tokens conditioned on the latent reasoning blocks. To this end, given the question q, the reasoning blocks B(5B), and the past answer tokens y<u. the model predicts the next answer token you with distribution Pv 9. Z(S)), "yc)), The training objective for those answer tokens the cross-entropy loss:
Cus
Σ log (w q, (SBB) V-w))
Special Token Loss. To explicitly control the number of latent blocks, introduce special binary classification head on top of the same LLM transformer backbone v. It predicts whether the next block begins with <SOA> (start-of-answer) or <BOT> (begin-of-thought) token when- ever an <BOT> cand-of-thought token is generated. Formally, let index positions of <EOT> tokens in the output. For each т, the model produces a distribution (* q. ((5B) UUT)), 87 E (<SOA>, <BOT>}, and we minimize the corresponding classification loss, which supervises the model to predict special tokens given the question and latent reasoning blocks up to position T:
Lspec =
E TETo
log Po 1 ((SB).
(3)
3.3.1 STAGE 1: TEACHER-FORCING TRAINING
In the first stage, the model is trained under a teacher-forcing regime, where it has access to oracle latent blocks produced by the VAE encoder, denoted as (1:B), At every step, these oracle latents are concatenated between special tokens <BOT> and <ECT> and provided as context to the flow- matching model fp- The overall training objective jointly optimizes flow matching on latent blocks and cross-entropy supervision on both final answers and special tokens:
L = AFM LFM + AAns LAns + Aspee LSpect
(4)
where LFM is defined in Eq.
3.3.2 STAGE 2: ROLLOUT TRAINING
After Stage 1, there is a mismatch between training and inference. During inference, the model must be conditioned on previous self-generated latents without access to oracle latents, suffering from eтror accumulation issue. To address this issue, Stage 2 adopts an rollout training. We keep the same number of blocks B as in the ground truth, but instead of conditioning on oracle latents, the model generates its own latents Z (1:B) from random noise using fewer denoising steps (i.e., 50 + 10, following FlowGRPO (Liu et al. 2025)). We keep the gradients on (1:B) during denois- ing, allowing answer supervision backpropagate through the trajectory and directly shape latent predictions. To avoid latent collapse as in Coconut w/o curriculum learning (Hao et al., 2024). we keep the flow matching loss. Therefore, the training objective same as Eq.
3.4 REASONING MODEL INFERENCE
At inference time, the model generates a chain of latent reasoning blocks and subsequently produces the final answer in text space. The process unfolds in two phases: (i) latent block generation via iterative denoising, and (ii) answer generation via autoregressive decoding.
Iterative denoising. Following the inference process of the standard latent diffusion model as in 2.2 for each block, we initialize with Gaussian noise, and we gradually transforms the noise into a semantically coherent latent reasoning block (b).
Stopping criterion. Latent block generation continues until the model explicitly predicts the spe- cial token <SOA> (start of answer). This token signals that sufficient reasoning has been performed and the model should transition from block diffusion final answer generation.
5

---

Published as a conference paper at ICLR 2026
Answer generation. Once reasoning terminates, conditioned on the generated latent reasoning sequence (1::B) and the input question 2; the model predicts output tokens y ( yr) au- toregressively.
Diversity improvement in parallel. Unlike AR models that generate a single reasoning trajectory sequentially, framework generate multiple diverse reasoning trajectories in parallel within batch. To encourage exploration of alternative solutions, we incorporate two complementary mech- anisms:
1. Increased initial noise. By sampling with an increased variance o² as initial noise at the first denoising step, we broaden the distribution of starting points for latent trajectories.
2. Diversity gradient guidance. At each denoising step, we enhance diversity by adding a repul- sion term to push the latent tokens in a batch apart. First, we compute bandwidth parame- ter the median pairwise distance between the latent tokens in batch the current step a mediantei ||z 2j112. The repulsion force field for a latent token 2; is then defined as
F(2)) = -E 2 2362
2(1
22 2,
exp
||
(2+ - 22), S B. (5)
where is any other latent token in the same batch with batch size B. We apply strong repulsion the beginning of inference and gradually decay its effect over time. Specifically, the time- dependent scale is defined as = Tmax (f). where I' is the total number of inference steps, t decreases from T to 7max is the initial repulsion strength as a hyperparameter.
Finally, the diversity-guidee gradient, a form analogous to classifier-free guidance (Ho & Salimans 2022); 2 (x t, x) + Ye Fz)), where (x, t, is the model's prediction at step
prediction combines the base model output with the repulsion
4 EXPERIMENTS
We evaluate LaDiR across three domains: mathematical reasoning (7 benchmarks), code generation (4 benchmarks), and puzzle planning (Countdown), comparing AR, latent, and diffusion base- lines. Our experiments demonstrate its effectiveness on benchmark datasets, while the analyses in Section 4.5 provide further insights of our method. See Appendix D for experimental details.
4.1 MATHEMATICAL REASONING
We begin by assessing LaDiR on range of mathematical reasoning benchmarks, covering both in-domain datasets, where training and test distributions are closely aligned, and out-of-domain benchmarks that require generalization to unseen problems.
Datasets We fine-tune pretrained LLMSs the DART-MATH dataset (Tong et al. 2024b),
large-scale dataset synthesized to enhance mathematical reasoning. For evaluation, we adopt two in- domain benchmarks, Math (Hendrycks et 2021) and GSM8K (Cobbe et al. [2221) and five out- of-domain benchmarks to assess generalization: College-Math (Tang et al.] 2024b). Deep-innd- Math (aaxton al., 2019). OlympiaBench-Math (He et al. 2024). TheoremQA (Chen et al.,
2023 and Trresh-Gaokao-Mtth2222 vided Appendix
(Tang et al. 2024b). Detailed dataset descriptions are pro-
Baselines We benchmark LaDiR against comprehensive set of baselines categorized into three distinct groups.
gustusct.rouus.. First, for Masked Diffusion Methods, we utilize the LLaDA 8B backbone et al. 2025), evaluating both the base model and version fine-tuned (SFT) the same CoT dataset. Sec- ond, we compare against Autoregressive Methods based on LLaMA 3.1 8B Dubey et al.] (2024). This category contains traditional supervised baselines--peccffccall Solution-Oni (No CoT) SFT and CoT SFT with different decoding temperature for randomness as well as recent AR latent reasoning approaches, including Pause Token Goyal et al. (2023). iCoT Deng et al. (2023). Co- conut Hao et al. 2024), CODI Shen et al. (2025a). Discrete Latent Su et al. (2025), Soft Token Butt al. 2025), Soft Think ZZang et al. (2025). ) and TaH+ Fu al. (2025). Finally, we compare against prior Latent Diffusion Methods, specifically LD4LG Lovelace et al. (2024) and PLANNER /Zhang
al.
(023). Detailed baseline descriptions are provided Appendix D.I
Results As shown in Table T LaDiR achieves the strongest overall performance, surpassing cent competitive latent reasoning baselines including CODI. Soft Thinking, and TaH+. On average, it improves Pass @ accuracy by 1.5% over the best prior approach (TaH+). Crucially, LaDiR

---

Published as a conference paper at ICLR 2026
In-Domain
Out-of-Domain
Method
Avg.
MATH
GSM8K
Gaokao
DM-Math College
Olympia TheoremQA
Masked Diffusion Methods - -LLADA 8B
Base Model
36.2/48.6 39.0/51.0 82. 20.1 36.0 43.7/50.7 38.9/ 48.3 5.9/10.2
77.3/82.9 10.2/ 18.5 29.8/36.2 30.2/49.1 3.0/7.7
16.6/21.6 20.9/26.3
29.0/37.8
CoT SFT
35.8/44.3
Autoregressive Methods - LLAMA 3.1 8B
Sol-Only SFT
13.3/18.9
16.4/20.7 0.0/2.8
18.2/23.3 20.9 1.7/
16.9/22.2
12.2/-
SFT (a 0.1)
43.8/46.8
84.9/86.2
31.2/34.7
48.4/49.3 46.5/510 10.5/11.1
22.0/22.8
41.3//2.66
SFT (a 0.7)
42.9/49.4
84.2/88.6
30.4/37.5
47.5/51.6
45.3/54.3
9.9
20.9/24.6
40.1/45.4
SFT (a 1)
42.1/51.0
83.5/90.1
29.7/39.0
46.6/53.1
44.6/56.0
9.4/13.5
20.3/26.0
39.3/47.1
SFT (a = 1.2)
41.2/52.5
82.6/91.8 28.9/40.6 45.5/54.6
43.6/57.8
8.9/14,6
19,5/27.5
38.4/48.8
Pause Token
27.2/31.4
62,4/67.2 21.3/25.7
26.7//0.8
27.2/32.1
3,5/7.8
8.3/12,9
25.2/29.7
iCoT
35.2/37.9
61.8/63.9
30.0/32.9 30.6/33.0
32.6/34.8
4.3/7.1
11.5/14.0
29,4/31.9
Coconut
37.3/39.3
68.3/74.3
26.8/29.3
33.5/36.9
40.2/42.9
5.8/6.3
11.4/4.9
31,9/34.8
CODI
38.5/45.1
76.3/81.7
27.5/35.2
38.8/449
43.2/49.0
7.6/14.8
8.8/1577
34.3/40.9
Discrete Latent
43.2/47.3
83.9/88.6
33.3/39.7 -1-
47.1/53.7 44.7/49.5
13.3/17.8
20.3/28.5 40.8/46.4
Soft Token"
44.7/-
83,9/-
-/-
-/-
-/-
-/-
-/-
Soft Think
44.3/46.7
83.7/86.6
31.2/334 49.1/51.7 45.9/48.0 30.9/34.0 48.2/51.8
10.4/13.1
22.2/24.7
41,0/43,5
TaH+
46.1/49.4
85.9/89.7
46.7/50.1 12.2/15.2
24.5/28.4
42.0/45.5
Latent Diffusion Methods
LLAMA 3.1 8B
LD4LG
9.1/15.3
32.9/38.4 13.2/ 20.1 24.5/29.6 17.9/24.3
2.8/8.6
9.6/15.7
15.7/21.7
PLANNER
5.7/13.1
18.7/24.6
15.1/21.3 19.9/27.6
23.6/29.1
2.2/9.0
10.3/17.3
13.6/20.3
LaDiR
46.2/63.7
84,8/93.7
35.4/ 45.8
52.3/54.2
48.6/ 60.3 12.9/15.3
24.7/30.7
43.5/52.0
-w/o Stage 2
30.7/35.8
62.8/62.6 24.7/ / 30.2 32.0/36.9
32.8/ 38.0 5.9/
11.9/16.9
27.9/33.0
Table 1: Math reasoning results across in-domain and out-of-domain benchmarks. Each cell reports Pass@1 (left)/ / Pass@ @ 100 (right) accuracy. a is the decoding temperature of LLMs. "Results taken from the original paper; missing Pass@ 100 results (denoted as "-") indicate that the codebase not open-sourcee for reproduction.
demonstrates better test-time scalability and diversity, achieving the highest Pass@ 100 across all benchmarks with 6.1% absolute gain over AR CoT SFT. Compared to text-based CoT and soft- token methods, our latent reasoning yields more robust solutions on complex benchmarks like DM- Math and College, suggesting that diffusion objective captures abstract reasoning patterns more effectively than previous curriculum- based (e.g.. Coconut) or VQ-VAE (e.g., Discrete Latent) ap- proaches. The performance improvement from Stage 2 rollout training further underscores its impor- lance in mitigating етгоr accumulation, contrasting with prior latent diffusion methods like LD4LG and PLANNER which struggle on reasoning tasks. Overall, LaDiR combines the interpretability of CoT with the expressiveness of continuous diffusion, producing generalizable reasoning traces.
4.2 CODE GENERATION
We extend our evaluation to the domain of code generation. In this section, we assess the capability of LaDiR to generate executable code by comparing it against a wide range of diffusion- and autoregressive coding models and training methods.
Baselines We categorize our baselines into four distinct groups to provide comprehensive eval- uation: (1) Auto-regressive (AR) Coding Models, including the state-of-the-art Qwen 2.5 Coder 7B Hui et al. (2024) and OpenCoder Huang et al. (2025); (2) Masked Diffusion Models, such as LLaDA, Dream, and the specialist Diffu-Coder; and (3) Looped Latent Reasoning Models, rep- resented by Ouro 2.6B Zhu et al.] (2025b). Additionally, to rigorously isolate the efficacy of training methodology, we compare against (4) Reasoning Methods built upon the same Qwen3- 8B-Base Yang al. (2025) backbone. This control group includes a standard AR SFT baseline, the training-free latent reasoning approach Soft Thinking Zhang et al. (2025), and TaH+ Fu et al. 2025). strong supervised latent reasoning baseline.
Datasets We construct SFT dataset by filtering 73k Python samples from the LingCode
dataset Codefuse & Team (2025) For evaluation, we employ four widely adopted bench-
marks: MBPP Austin et al.
2021). MBPP+ Liu et al.] (2023), HumanEval Chen (202T). and Hu-
manEval+ Liu et al. (2023).
Please refer to Appendix D.I for the detailed filtering pipeline and
dataset descriptions.
7

---

Published as a conference paper at ICLR 2026
Results As detailed in Table LaDiR achieves the highest average performance among all meth- ods. When compared to the reasoning methods sharing the same Qwen3 backbone, LaDiR demon- strates substantial advantage, yielding absolute average improvement of 5.2% over the AR SFT baseline. This gain is more obvious on the challenging HumanEval+ benchmark, where LaDiR sur- passes SFT by nearly 8%, proving its capability synthesize robust and correct solutions for com- plex programming tasks. Furthermore, LaDiR consistently outperforms competing latent reasoning methods, exceeding both the training-free Soft Thinking and the training method TaH+ baselines across all metrics. While specialized models like OpenCoder and Ouro 2.6B show strong perfor- mance on MBPP benchmarks, LaDiR establishes perform better on HumanEval and HumanEval+ by more than 10%, which also outperform strong looped latent reasoning model Ouro 2.6B on these datasets.
Method
MBPP
MBPP+
HumanEval
HumanEval+
Avg.
AR Coding Models
Qwen 2.5 Coder 7B
61.6
53.0
76.9
62.9
63.6
OpenCoder
79.9
70.4
66.5
63.4
70.1
Masked Diffusion Models
LLaDA
50.1
42.1
35.4
30.5
39.5
Dream
68.7
57.4
56.7
50.0
58.2
Diffu-Coder
75.1
61.9
72.0
65.2
68.6
Looped Latent Reasoning Models
Ouro 2.6B
80.4
66.6
78.2
70.7
74.0
Reasoning Methods
AR SFT
63.2
52.8
84.6
76.5
69.3
Soft Thinking
64.2
53.1
85.0
75.2
69.4
TaH+
65.6
56.5
85.8
79.3
71.8
LaDiR
66.8
59.5
87.4
84.2
74.5
Table 2: Code generation performance MBPP and HumanEval benchmarks (including their Plus versions). We report Pass@ accuracy (%). The best results for each metric are highlighted bold.
4.3 PUZZLE PLANNING COUNTDOWN
We evaluate the planning ability of our method using Countdown, a combinatorial arithmetic game. Given of input numbers, the goal is to reach a target in [10, 100] by applying basic operations {+, -, x,,),,+... J. Solving problem thus demands decomposing the target into intermediate subgoals and chaining them correctly. For example, given input numbers {97, 38, 3, 17) and target 14, one valid solution is: 97 38- 59, 59 17 = 42, 42 + = 14. Following Gandhi et al. (2024), we construct dataset of 500k examples, holding out 10% of target numbers for out-of-distribution evaluation as test set. We study two settings of growing complexity: CD-4 and CD-5, which use four and five input numbers, respectively.
Baselines We compare against: (1) LLaMA 8B SFT, an autoregressive baseline finetuned on the same dataset; (2) LLaDA 8B SFT (Nie et al. 2025) and (3) Dream 7B Base (Ye et al., 2025b) (evaluated without finetuning) as ceneral-purposs diffusion models; and (4) MGDM (Ye et al, 2024a), task-specific diffusion model for Countdown.
Results On the Countdown tasks, shown in Table 3 our method outperforms autoregressive baselines and remains competitive with specialized diffusion models. In CD-4, it improves Pass@1 by more than 25% over LLaMA 8B SFT and over 20% over LLaDA SFT, demonstrating stronger global planning ability beyond token-by-token generation, while also delivering the best Pass @ 100 and over two% higher diversity than any baseline. On the more challenging CD-5 task, model surpasses AR baselines by nearly 30% in Pass@1 and over 30% in Pass 100. addition, as shown in Figure 6 our pass k curve rises steeply with k surpassing MGDM larger k. This high pass @k k reflects both diverse trajectory exploration and strong potential for reinforcement learning for post-training (Yue al. 2025)
8

---

Published as a conference paper at ICLR 2026
Model
CD-4 P@1
CD-4 P@ 100
CD-4 Div.
CD-5 P@
CD-5 P@ 100
CD-5 Div.
Dream 7B Base"
16.0
24.7
4.1
4.2
10.3
5.6
MGDM
91.5
95.2
3.2
46.6
70.4
4.9
LLaDA 8B SFT
51.2
75,2
5,4
34.4
45.2
6.2
LLaMA 8B SFT
46.7
65.3
3.0
8.9
15.4
3.5
LaDiR
76.6
96.4
7.3
38.5
75.2
8.9
Table Results on Countdown tasks. We report Pass@ Pass @ 100, and Diversity (Div.). Diversity is measured as the number of unique valid solutions discovered among 100 samples. Best results are in bold, and second-best are underlined. "Dream 7B Base refers to the open-sourced base model without finetuning this task. MGDM is task-specific small discrete diffusion model rather than a general-puu purposs language model.
4.4 ABLATION STUDY
In this section, we study how inference-time stochasticity and diversity guidance affect solution diversity and accuracy by varying (i) the initial noise scale, which controls the variance of Gaus- sian initialization, and (ii) the maximum diversity scale "mmax, which regulates the repulsion strength among latent tokens (see Sec. 3.4). We evaluate both the average number of unique solutions and best-of-100 accuracy. Table 3 shows that increasing noise from to 2 improves both diversity and accuracy, but excessive noise (scale 3) harms convergence despite higher diversity. For diver- sity guidance, removing repulsion (7max 0) yiclds the lowest diversity, while moderate values (0.3-0.5) strike the best trade-off. Stronger repulsion (?max Z 1.0) further boosts diversity but causes accuracy to drop, suggesting that over-dispersing latents destabilizes reasoning.
8.50
Best-of-100 Acc. (%%
8.5
Best-of-100 ACC. 196)
56.4
01.5
-95
Diversity
Diversity
8.0
535
94
7:18
7.5
20.3
01.
2.5
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Initial Noise Scale
Max Diversity Scale Ymax
(a) Effect of Initial Noise Scale 8² (Ymar 0.8).
(b) Effect of Max Diversity Scale (0 2).
Figure 3: Ablation study on parameters for diversity during inference the Countdown-4 dataset.
4.5 ANALYSIS
We present analyses two features of our method: 1) iterative refinement and 2) adaptive test-time compute. Also, we provide additional analyses on inference latency, interpretability and semantic- level reasoning in Appendix
Input
43,9, 81 81
Iterative Refinement Table 4 shows how the our
GT Answer
4349=52, 54-25-29, 52+29=81
flow-matching model refines its reasoning across de-
Decode(2 Zz=1)
"I. (random noise)
noising steps. From pure noise at T' = 1. the model quickly produces structured equations, though early steps contain arithmetic errors (e.g., off-by-one mis- takes). As denoising progresses, partial results sta- bilize-such as 43 9 52 appearing consistently from T ally corrected. By T 0.25, the full reasoning matches the ground truth and remains stable through T 0. This demonstrates that method hibits the same iterative refinement ability as reason- ing models (Shao al. 2024). progressively correct- ing previously generated steps. See Table 10 for an example on GSMSK.
Decnde( 74=0.8)
4348=51,,5-24430, 51+30
Decade( 7==0.7)
33+10=53,54-25-22 53428=81
Decode( Z2
54-27--77,
5242777 52+27=79
Decode Z.=0.5)
3349=52.54-25=29, 55--5=29, 52+28=80
onward and later steps are gradu-
Decode Z.=0.4)
4349=52, 54-25=29, 52+30=82
Decode Decode(Z)=o.25 (2.=0.25
4349=52, 54-25=29, 52+29=81
Decode Zt=0)
4349=52, 54-25=29, 52429=81
Table 4:
Examples of iterative self-
refinement of decoded text from the VAE de- coder on the Countdown-4 dataset across dif- ferent denoising timesteps (t).

---

Published as a conference paper at ICLR 2026
Adaptive Test-Time Compute. As shown in Figure using more denoising steps consistently improves accuracy across different math bench- marks. For example, increasing from 5 to 10 steps (a 2x compute increase) yields large jump of +11.7 points in accuracy on average of 7 bench- marks. Starting from 10 steps, tripling the com- pute to 30 steps provides an additional +4.8 points on average, while a 5x compute increase to 50 steps brings a total gain of +9.8 points on average. These results demonstrate that our method can flexibly trade test-time compute for higher per-
Dstasets
06M3K
MATH
Apcuracy (%)
Derioan
Ohymloo
Muthods
LaDiR
Cirl SP
# Denoising Steps
Figure 4: Effect of number of denoising steps on downstream reasoning performance on the math reasoning tasks.
formance as an alternative paradigm in reasoning for long CoT of existing reasoning models (Jaech
et al. 2024 Muennighoff et al. 2025) Liu et al.]
2024a Li et 2025). This may motivate adaptive policies that dynamically assign more denoising steps to harder queries, maximizing the overall accuracy-comput trade-off.
5 RELATED WORKS
Text CoT Reasoning Chain-of-thought reasoning WWii et al. 2023) refers to methods which elicit
LLMs to generate intermediate reasoning steps in language prior to outputting a final answer in
order to improve performance on reasoning tasks. This can be accomplished via prompting methods
Wei et al. 2023 Khot et al. 2023 Kang et al. 2025 Zhou et al. 2023) or through training LLMs (by SFT, RL, or combination of the two) to output the intermediate reasoning steps
(Nye et al. 2021
al. 2023 Shao et al. 2024 Yu et al. 2025a). Works have also extended CoT to allow LLMs to
mimic various tree search algorithms such as BFS MCTS, which especially improves performance on more complex tasks (Xie et al. 2003] Yao et al. 2023 |023]Zhang Zhang et al. 2024 et 2025). Beyond (combining extensive reasoning. exploration, and reflection) have also demonstrated improved reasoning performance (Shinn et al.) [ Gandhi et al. [2025] Saha et al., 2025 Xie al. (025). One limiting factor with these CoT methods is that they fundamentally work at text token level, constraining the outputs to the token space and limiting the model's horizon.
following specific algorithms, works that implement long
Latent Reasoning Latent reasoning methods address token-level limits of CoT's Wei et al. (2023) by enabling reasoning in a latent space, which only requires higher levels of semantic abstrac-
tion (Hao et al., 2024, Přau et al. 2024 Wang al., 2024 Zelikman et al. 2024 Jin et al. 2025) Prior works show that reasoning in latent space, rather than discrete tokens, improves performance by allowing L.MS to generate continuous tokens, either self-generated or provided by an auxiliary
model (Cheng & Durme 2024 Liu et al.) 2024b; [Shen et al. 2025b) Tack et al. 2025 Zhu et al.) 2025a Butt et al. 2025; Zhang et al. 20255 Wu et al., 2025). This has been further extended to recurrent or looped architectures that induce latent reasoning internally, removing the need to repre- sent reasoning steps explicitly as tokens (Chen et al. 2025cc] Geiping et al., a 2025 Mohtashami et al.) 2023 Saunshi et al. 2025 Bac et al. 2025 Zhu et al.) 2025b). However, prior latent reasoning ap- proaches lack interpretability. whereas our method constructs the latent space with a VAE, making each step explicit and thus more transparent. Due to the page limit, we discuss further related works in Appendix A
6 CONCLUSION
We introduced LaDiR, latent diffusion reasoning method that utilizes the iterative refinement ca- pability of latent diffusion models to perform reasoning at the semantic level, our framework offers three key benefits: (1) better tradeoff between accuracy and test-time compute through iterative de- noising steps with self-refinement, (2) diverse exploration of reasoning trajectories, and (3) enhanced interpretability through semantically meaningful latent representations. Our experiments on mathe- matical reasoning, code generation and puzzle planning benchmarks show that LaDiR consistently outperforms AR and diffusion baselines, achieving both better accuracy and diversity in reasoning.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENT
This research is supported by a research grant from Apple. Also, it is partially supported by the NSF award CCF-21 12665 (TILOS). It is also supported in part by the CDC-RFA-FF123-0069 from the CDC's Center for Forecasting and Outbreak Analytics.
8 ETHICS STATEMENT
We have read and carefully considered the ICLR Code of Ethics. Our work develops LaDiR, a latent diffusion reasoning method for large language models, with its application scope primarily constrained to mathematical and puzzle-solving domains, We do not believe that this architecture raises any ethical concerns. We affirm that our research and its intended use fully adhere the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
In order to ensure reproducibility, we release our code for both the VAE training in Section 3.2.1 and the reasoning model (3.3) training the Supplementary Material. Note that training follows two-stage pipeline as outlined in Section first the VAE is trained and second the latent reasoning model trained, using the VAE encoder outputs during its training. All datasets used are referenced and publicly available.
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. arXiv preprint arXiv:2503.cm 09573, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2025. URL
https:
Sangmin Bae, Yujin Kim, Reza Bayat, Sungnyun Kim, Jiyoun Ha, Tal Schuster, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji, Aaron Courville, et al. vixtureeof-reccursion::: Learning dynamic recur- sive depths for adaptive token-level computation. arXiv preprint arXiv:250..10524, 2025.
Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, and Yunhe Wang. Forest-of-thought Scaling test- time compute for enhancing Ilm reasoning, 2025. URL
09078
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. no: A vision- language-action flow model for general robot control, 2024. URL sttEps://arxiv.ok abs/ 2tts/2410.2411.
Natasha Butt, Ariel Kwiatkowski, Ismail Labiad, Julia Kempe, and Yann Ollivier. Soft tokens, hard truths. arXiv preprint arXiv:2509.197 2025.
Edoardo Cetin, Tianyu Zhao, and Yujin Tang. Large language models to diffusion finetuning. arXiv preprint arXiv:2501.1578 2025.
Mark Chen. Evaluating large language models trained on code. arXiv preprint arXlw:2107.03374,
2021.
Shoufa Chen, Chongjian Ge. Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui WU, Zhichao Lai, Yifei Hu, Ting-Che Lin, Shilong Zhang, Fu Li, Chuan Li, Xing Wang, Yanghua Peng, Peize Sun, Ping Luo, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Goku: Flow based video generative foundation models, 2025a. URL https: //axxiv.... wxzzv.org///
abs abs/2502. 2502 04896
11

---

Published as a conference paper at ICLR 2026
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: theorem-driven question answering dataset. arXiv preprint 122.:22300..2524.. 2023.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Jamus-pro: Unified multimodal understanding and generation with data and model scaling, 2025b. URL htRL:htts: wwwp:eiwwwoooorroooo
org/ abs 2
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-ikee Ilms, arXiv preprint arXiv:2412.211877 2024.
Yilong Chen, Junyuan Shang, Zhenyu Zhang, Yanxi Xie, Jiawei Sheng, Tingwen Liu, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Inner thinking transformer: Leveraging dynamic depth scaling to foster adaptive internal thinking, 2025c. URL httts::/axxxv..rrggg/e/eo/
22502.1844
Jeffrey Cheng and Benjamin Van Durme. Compressed chain of thought: Efficient reasoning through dense representations, 2024. htll/ttps://aratt.. wwwexiivrrg//ee......o abs/2412 .1171
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 2021.
Codefuse and Ling Team. Every sample matters: Leveraging mixture-of-experts and high-quality data for efficient and accurate code 1lm, 2025. URL /tttps://axxivv.rg///aa abs/53333
17793
Yuntian Deng, Kiran Prasad, Roland Fernandez, Paul Smolensky, Vishrav Chaudhary, and Stu- art Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311.014660 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv e-prints, pp. 24077 2024.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xi- ang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi. Faith and fate: Limits of transformers on compositionality, 2023. URL
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens, 2024. Utthttts::/ arxiv org/ /400 13863
Tianyu Fu, Yichen You, Zekai Chen, Guohao Dai, Huazhong Yang, and Yu Wang. Think- at-hard: Selective latent iterations to improve reasoning language models. arXiv preprint arXiv:2511.08577, 2025.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos); Learning to search in language. arXiv preprint arXiv:2404.088 000-23344.. 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cogni- tive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL httLtttsss//aaiiii org/ abs//bb3. 01307
Silin Gao, Mete Ismayilzada, Mengjie Zhao, Hiromi Wakaki, Yuki Mitsufuji. and Antoine Bosselut. Diffucomet: Contextual commonsense knowledge diffusion, 2024. URL
hrg/abbs2402.. 17011
Jonas Geiping, Sean McLeish, Neel Jain, John Kirchenbauer, Siddharth Singh, Brian R Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, and Tom Goldstein. Scaling up test-time compute with latent reasoning: A recurrent depth approach. arXiv preprint 1rXiv:2502.05/71, 2025.
12

---

Published as a conference paper at ICLR 2026
Shansan Gong, Mukai Li, Jiangtao Feng. Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933, 2022.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi. Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models, 2025. URL https::/aaxxiei www.woorg/aaee abs/
241 0.10... 17891
Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens, arXiv preprint arXiv:2310.02226, 2023.
Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Ad- vances in Neural Information Processing Systems, 36:16693-16715. 2023.
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space, 2024. URL https:
7/arxiv. /awxxiv.rg/aab////aaaaaaa http:/aaaa2221....
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03877 2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Jonathan Ho and Tim Salimans. arXiv:2207.12500 12598, 2022.
Classifier-free
diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet, 2024. URL https::
warxii.org/abs/223333 2atdbb//2111.m 017998
Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, et al. Opencoder: The open cookbook for top-tier code large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 33167-33193, 2025.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409. /2 186, 2024.
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion llms via adaptive parallel decoding, 2025.
00413
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint 1.X2:22412.16700. 2024.
Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. Disentangling memory and reasoning ability large language models, 2025. UtI.httts:://eeec
abs 22411 13504
13

---

Published as a conference paper at ICLR 2026
Haoqiang Kang, Enna Sachdeva, Piyush Gupta, Sangjae Bae, and Kwonjoon Lee. Gflowvlm: En- hancing multi-step reasoning in Sanounla models with generative flow networks. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 08855,825,, June 2025.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks, 2023. URL
https: //erxiv /ttp:///wwwx.rraaao////////a/a//
Diederik P Kingma and Max Welling, Auto-encoding variational bayes.
arXiv preprint
13 14, 2013.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto. Diffusion- Im improves controllable text generation. Advances in neural information processing systems, 35: 4328 4343, 2022.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprint 2XXi:25502117411 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chat- gpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558-21577 2023.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.0544 2025.
Tianqiao Liu, Zui Chen, Zitao Liu, Mi Tian, and Weiqi Luo. Expediting and elevating large language languaae model reasoning via hidden chain-of-thought decoding, 2024b. URL hnttps://arxiin org/
abs/2400 08561
Aaron Lou, Chenlin Meng. and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310. 16834, 2023.
Justin Lovelace, Varsha Kishore, Chao Wan, Eliot Shekhtman, and Kilian Q Weinberger. Latent dif- fusion for language generation. Advances in Neural Information Processing Systems, 36:56998- 57025, 2023.
Justin Lovelace, Varsha Kishore, Yiwei Chen, and Kilian Q Weinberger. Diffusion guided language modeling. arXiv preprint arXiv:2408. 04220, 2024.
Justin Lovelace, Christian Belardi, Sofian Zalouk, Adhitya Polavaram, Srivatsa Kundurthy, and Kil- ian Q Weinberger. Stop-think-autoregresss Language modeling with latent diffusion planning. arXiv preprint arXiv:2602.20528. 2026.
Viacheslav Meshchaninov, Egor Chimbulatov, Alexander Shabalin, Aleksandr Abramov, and Dmitry Vetrov. Compressed and smooth latent space for text diffusion modeling. arXiv preprint arXiv:2506.211700 2025.
Amirkeivan Mohtashami, Matteo Pagliardini, and Martin Jaggi. CoTFormer: A chain of thought driven architecture with audget-adaptive computation cost at inference. In The Thirteenth In- ternational Conference on Learning Representations, 2025. URL https: htttp:://openreviee het./forum2io=71qpp00
14

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fee, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, arXiv preprint arXiv:2501.19393 2025.
Ranjita Naik, Varun Chandrasekaran, Ment Yuksekgonul, Hamid Palangi, and Besmira Nushi. Di- versity of thought improves reasoning abilities of 1lms, 2024. URL https: //axiiv////iii../// org/
abs/2310. 07088
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025. URL https://
arxiv..rgg/bss/55222 02900
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Au- gustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.
00114
Xichen Pan, Satya Narayan Shukla, Aashu Singh, Zhuokai Zhao, Shlok Kumar Mishra, Jialiang Wang, Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, Hou, and Saining Xie. Trans- fer between modalities with metaqueries, 2025. URL http:ps://axii..rr/////// 06256
Jacob Pfau, William Merrill, and Samuel R. Bowman, Let's think dot by dot: Hidden computation in transformer language models, 2024. URL ftttps://arivvv org/ abs /2004 15758
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with mainking-llm-ss--jjueee 2025. URL/ttps://arxii..rr
2bbs/2501.8899
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136-133 130114, 2024.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.005122 2022.
Nikunj Saunshi, Nishanth Dikkala, Zhiyuan Li. Sanjiv Kumar, and Sashank J. Reddi. Reasoning with latent thoughts: On the power of looped transformers, 2025. URL bttps://arxiv:
htg//bb//00 2502. 17416
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical rea- soning abilities of neural models. arXiv preprint 1XXiv:1904.00555 2019.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL
03300
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing chain-of-the thouuht into continuous space via self-distillation. arXiv preprint arXiv:2502.21074, 2025a.
Zhenyi Shen, Hanqi Yan, Linhai Zhang, Zhanghao Hu, Yali Du, and Yulan He. Codi: Compressing ch into continuous space via dittliitillation, 2025b. URL https://arxxi:. org/ bbg//bb//a / 2502. 21074
Jiaxin Shi, Kehang Han, Zhe Wang, Araud Doucet, and Michalis K. Titsias. Simplified and gener- alized masked diffusion for discrete data, 2025a. URL https::///// /
04322
15

---

Published as a conference paper at ICLR 2026
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Lmfusion: Adapting pretrained language models for multimodal generation, 2025b.
https::/aaxxiiyt: org/ abs 222224122222 15188
Noah Shimn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL
https: //aa::////xvvvi org. abss /2.03 11366
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Lc, Carina Negreamu, and Gust Verbruggen. Codefusion: A pre-trained diffuusinn model for code generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11697-11708, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Yuxuan Song. Zheng Zhang, Cheng Luo. Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang. Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang, Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei- Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu, and Hao Zhou. Seed diffusion: A large- scale diffusion language model with high-speed inference, 2025. URL
hittps://arxiv http://aaxxvv..r//bbb.seee 2508 02193
DiJia Su, Hanlin Zhu, Yingchen Xu. Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint arXiv:2502.03275, 2025.
Jihoon Tack, Jack Lanchantin, Jane Yu, Andrew Cohen, Ilia Kulikov, Janice Lan, Shibo Hao, Yuan- dong Tian, Jason Weston, and Xian Li. Llm pretraining with continuous concepts, 2025. URL ats.aaa.////oooooooaaoo.a.a.aa.a abs 2502. 08524
pictps:/aazie
Haotian Tang, Yecheng Wu, Shang Yang. Enze Xie, Junsong Chen, Junyu Chen, Zhuoyang Zhang, Han Cai, Yao Lu, and Song Han. Hart: Efficient visual generation with hybrid autoregressive transformer, 2024a. URL https: // www.xxiv.ooggaaaaa 24.0..11.1100
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruction tuning for mathematical reasoning. arXiv preprint arXiv:2403. 028884, 2024b.
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning, 2024a. URL https://arxivv org/ abs 2412. 141644
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficult--wwree rejection tuning for mathematical Sabblem-solvingg Advances in Neural Information Processing Systems, 37:7821-7846,, 2024b.
Emile Krieken, Pasquale Minervini, Edoardo Ponti, and Antonio Vergari. Neurosymbolic dif- fusion models, 2025. URL
Xinyi Wang. Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessan- dro Sordoni. Guiding language model reasoning with planning tokens. In First Conference on Language Modeling, 2024. URL //tp:////ee...e.eee///ee net/ forum ? wi9IffREEM
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models, 2023. URL
a2spg///bb/221..... 11903
Ashen Weligalle. Discrete diffusion models for language generation. arXiv:2507.07050, 2025.
arXiv preprint
Chünhung Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai, and Hua Wu. Lims single-threaded reasoners: Demystifying the working mechanism of soft thinking. arXiv preprint arXiv:2508. 03440, 2025.
16

---

Published as a conference paper at ICLR 2026
Jianxiang Xiang, Zhenhua Liu, Haodong Liu, Yin Bai, Jia Cheng, and Wenliang Chen. Diffu- siondialog: A diffusion model for diverse dialog generation with latent space. arXiv preprint arXiv:2404.06760, 2024.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation, 2024. URL
https::/arxiie org/ /
/ot/o///24433 2409. 11340
Tian Xie. Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong. Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning, 2025. URLhttps: / arxiv.. org/ abs 2502. 14768
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min- Kan, Junxian He, and Qizhe Xie. Self-evaluation guided beam search for reasoning, 2023. URL https::/erxxv..rrrrrr/ org/bbs/ 2335.00633
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
httpt:///a/xxii..//// org/ abs/2305...... 10601
Jiacheng Ye, Jiahui Gao. Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning, arXiv preprint 14.4:: 2024a.
Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusion of thoughts: Chain-of-thought reasoning in diffusion language models, 2024b. URL https:://////.... httpss///axxvvv aawaaaiivvv.......a..aaa org/ abs 2402. 07754
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong.
Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2025a. URL
https://arxiv. wwwp:/wwwwww..raa////a/ooaaa
14157
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion large language models. arXiv preprint arXiv:2508.15453 2025b.
Jiasheng Ye, Zaixiang Zheng, Yu Bao, Lihua Qian, and Quanquan Gu. Diffusion language mod- els can perform many tasks with scaling and cnarucciionfinetuniin 2025c. URL https:
//arxiv. www./rr//////////////...//eeeeee abaaa/aa33000 /2008. 12219
Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Efficient training of I1m policy with divergent thinking. arXiv preprint arXiv:2406.0566 1(2):6, 2024.
Fangxu Yu, Lai Jiang, Haoqiang Kang, Shibo Hao, and Lianhui Qin. Flow of reasoning: Training Ilms for divergent reasoning with minimal examples. In Forty-second International Conference on Machine Learning, 2025a.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.
Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding. arXiv preprint a5Ssv: 16990, 2025b.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint xxXiv22504.138. 2025.
Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah Good- man. Suiet-STar: Language models can teach themselves to think before speaking. In First Conference on Language Modeling. 2024. URL
net fforrmiidi
RRXP 9
17

---

Published as a conference paper at ICLR 2026
Yiming Zeng. Jinghan Cao, Zexin Li, Yiming Chen, Tao Ren, Dawei Xiang, Xidong Wu, Shangqian Gao, and Tingting Yu. Treediff: Ast-guided code generation with diffusion llms. arXiv preprint arXiv:2508.01473. 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. ReST-MCTS*: LLM self-training via process reward guided tree search. In The Tiirty-eighth Anmual Confer- ence on Neural Information Processing Systems, 2024. URL /eLLattps//peeeevieee.uo
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua Susskind, and Navdeep Jaitly, Plan- ner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 36:80178 11880017888190,,, 2023.
Zhen Zhang, Xuehai He, Weixiang Yan, Ao Shen, Chenyang Zhao, Shuohang Wang, Yelong Shen, and Xin Eric Wang. Soft thinking: Unlocking the reasoning potential of Ilms in continuous concept space. arXiv preprini 1XXx:22505.55755 2025.
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. reparameterized discrete diffusion model for text generation, 2024. httptttss///.x..rrrg/////
05737
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Ja- cob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024a. https:: org/ /bs/
14C8.11039
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables com- plex reasoning in large language models, 2023. URL
10625
Zixuan Zhou, Xuefei Ning, Ke Hong. Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, and Yu Wang. A survey on efficient inference for large language models, 2024b. URL https:
14294
Hanlin Zhu, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. Reason- ing by superposition: A theoretical perspective on chain of continuous thought. arXiv preprint MrXiv:2505.12514 2025a.
Rui-Jie Zhu, Zixuan Wang, Kai Hua, Tianyu Zhang, Ziniu Li, Haoran Que, Boyi Wei, Zixin Wen, Fan Yin, He Xing, et al. Scaling latent reasoning via looped language models. arXiv preprint arXiv:2510.25741, 2025b.
18

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORKS
Diffusion Language Models for Text Reasoning Masked diffusion language models attempt to address some common limitations of autoregressive LLMs such as rigid left-to-right decoding and inefficiency-by iteratively denoising masked tokens, enabling parallel and order-agnostic text gen- eration. Prior studies show that these models achieve better inference efficiency compared to AR models while maintaining comparable performance on both general tasks (Zheng al.
2024 Gong
et al. 2025 Nie et al. 2025: Shi et al. 2025a Song et al.) 2025 Ye et al. 2205cc b) and reasoning benchmarks with chain-of-thought (Gao al. 2024 Ye et al. [024bb van Krieken et al.) 2025 Ye
et al. 2025a). However, these approaches remain constrained to language space, unable to capture reasoning at an abstract semantic level or revise previously generated tokens as continuous diffu- sion models (Ho et al., 2020; [Song et al. (020)) do, and they require training massive datasets rather than leveraging a well-trained LLM. In contrast, our method overcomes these limitations by structuring reasoning in an interpretable continuous latent space, producing abstract CoT representa- tions with self correctio ability for an existing LLM, while keeping diffusion's 's strengths in parallel generation to enhance exploration and diversity.
Latent Diffusion for Language Generation Generative text modeling has recently expanded from autoregressive paradigms to diffusion- based approaches that allow for global iterative refine-
ment. One of the first, Diffusion-LM (Li et al. 2022), frames generation as denoising continuous
word embeddings to enable fine-graines control, concept Lovelace et al. (2023 2026) extended
by performing diffusion in a compressed latent space for improved quality and diverse generation modes. For sequence- -o-suueeeeee tasks, DiffuSeq (Gong et al., 2022) enables parallel generation with high diversity, while PLANNER (Zhang et al. 2023) addresses long-form text by combining latent semantic diffusion planner with an autoregressive decoder reduce repetition. Similarly, Cosmos (Meshchaninov et al. 2025) learns a compressed latent space for diffusion, enabling par- allel text generation with robust semantic grounding. In specialized domains. DiffusionDialog (Xi- ang et al. (224)) utilizes latent variables to handle open-ended conversations, whereas CodeFu- sion (Singh et al. 2023) and Tife (Zeng et al., 2025) apply diffusion to code synthesis. While prior latent diffusion models focus on text generation, they lack the granularity to model the multi- step causal dependencies required for reasoning tasks. We address this by introducing blockwise variable-length diffusion and rollout training, explicitly shifting the objective from generating fluent text to optimizing reasoning trajectories that lead to correct answers.
Hybrid AR AR+Diffusion Model Architecture Other AR-Diffusion hybrid models have shown suc- cessful results rivaling their AR and diffusion counterparts, particularly in multimodal generation and image understanding. The Transfusion (Zhou et al.) 2024a) architecture demonstrated that hybrid models could outperform standard AR models and compete with state-of-the-art diffusion benchmarks, phenomenon further reinforced by other studies of hy- brid models (Fan et al. 2024 Tang et al., 2024a Xiao et al.) 2024). This extends beyond image generation, with several works demonstrating the effectiveness of hybrid AR-diffusion models in other domains such as image understanding, video generation, and robot control (Black et al. 2024] Tong et al. 2024a Chen et al. 2025a b). Furthermore sineeemmorr---milarrr similar to model architecture works have demonstrated successful adaptations of frozen models for these hybrid AR-diffusion archictures in multimodal domains (Pan et al. 2025 Shi et al.) 2025b). Aside from the difference in domain from these works, many do not use block diffusion for variable-length generations as in LaDiR and we critically introduce CE loss guide better latent predictions.
models in smage-generatee
B ADDITIONAL PRELIMINARIES AND BACKGROUND
We provide more details about the background information of VAE and Diffusion models in this section.
B. VARIATIONAL AUTOENCODER AND 3- VAE
The Variational Autoencoder (VAE) Kingma& & Welling 2013) is a latent-variable model that learns compressed representation of data through encoder decoder pair. The encoder (=(2)=) maps inputs into a distribution over latent variables z, typically parameterized as a diagonal Gaussian. The decoder Po (z/²) reconstructs the input from 2, enabling generative sampling. Training maximizes
19

---

Published as a conference paper at ICLR 2026
the evidence lower bound (ELBO):
VAV (xlo) lo + (T(((z)))) I|((z))
(6)
where the first term ensures faithful reconstruction and the second term regularizes the posterior toward simple prior p(z), usually (V,0, I). The reparameterizatioo trick.
( + (x(x) O c~N(0, I),
enables low-variance gradient estimates for stochastic optimization.
3-VAE. The 3-VAE (Higgins et al. 2017 introduces hyperparameter 3 to control the KL weight:
LaVAR
(ooesta)lllooo(1100 + K[(go==/x) ([p=))
When 3
the model enforces stronger alignment to the prior, which encourages disentangled and interpretable latent variables the expense of reconstruction fidelity. This property is desirable when latent codes are later used as the substrate for generative modeling.
Why VAE for Latent Diffusion. Latent diffusion models (L.Ms)) (Rombach et 2022) operate not on raw high-dimensional inputs (e.g.. images or sequences), but in a compressed latent space learned by VAE. This design provides three key advantages:
1. Efficiency. Operating in latent space reduces dimensionality, leading to faster training and inference while maintaining semantic richness.
2. Semantic abstraction. The VAE learns to discard imperceptible details and retain high- level structure, making diffusion steps focus on meaningful features rather than pixel-level noise.
3. Flexibility. The decoder pr(z)) ensures that even when denoising occurs in latent space, the final output remains in the original input domain. This separation enables diffusion generalize across modalities with shared latent backbone.
B.2 LATENT DIFFUSION; TRAINING AND INFERENCE
Latent diffusion operates in the compressed latent space zo of a pretrained VAE.
Forward process. Noise is added gradually:
( N(2 Voa 2o, (1 -1- āt)
with 11 (1 B.)-
Training objective.
The denoiser e (0(z () predicts the injected noise:
LLDM Ezo,s,t stt [lle ca (= [)1
Inference.
Generation starts from 2T NO, I) and denoises iteratively:
(2x
Br v1 ās co(2r1t) + ozE.
24-1
Vor
B.3 COMPARISON OF PARAMETERIZATIONS
Diffusion training can be expressed through different target parameterizations, all of which can be interpreted as variants the same continuous- time flow. Below we summarize the most common forms:
B.3.1 <-PREDICTION
The denoiser directly predicts the added Gaussian noise
C. Ezg.e, ||e e (z t)1..
(8)
where 2, = Varza + VT C.C. This the standard DDPM formulation Ho et al. 2020 is
stable but sometimes less efficient for long horizons.
20

---

Published as a conference paper at ICLR 2026
Objective
CD-4 Pass@ (%)
MSE Loss
46,0
30
53.0
58.0
62.0
u (ours)
73.5
Table 5: Ablation study on latent prediction objectives on the Countdown-4 dataset.
B.3.2 To- CO-PREDICTION (DDIM-X0)
Instead of noise, the model predicts the clean latent zp:
L.o zo. || 20,0(21,
(9)
This corresponds to the DDIM formulation (Song et al. 2020). enabling deterministic sampling and fewer inference steps, but overfit to data scale.
B.3.3 v-PREDICTION
Proposed by Salimans & Ho 2022 v defined as a linear combination of noise and clean latent:
Eāge- vI ac 20:
(10)
with objective
C. Ezo,e, [||: c0 t)]?]]
(11)
preddiction is numerically better conditioned, often improving stability across timesteps.
All four parameterizations can be viewed as different instantiations of the same underlying genera- tive flow. e-prediction, ro-prediction, and u-prediction specify which quantity the denoiser regresses Flow matching directly learns the continuous velocity field, avoiding discretization artifacts.
B.4 BLOCK DIFFUSION
To support flexible and variable-length sequence generation, employ block diffusion scheme (Ariola et al. (025) that integrates autoregressive modeling with diffusion. Instead of applying diffusion to individual latent tokens full sequence, the sequence divided into contigu- ous blocks, and diffusion is performed at the block level. This hybrid design retains the open- open-ended generation of autoregressive models while introducing global coherence within each block.
Suppose we are using e-prediction, and let a sequence be segmented into M blocks (BI...., Bu }. where Bm E pxxd contains k latent tokens of dimension d. The forward noising process for block Bm is
q (Bmt Bm,o) N(Vo, Bm T))
(12)
and the denoiser fo is trained to predict the noise the block level:
Colock Em.t. [||s fo (B =]]
(13)
Blocks are generated autoregressively. i.e..
/ q( q(Bm,o x))
BBmm Bem)
(14)
x I Po (Bm.-11 Bm, Beg)) dBm,n-
that each block denoised iteratively while conditioning on all previously generated blocks.
21

---

Published as a conference paper at ICLR 2026
Reconstruction Accuracy (%)
100
99.3
100.0
100
Downstream Accuracy (%)
80
84.4
95
66.3
60
8
40
22
85
12.3
16
# Latent Tokens per Block (Lb)
Figure 5: Ablation analysis of block size the GSM8K benchmark.
Latent Prediction Objective To assess the impact of different training objectives for latent pre- diction, we compare several widely used formulations. The first baseline is MSE loss, which directly minimizes the mean-squared error between predicted and ground truth latents but yields the weakest results. We then adopt three DDIM-based (Song et al. 2020) objectives: predicting the clean latent state (2o),, the added noise vector (e), and the velocity (w). These diffusion objectives consistently improve accuracy, highlighting the advantage of explicitly modeling the denoising process rather than relying on direct predictions. Moreover, the flow matching (u) objective achieves the strongest gains, suggesting that learning the latent vector field is more effective for capturing the pattern of reasoning compared to DDIM's denoising objectives.
Effect of the Block Size. We investigate how the number of latent tokens per block (Ls) influences reconstruction quality and downstream reasoning accuracy on GSM8K. As shown in Figure 5 too few tokens (i.e., token) limit the model's s ability to capture necessary information, harming recon- struction. Performance improves as the number of tokens increases. reaching near-perfect recon- struction at = 6. Beyond this point, however, adding more tokens introduces redundancy, which makes the latent diffusion model harder to predict accurately and leads to diminished reasoning ac- curacy. This reveals trade-off between compact latent representations and effective downstream reasoning.
VAE Robustness Augmentations. When training our VAE, to improve the robustness VAE latent space, apply two augmentation strategies during VAE training: (1) adding Gaussian noise to latent representations with standard deviation k; and (2) randomly substituting input tokens with probability p. Table 6 shows the impact of these augmentations formance is achieved at h = 3 and p = 0.3. Too little augmentation (k 0 or p = 0) results in overfitting to clean inputs, while excessive augmentation (I: 5 0.5) degrades the latent space quality.
GSM8K accuracy. Best per-
Gaussian Noise (p=0.3)
Token Substitution (k=3)
A (std)
Acc. (%)
p (prob.)
Acc. (%)
0
68.3
0.0
70.2
73.4
0.1
78.3
3
84.2
0.3
84.2
79.4
0.5
0.7
32.4
Table 6: Ablation study VAE robustness augmentations GSM8K.
22

---

Published as a conference paper at ICLR 2026
Blockization Strategy We investigate the sensitivity of the model to different blockization strate- gies by varying the number of sentences per block. Thbls results for 1, and 3 sentences per block. Using more sentences ity and significantly increases difficulty for the diffusion model. We find that sentence per block with 4 latent tokens offers the best balance between latent compactness and reasoning accuracy.
block requires more latent tokens to maintain reconstruction qual-
# Sentences
# Latent Tokens
GSM8K
MATH
1
4
84.2
45.2
2
8
78.4
39.6
3
12
72.0
36.1
Table 7: Ablation study on blockization strategy (sentences per block).
23

---

Published as a conference paper at ICLR 2026
100
90
pass@k (%)
80
70
60
LLaMA SFT
50
LLaDA SFT
MGDM
40
LaDiR (Ours)
30
0
20
40
60
80
100
k
Figure 6: Results for pass@kk performance on Countdown-4 with 1: E (1, 10, 50, 100).
C ADDITIONAL RESULTS AND ANALYSIS
Intepretability In addition to achieving superior or competitive accuracy across each benchmark, as shown in Table & LaDiR also benefits from being more interpretable by nature compared to standard diffusion-barr baased methods.
Block
Text
Question
Billy sells DVDs. He has 8 customers Tuesday. The first buy one DVD each, the next 2 buy DVDs each, the last 3 buy none. How many DVDs did Billy sell?
Decode (rrodeZ(()) (2))::
Billy's
customers buy one DVD each, so that's His next 2 customers buy DVDs each, that's 2 4 DVDs.
((3 + 3))3 DVDs.
Decode( (z))):
Decode(2 (())):
His last 3 customers don't 't buy any DVDs, so that's 0 DVDs sold.
Decode (()): Therefore, Billy sold total of 4 + DVDs on Tuesday.
Answer
The answer is:
Table 8: Example of interpretable continuous thought tokens: each latent block Z' is able be de- coded to human-readable text through the VAE decoder. Each latent block is decoded individually, so the entire latent thought is represented by the block isolation. This allows for clear interpretibil- ity of each latent thought, while still allowing for a model to reason in a latent space.
Inference Efficiency Table 9 compares inference latency on the MATH dataset using identical hardware and batch size (8). LaDiR with 10 diffusion steps matches the latency of the AR baseline while achieving comparable Pass and higher Pass 100. With 30 steps, LaDiR offers a flexible accuracy-compuu trade-off. This efficiency latent block contains only 4 latent tokens, representing on average 22 text tokens, reducing per-step computation and context length compared autoregressive decoding over long text sequences. Reasoning at Semantic Level Table T0 demonstrates that LaDiR refines its reasoning through semantic information rather than lexical connections. The pink segments trace how the model in-
from our compact latent representation: each
Method
Latency
Pass@1
Pass
I.LaMA CoT SFT
4.7
43.1
89.0
LaDiR (10 steps)
4.9
42,9
LaDiR (30 steps)
14.8
44.9
92.8
Table 9: Inference latency (seconds/example) comparison MATH dataset.
24

---

Published as a conference paper at ICLR 2026
erementally refines its intermediate reasoning steps transforming rough incomplete arithmetic expressions into logically consistent equations. For example, the model first produces an approxi- mate relation such as 2 1 and later reconstructs the correct multiplicative form 2 eventually combining these into a complete causal chain that leads to 2 green segments serve as stable anchors capturing the essential reasoning cues- cues-phrases like "two times older than" and "four times as old"-which define the causal dependencies between Suzy, Ben, and Brandon. In contrast, the yellow segments correspond non-reasoning noisy parts (e.g., "Alice," "Instagram") ") that the model learns to "ignore" during early denoising, prioritizing semantically coherent and logical grounded reasoning over irrelevant lexical content. In contrast, the SFT baseline remains syntactically fluent but semantically brittle- pridule-pooducig text that appears correct on the surface yet fails to capture the true causal logic. These observations indicate that LaDiR's denoising process operates on the level of semantics and relational consistency, enabling reasoning that aligns with human-like conceptual understanding.
2, 8. Meanwhile, the
Question
Brandon's 's iPhone is four times old Ben's iPhone. Ben's 's iPhone is times older than Suzy's 's iPhone. Suzy's 's iPhone year old, how old is Brandon's iPhone?
SFT (Baseline)
Suzy's iPhone year old. Ben's iPhone would be + 1 2 years old. Since Brandon's 's iPhone is four times as old as Ben's, we compute x 1 = 4. Therefore, Brandon's 's iPhone is 4 years old. The answer is: 4.
LaDiR (Ours)
Block
Block 2
Decode (
10 and 10 10.
[garbled tckens]
Decnde (2
If Alice is year old, then Bob's age
3///natccaandra' Instagram
is years older than Alice, which
is
Four times as old Instagram, so Ben's 1+4= years old.
as Ben's
means Bob 2 2
years old.
Instagram is
If Suzy's age
year old. then
If
Brandon's
phone is
two times older than Ben's 'sn'san age is
four times as old phone. then Brandon's 2 2 years old. Inbsp:nbsp:nbsp.....
as Ben' 's
Suzy's
age.
which is
phone is
= 2 2 years old.
Decodo(2iga.7)
Suzy's 's iPhone is year old, then Ben's iPhone is two times older than Suzy's iPhone, 2 years old.
If
Brandon's
iPhone is
Four times as old as Ben's
Ben's iPhone
iPhone. then Brandon's iPhone is
2 8F4=8years old.
Answer
The answer is: 8.
Table 10: An example of self-refinement during inference on the GSMSk dataset, showing how reasoning becomes progressively clearer as t decreases. Later denoising steps correct arithmetic emrors while maintaining earlier structure, demonstrating semantic self-refinement. Pink segments highlight refined reasoning portions, yellow segments indicate non-reasoning noisy parts that the model gradually corrects, and green segments denote key reasoning cues essential for correct logic.
Consistency Across Long Reasoning Blocks As shown in Table I1 LaDiR maintains coherent and semantically consistent reasoning across long sequence of blocks. Each block builds logically on the previous one one-starting from variable definition (a as silver coins), to constraint formulation (x (x 30) 110), simplification, and finally the correct numerical conclusion (z 40, gold 70). Unlike the SFT baseline, which produces a single-step approximation that conflates interme- diate relations, LaDiR preserves arithmetic and causal consistency throughout the reasoning chain, demonstrating stable multi-step inference even with large number of reasoning blocks.
25

---

Published as a conference paper at ICLR 2026
Question
Gretchen has 110 coins. There 30 gold coins than silver coins. How many gold coins does Gretchen have?
SET (Baseline)
Let's assume Gretchen has 110 coins in total and 30 more gold than silver. Half of coins plus 30 should be gold, so 110/2 + gold coins.
LaDiR (Ours)
Decode( 2
Let's assume the number of silver coins Gretchen has æ silver coins.
Decode (2))
We also know that there are more gold coins than silver coins, number silver coins is x + 30 gold coins.
Decode 2(())
The total number of coins Gretchen has is x + (ar + 30) = 110.
Decode( )
Combining like terms, get 2z + 30 110.
Decode( 2(0))
Subtracting from both sides, get 2r 80.
Decode(Z(0)
Dividing both sides by 2, we get æ
Decode(2
Therefore, Gretchen has 30 + 40 70 gold coins.
Answer
The answer is: 70.
Table 11: An qualitative example in GSM8K illustrating the long reasoning blocks generated by our method compared to the baseline SFT.
26

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS
Metries. We report Pass@1 and Pass 100 accuracy, using an exact string match between the generated arithmetic equations and the ground-truti solution. @ reflects the aceuracy that at least one valid solution is found among samples. In addition, we report Diversity in the Countdown task, measured the number of unique valid solutions discovered among 100 samples. All models are evaluated with a decoding temperature of 1.0.
D.1 MATH REASONING
Implementation Details The CoT data segmented into thought-level blocks, where each block corresponds to single sentence and is represented by 6 latent thought tokens. For VAE training, we set the 8 10- which the encoder finetuned from the backbone model while the decoder remains frozen. The flow-matching model is trained with the objective in Fg.. 4, using λFM = 5, AAns = 1, ASper = 2. During inference, we initialize the Gaussian noise of scale and apply diversity guidance with maximum scale 0.8.
Baselines We evaluate LaDiR against comprehensive suite of baselines categorized into three distinct groups:
(1) Masked Diffusion Models: We utilize LLaDA Nie al. (2025), scalable masked diffusion transformer, as the primary baseline. We evaluate both the Base Model (zero-shot) and a super- vised version (CoT SFT) fine-tuned on the same Chain-of-Thought dataset assess the impact of standard instruction tuning on diffusion architectures.
(2) Autoregressive (AR) Methods: Using LLaMA-3. 3-B-8B Dubey al. (2024) as the backbone, we compare against both standard and advanced latent reasoning approaches.
Standard SFT: We include Sol-Only SFT (trained on direct question-answer pairs) and CoT SFT (trained with full reasoning traces) to establish lower and upper bounds for standard autoregressive training.
Latent Reasoning: We compare against methods that internalize reasoning: Pause Token Goyal eet al. (2023) inserts learnable tokens to extend computation; iCoT Deng et al. (2023) gradually removes explicit reasoning steps via curriculum learning; and Coconut Hao et (2024) lever- ages continuous hidden states as latent thoughts. We also include CODI Shen et al. (2025a), which employs self-distillation to compress CoT into continuous space; Discrete Latent Su al.
(2025). which utilizes VQ-V to encode reasoming into compact discrete codes; Soft Token Butt et al. (2025), which utilizes aababilittyweightee token mixtures; and Soft Thinking Zhang et al. (2025). training-free paradigm operating in a continuous concept space. Finally, we compare with TaH+ [Fu et al. (2025), a strong supervised baseline that dynamically triggers latent iterations only for "hard" tokens using learned decider.
(3) Latent Diffusson Models: To benchmark against prior diffusion based text generation, we eval- uate LD4LG Lovelace et al. (024) and PLANNER Zhang al. (2023). which apply diffusion processes in the latent space of an encoder-decoder architecture.
Datasets We train on only the DART-MATH dataset, holding out all other benchmarks for evalu- ation only. Table [2 summarizes the datasets. While training is limited mathematical reasoning problems, our evaluations also include out-of-domain tasks such as engineering and physics, pro- viding both in-domain and out-of-domain benchmarks assess the reasoning and generalization capabilities of LaDiR and the baselines.
D.2 PUZZLE PLANNING
Implementation Details In this setting, we deliberately disable the answer generation and restrict the reasoning process to single latent block, which is compressed into fixed-size representation (4 tokens). The model is trained under a teacher-forcing regime and evaluated on decoded text tokens from VAE, thereby isolating the latent diffusion model's capacity capture planning dynamics without autoregressive supervision. During inference, we set the initial noise scale to and the maximum diversity guidance scale to 0.8.
27

---

Published as a conference paper at ICLR 2026
Dataset
# Samples
Domain / Level
Notes
DART-MATH
585k
Mixed math (train)
Syathesized for reasoning. based on GSMBK/MATH
MATH
500
High school / competition
In-domain benchmark
GSM8K
Grade school arithmetic
In-domain benchmark
College-Math
2.8k
College-level
Lineur algebra, differential equations,
DM-Math
1k
K- 12 curriculum
Out-of oo-ain generalization
OlymplaBench-Math
Olympiad-lever
Advanced competition problems
TheoremQQA
800
STEM theorem-drive driven
Math, physics, engineering Real-world test distribution
Fresh-Gaokao-Math-DD0
30
Gaokao exam
Table 12: Summary of datasets used in our experiments. We use DART-MATH (Tong et al. 2024b)
MATH (Hendrycks et al. 2021). GSM8K (Cobbe et al. 2021). College-Matt (Tang et al. 2024b),
(He et al. 2024), TheoremQA (Chen et al. 2023). and Fresh-Gaokao- Saaokao-Math--ae Math-2023 (Tang et al. 2024b).
DeepMind-Math (Saxton et al. 2019). OlympiaBench-Mii
D.3
HYPERPARAMETERS
Table 13 provides complete summary of all hyperparameters used in our experiments, covering VAE pretraining, Stage-1 teacher-forcing training, Stage- rollout training, and inference settings.
Component
Hyperparameter
Value
Latent dimension (dz)
# latent tokens per block
KL weight 8
10
Pretraining
Learning rate
x 10-5
Batch size # of Epochs
128
2
Flow- Now-matching loss weight (ArM)
CE loss weight (Aams) Special-token loss weight (ASpec)
Stage-1 Seacher-Forcing
Learning rate
x 10-6
Batch size # of Epochs
64
20
Learning rate
10 5
Stage-2 Rollout Training
Batch size # of Epochs
20
Classifier-free guidance scale
Inference
Answer token decoding temperature
0.7
Table 13: Complete hyperparameter settings for all training and inference stages,
E ADDITIONAL MODEL DETAILS
VAE Training Architecture Figure is a more in-depth diagram the training of the VAE. The encoder LLM first maps the input sequence of original text embeddings and learnalle embeddings into hidden states. These hidden states are then projected through two linear layers to produce the mean and variance of the latent distribution, from which we sample the thought tokens via the reparameterization trick. The sampled latent tokens ž1 ,2, Es are passed to the decoder I.M,, which reconstructs the original text tokens under teacher-forcing setup. This design enables the model to compress high-dimensional text into smaller set of semantically meaningful latent variables, while still maintaining faithful reconstruction of the original reasoning process.
28

---

Published as a conference paper at ICLR 2026
Original Text Tokens
Decoder
LLM
Thought Tokens
2k
(w1)) e(w2)
Teacher-Forcing
Sampling
Linear #
(22)
hu(ax)
Encoder
LLM
e(un)
e(ers)
e(ar)))
(a)
(2))
Original Text Embeddings
Learnable Embeddings
Figure 7: Detailed architecture of the variational autoencoder for latent reasoning. The encoder is a finetuned L.LM that takes both original text embeddings e(w;) and learnable embeddings eg (2.). producing mean and variance vectors through linear projections of the last hidden state h. Latent thought tokens &, are then sampled from .V [(, (²), The decoder is a frozen LLM that reconstructs the original CoT text under teacher forcing, conditioned on both the sampled thought tokens and the original text embeddings.
29


Reviews:
[{"review_id": "QGPSHkseXj", "reviewer": "Reviewer_FWxU", "summary": "This paper studies LaDiR (Latent diffusion for reasoning), where the model utilizes blocked latent thoughts generated by diffusion methods and generates the final answer conditioned on these latent thoughts. The latent thoughts used for training are generated by training a VAE (variational autoencoder), which encodes the original textual CoT to latent representations. Comparisons on numerous benchmarks with several baseline methods (including SFT+CoT and several latent reasoning methods) prove the advantages of the proposed methods.", "questions": "1. When you are generating the next block, will you remove the timestamp from the previous block?\n2. I’m a bit confused by Equation (4). According to your notation, $Z^{(<=B)}$ should be the whole thought, and $y\\_{\\leq \\tau}$ is a prefix of the final answer. Why will $y_{\\tau}$ be a $< EOT >$ token since all $< EOT >$ should appear in $Z^{(<=B)}$. Am I missing anything?\n3. In line 294, what is the definition of $x\\_t$ and $x$, and what are their shape?\n4. In line 342, should it be 1\\% instead of 2\\%? (41.8 vs 40.8)\n5. In line 344, it seems that for DM-Math, CoT outperforms LaDiR.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using diffusion methods to generate latent thoughts is a good idea.\n2. The proposed method can flexibly control both the computational budget and the diversity of trajectories.\n3. The decoder of VAE makes the latent thought interpretable.", "weaknesses": "1. For many equations, it would be better if the authors could define the function and variables more rigorously and clearly (e.g., by providing the shape of inputs or outputs, as discussed in the question section).\n2. In lines 35-38, the authors argued that “AR models generate a single linear chain of thought (CoT), which limits reasoning diversity and restricts exploration of multiple valid solutions”. However, AR models can also generate continuous CoT, which can explore multiple solutions in superposition as theoretically demonstrated by [1]. I think it’s worth discussing it or changing the wording to, e.g., “AR models with discrete CoT are limited by …”. \n3. In line 95, equation (1), I think the notation and definition of each function or variable should be made clearer and more rigorous. For example, what is $p\\_{\\theta}$, $p(\\cdot)$, and what’s the shape of their input and output? What is the shape of x and z, and what’s the distribution of $x$? \n\n**References**:\n\n[1] Zhu, Hanlin, Shibo Hao, Zhiting Hu, Jiantao Jiao, Stuart Russell, and Yuandong Tian. \"Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought.\" arXiv preprint arXiv:2505.12514 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I0QVSqFF6r", "reviewer": "Reviewer_nMv9", "summary": "This work adapts latent language diffusion to reasoning tasks. This work follows the typical pattern of first training a language autoencoder and then training a diffusion model in that continuous space. This work then trains a reasoning model which sequentially diffuses latent representations of textual thoughts before producing the final answer. They evaluate their approach for mathematical reasoning tasks and a synthetic puzzle task.", "questions": "I do not have additional questions.", "limitations": "", "rating": null, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "This work focuses on an interesting area (latent reasoning) that is gaining interest in the prior year.\n\nOne big problem with prior latent reasoning approaches (e.g. Coconut) is that it is challenging to appropriately supervise the latents. Diffusion modeling provides an elegant solution to that problem.\n\nThe authors report good results across both math reasoning tasks and a synthetic puzzle task. The visualization of the decoding process provides some nice intuition about the decoding process.", "weaknesses": "This paper cannot be accepted without properly situating itself within the literature. This work extensively discusses discrete diffusion models and recent latent reasoning work, but completely omits any discussion of the most methodologically relevant work: continuous diffusion models for language generation. Although this work falls short of claiming to introduce continuous/latent language diffusion, they also fail to acknowledge or discuss prior work that has explored this approach. A reader unfamiliar with the area would reasonably be given the impression that this approach is entirely unexplored before this current submission.\n\nWorks like Diffusion-LM, LD4LG, PLANNER, have already explored continuous diffusion for language generation across a variety of tasks, the latter two exploring latent diffusion in a very similar setup to this work. These are not obscure papers. They are highly cited papers published at top conferences 1+ years ago. Basic searches (e.g. “latent language diffusion”) return these papers as top results. In particular, the autoencoder design in the current submission seems to follow PLANNER very closely. The core technical approach (i.e. autoencoder-based latent diffusion for text) is established in prior work. This paper’s contribution is applying this approach to reasoning tasks, which is potentially valuable but must be communicated clearly. \n\nThis is not an isolated oversight. The related works section is otherwise quite comprehensive and discusses discrete diffusion and latent reasoning at length, but completely omits any mention of continuous/latent language diffusion. This gap covers the most methodologically relevant prior work.\n\nThere are various downstream weaknesses that stem from this omission. Given that this is primarily an application of an existing approach, the framing in the intro and in Figure 1 gives an uninformed reader an incorrect impression of the contribution of this work.\nAn unnecessary amount of space is given to motivating and describing the VAE design when it is almost the same as the PLANNER VAE architecture. \n\nThere are other weaknesses that can be discussed, but the primary issue with this work is the above mentioned. The combination of a complete omission of the most methodologically relevant prior work and a framing that implies novelty for an existing approach warrants rejection.\n\nThe description of the diffusion and autoregressive architecture and how precisely they are trained is unclear. The figure makes them appear to be one model, but the text uses f() and p() to denote the diffusion and autoregressive models, respectively, with different parameters. If they are one model, the training procedure is unclear given that the autoregressive model and diffusion model receive different inputs (clean vs noisy latent thoughts). Are they simply multi-tasked? Are there two separate models as the notation, but not the figure, implies?\n\nThis statement is untrue:\n“Unlike AR models that generate a single reasoning trajectory sequentially, our framework can generate multiple diverse reasoning trajectories in parallel within a batch“\nAutoregressive models are also capable of batched inference.\n\nThe modification to improve diversity seem somewhat arbitrary and are only used to drive up the PASS@100 metric for the Countdown task which of course benefits significantly from increasing diversity. The benefits of these changes are not demonstrated elsewhere.\n\nMany training and implementation details are omitted. How long were various models trained for? With what hyperparameter settings, etc?\n\nDiscussion of inference latency is not present. Inference should be much slower than the purely autoregressive baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xMCJ4X0Nss", "reviewer": "Reviewer_D4vn", "summary": "This paper presents a method for text generation via latent diffusion using a block autoregressive generation strategy. They train a beta-VAE to auto-encode text. During training they incorporate a number of ad-hoc data augmentations including adding additional noise to the auto-encoder latents, and randomly replacing a token with another token randomly chosen from the vocabulary. Diffusion training requires two stages, the first is supervised training to match target latent blocks as well as generate correct answers and special token placements. Stage 2, requires rolling out latent blocks autoregressively and supervising it with the answer, backpropogating that signal through the rollout. They also propose a guidance technique for encouraging diversity among a group of samples in a batch. They benchmark against autoregressive and discrete diffusion baselines on math datasets. They also benchmark in the countdown setting which is common for evaluating planning/reasoning ability.", "questions": "Figure 5 shows performance as a function of denoising steps for LaDiR vs. CoT, would be interesting to see wall clock times comparing the two methods. How many denoising steps can you complete in the time it takes to do CoT? Do you have a compute/time controlled comparison?", "limitations": "", "rating": null, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "* The rollout training is interesting - specifically teaching the model to determine how long to think for.\n* Consistent improvement over CoT in math benchmarks.\n* Strong improvement over LLaMA and LLaDA on Countdown Pass@1", "weaknesses": "* The introduction and related work (including the related work in the appendix) carefully side steps any discussion of prior work on latent diffusion for language (yet all other topics in related work is discussed thoroughly). The current draft is incredibly misleading (almost implying they are the first to do latent language diffusion). There is a significant body of work in place that must be discussed in this paper, as this work is a direct followup to the existing work. A few of the relevant citations include,\nZhang, Y., Gu, J., Wu, Z., Zhai, S., Susskind, J., & Jaitly, N. (2023). Planner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 36, 80178-80190.\nLovelace, J., Kishore, V., Wan, C., Shekhtman, E., & Weinberger, K. Q. (2023). Latent diffusion for language generation. Advances in Neural Information Processing Systems, 36, 56998-57025.\n* Further all experiments in this work only compare against autoregressive and discrete diffusion models, while the obvious and necessary comparison is the existing latent language diffusion works mentioned above. The contribution of this paper is not clear without the direct comparison to other latent language diffusion models.\n* The improvements on the math benchmarks are pretty marginal, and there is no time/compute comparison provided in the paper.\n* The robustness augmentations to the VAE training are ad-hoc, and they provide no ablation or study of the affects of this choice.\n* This sentence \"Unlike AR models that generate a single reasoning trajectory sequentially, our framework can generate multiple diverse reasoning trajectories in parallel within a batch.\" implies that AR models cannot do batched sampling. This is incorrect you can do batched sampling with AR models.\n* The diversity improvements are not ablated. Even more concerning they are only used in the Countdown setting. If you are going to report with diversity improvements on Countdown you need to report with and without the diversity improvements in both Countdown and the mathematics benchmarks (otherwise it comes off as a trick to improve one setting).\n* Pass@100 is a strange metric to report, Pass@1 is much more meaningful.\n* On line 301 I believe you meant to write \"... analysis in Section 4.4 provide ...\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "arGCud5JG5", "reviewer": "Reviewer_VgbG", "summary": "This paper proposes LaDiR (Latent Diffusion Reasoner), a new framework to improve the reasoning capabilities of LLMs by modeling the reasoning process in a continuous latent space using a diffusion model. The framework consists of first learning a VAE to encode the textual reasoning traces to blocks of continuous though tokens, and then a latent diffusion model to generate a sequence of blocks of latent reasoning tokens.The authors demonstrate empirically that LaDiR, using a LLaMA 3.1 8B backbone, outperforms or matches autoregressive baselines (COT and SFT) and other latent-space reasoning methods on a suite of mathematical reasoning and planning benchmarks.", "questions": "1. Could you provide a more direct comparison of the computational cost (e.g., total FLOPs or wall-clock time) between LaDiR and the baselines?\n2. The example decoded reasoning trace is surprisingly readable from Table 3 on the Countdown task. While interpretability is nice, it also seems to suggest that the latent space is still very aligned with the text space. Did you experiment with other settings that induce less interpretable latent thoughts and if so how did that impact the downstream task performance?\n3. The method adopts a one sentence per block strategy. Although it seems natural for the tasks in the paper, it might not always be the optimal choice for all the cases. It could be helpful to include some analysis on how sensitive the model is to the blockization strategy.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated and situated within the growing literature on reasoning in large language models (LLMs).\n2. The experiments provide convincing evidence of the method’s efficacy across multiple reasoning and planning tasks. Even though the gains over baselines are not always large or consistent, the method produces more interpretable reasoning traces (when decoded into text) and supports flexible trade-offs between inference compute and performance.\n3. The technical presentation is clear and well-organized, and the figures are particularly helpful in conveying key concepts.", "weaknesses": "My major concern is on the computational cost and complexity of the method. \n1. The multi-stage training pipeline — involving VAE pretraining followed by two stages of diffusion model training — is considerably much more involved than the baselines. Inference requires T denoising steps for B blocks, which is substantially slower than a single-pass autoregressive generation. The paper would be stronger if the paper would include a more direct comparison of the FLOPs against its baselines, and some discussion on how the induced overhead could be reduced in the future, as well as the difficulties of tuning the hyperparameters.\n2. All experiments are conducted on an 8B model. It would be helpful if the model could include a more explicit discussion / analysis on how the framework scales (with data and model sizes of different components).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)