Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models
Authors: Zemin Huang
Zhiyang Chen
Zijun Wang
Tiancheng Li
Guo-Jun Qi
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
diffusion language models
diffusion chain of lateral thought
Primary area: deep_learning
TL;DR: We propose DCoLT, a method to enhance diffusion language models by treating each reverse diffusion step as a latent "thinking" using reinforcement learning. Achieves promising results on several math and code metrics with SEDD and LLaDA.

Abstract: We introduce the Diffusion Chain of Lateral Thought (DCoLT), a reasoning framework for diffusion language models. DCoLT treats each intermediate step in the reverse diffusion process as a latent "thinking" action and optimizes the entire reasoning trajectory to maximize the reward on the correctness of the final answer with outcome-based Reinforcement Learning (RL). Unlike traditional Chain-of-Thought (CoT) methods that follow a causal, linear thinking process, DCoLT allows bidirectional, non-linear reasoning with no strict rule on grammatical correctness amid its intermediate steps of thought. We implement DCoLT on two representative Diffusion Language Models (DLMs). First, we choose SEDD as a representative continuous-time discrete diffusion model, where its concrete score derives a probabilistic policy to maximize the RL reward over the entire sequence of intermediate diffusion steps.  We further consider the discrete-time masked diffusion language model -- LLaDA, and find that the order to predict and unmask tokens plays an essential role to optimize its RL action resulting from the ranking-based Unmasking Policy Module (UPM) defined by the Plackett-Luce model. Experiments on both math and code generation tasks show that using only public data and 16 H800 GPUs, DCoLT-reinforced DLMs outperform other DLMs trained by SFT or RL or even both. Notably, DCoLT-reinforced LLaDA boosts its reasoning accuracy by +9.8%, +5.7%, +11.4%, +19.5% on GSM8K, MATH, MBPP, and HumanEval.

Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models
Zemin Huang 1.2
Zhiyang 2
Zijun Wang
Tiancheng Li
Guo-Jun Q12.3,4,
'Zhejiang Univeristy
MAPLE Lab, Westlake University
Matterwave Intelligence
Institute of Advanced Technology, Westlake Institute for Advanced Study
(huangzemin, hrgpaaasnnnaaaannnnnaa...aataaar edu. guojung@gmail.com
Abstract
We introduce the Diffiasion Chain of Lateral Thought (DCoLT), a reasoning frame- work for diffusion language models. DCoLT treats each intermediate step in the reverse diffusion process as a latent "thinking" action and optimizes the entire rea- soning trajectory to maximize the reward the correctness of the final answer with outcome- based Reinforcement Learning (RL). Unlike traditional Chain-of-Thouue (CoT)) methods that follow causal, linear thinking process, DCoLT allows bidirec- tional, non-linear reasoning with no strict rule on grammatical correctness amid its intermediate steps of thought. We implement DCoLT on two representative Diffusion Language Models (DLMs). First, we choose SEDD as representative continuous- discrete diffusion model, where its concrete score derives proba- bilistic policy to maximize the RL reward over the entire sequence of intermediate diffusion steps. We further consider the discrete-ting time masked diffusion language model - LLaDA, and find that the order to predict and unmask tokens plays an essential role optimize its RL action resulting from the ranking- based Unmasking Policy Module (UPM) defined by the Plackett-Luce model. Experiments on both math and code generation tasks show that using only public data and 16 H800 GPUs, Deo..Trreinforeed DLMs outperform other DLMS trained by SFT or RI. or even both. Notably, reeffoerddo LLaDA boosts its reasoning accuracy by +9.8%, +5.7%, + +1.4%, +19 +19.5% on GSM8K, MATH, MBPP, and HumanEval.
Introduction
To enable complex reasoning, most large language models (LLMs) [15] 20 learn to decompose problems into simpler sub-steps and generate intermediate reasoning natural language. Chain- of-Thought (CoT) 40 first reveals that step-by-step reasoning facilitates language models, as the outputs from previous steps could be rationales for more accurate next step prediction. Based on that, OpenAl's PRM 2 supervises these intermediate reasoning steps with progressive rewards, to ensure the correctness of each single step. More recently, DeepSeek-RR R1 [[5] eliminates the need of the reward model and verifies only the correctness of the final answer, relaxing the constraints on
"Equal contribution. 1 'Corresponding author.
This project was initiated and supported by MAPLE Lab at Westlake University.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Vertical Thinking
Htp
-..
a) Chain of Thouugh (CaT)
Pq
[1]
SearS
PM
[M]
pqi
[M
IN
PM
PM
PMI
pM
[P
PMI
PM
[N
pM
PM
PH
Lateral Thinking
PM]
PM
Abbaaa
[M]
PM
[M]
pM
PM
PM
Clowary
ywors
Ayysa
[M]
yeurs
[M]
PM]
PM]
PM
[M]
b) Diffusion Chain of Lateral Thought (DCoLiT)
Figure I: Comparison between CoT and DCoLT. (a) A typical CoT performs vertical thinking by following an auto-regressive convention that generates responses token by token from left to right in a linear way. (b) DCoLT performs lateral thinking that generates the responses in non-linear way without following the ruto-regressive order; moreover, at each step, it can generate multiple tokens at chosen positions. We focus on the lateral thinking in this paper by reinforcing the chain of such lateral thought as an entirety in Diffusion Language Models (DLMs).
the reasoning process. However, due to the causal nature of attention mechanisms, auto-regressive models are still forced to reason in single, sequential direction.
However, when developing ideas, human cognition does not always proceed through strictly sequential steps. At the beginning of thinking, human does not require intact linguistic structure. Concepts, words, or ideas emerge spontaneously and independently first, and are gradually refined and organized over time follow grammar rules. This non-linear and creative mode of reasoning, known as lateral thinking 17 contrasts with the structured, step-by-step approach of vertical thinking.
have also been adopted for text generation. The intermediate steps of the reverse diffusion process are naturally well-suited to emulate lateral thinking. Unlike auto-regressive models, diffuusion models generate all tokens in parallel from a prior distribution. Each token can attend freely to all others under a non-causal mask in self attention, and intermediate reasoning steps are not required to conform to grammatical rules during multi-step generation, thus leading to more divergent thinking. In this paper, we propose the Diffusion Chain of Lateral Thought (DCoLT)) to reinforce the lateral reasoning in diffusion language models, as shown in Fig. [ Rather than providing explicit supervision for the thinking process, we employ outcome-based reinforcement learning, offering a rule-based reward that evaluates only the correctness of the final responses. This reward encourages the model to explore diverse, creative, and non-linear thought trajectories that ultimately lead to correct answers.
Contrary to auto-regressive models, Diffusion Language Models (DLMs) 24 27 43]
We study two paradigms of diffusion language models to reinforce the DCoLT, continuous time diffusion language models 6 [24] and discrete-time diffusion language models [ 19
27 37 43
For the continuous-time paradigm, we consider SEDD (24 as a representative DLM. SEDD predicts the concrete score, allowing for closed-form expression of the predicted diffused distribution of generated tokens at each step. This distribution can be viewed as a probabilistic policy for sampling tokens, which can be trained to optimize the reward on the final answers by reinforcing DCoLT.
For the discrete-time paradigm, we consider a masked DLM - LLaDA (77. Besides the output distributions over discrete tokens, we note that the unmasking order plays an important role to decide which tokens ought to be kept to form the current step of lateral thought, and thus should be part of the learnable reasoning process. To this end, we introduce Plackett-Luce model [28 32 to define ranking-basse based unmasking policy, where each masked token is assigned a predicted ranking score, and the unmasking policy selects the top-K ranked tokens to retain in the output sequence at each diffusion step. The unmasking policy is trained together with the token generation policy to optimize the reward on the final answers.
We conduct experiments on both Math and code generation tasks to demonstrate the efficacy of DCoLT. After training the DCoLT on the SEDD 400M model, it achieves 96.2% and 57.0% in accuracy on Sudoku 4x4 and GSM8K-Aug [00 tasks. On the LLLDA 8B model, DCoLT achieves the state-of-ther the--rrtttttttttt performance among existing DLMs trained with SFT or RL even both. Using

---

only public data and 16 H800 GPUs, the DCoLT-reinforced LLaDA model achieves 88.1% on GSM8K 8 44.6% on MATH [6],, 51.6% on MBPP 4 and 59.1% on HumanEval in the challenging zero-shot setting. Even compared with auto-regressive models that are trained with significantly more proprietary data and fully annotated CoT reasoning processes, still demonstrates competitive performances.
Related Works
Diffusion Language Models Diffusion models have achieved impressive results in image modeling
[18
21
33
38 44], with recent efforts 2 B
13 24 27
29 37
43 48 44 extending to language
tasks. Depending on the types of diffused distributions over text tokens, diffusion language models
can be categorized into continuous diffusion models [13 and discrete diffusion models [ 24]
with our work focusing primarily on the latter due to the discrete nature of language.
Among discrete diffusion models, masked diffusion models 24
emerge as a promising
approach due their superior performances. Recent advances simplify model training and design by
adopting the straight cross-entropy loss [29] 37 49) and removing the time embeddings [48]. The
resultant models 27 43 achieve the state-of-the-art performance among existing diffusion language
models.
Particularly, consider two representative examples of discrete diffusion language models in this paper: SEDD 2 a discrete diffusion model with a continuous diffusion time, and LLaDA (7 masked diffusion model with discrete diffusion timesteps. We will reinforce the entire sequence of their reverse diffusion processes containing non-linear text generations to release their lateral thinking ability. In contrast, DoT (Diffusion of Thouuhtt 42 uses annotated step-by-step CoT data for supervised fine-tuning with existing diffusion losses such as the score entropy loss [24] or the noise prediction loss [13]. Thus, it still encourages conventional stepwise reasoning process in 17
vertical thinking
Reinforcement Learning for Language Models Reinforcement Learning (RL) helps language
models better align with human preference (30) verifiable knowledge [22] (e.g. rewards on
math/code tasks). Recently, there appear various algorithms for RL optimization [ 25 35 [6). These methods enhance model's ability produce aligned outputs and show strong potential in inference-time scaling for challenging problems [15 20].
However, existing RL approaches primarily applied regressgeee language models [15] 30]. For DLMs, previous works explored policy gradient methods within the concrete score-maan matching framework 46), estimated the log-probability of the sampled responses perform reinforcement learning 47 However, both models merely focused on the generated text response in the final action to optimize, ignoring the role of intermediate diffusion steps as lateral thought to reinforce amid the reverse diffusion process.
diffusion step
Instead, in this paper, we seek to explicitly reinforce the reverse diffusion process to enable the training of lateral reasoning. We demonstrate experiments that even though these intermediate diffusion steps perform non-linear rather than step-by-step reasoning, reinforcing them eventually lead to correct answers at the final step that outperforms many CoT models [10 40].
3 Methods
3.1 Diffusion Chain of Lateral Thought
To model the probability distribution Pdata over finite vocabulary V= [I....,
.,......, V) for text generation tasks, discrete diffusion processes model how the unknown data distribution Paau t 0 gradually evolves into a prior distribution Pprior at f = T 6 The distribution at an intermediate diffusion time t is denoted by Pt-
Then a generation process is realized by reversing this diffusion process. Practically, it begins by sampling zo from the prior Parior and then iteratively denoising through sequence of diffusion time lo.v- each step n, the model estimates the diffused distribution pe.t. time 4, from which an intermediate sample Tn is drawn, shown in Eq. I The diffusion time t, decreases progressively as

---

increases. After completing all denoising steps, reaching EN = O, the final response NN is obtained.
In Po,t. (-1x
(I)
This naturally generates series of intermediate results 20:N-1 before arriving at the final output IN. This behavior is analogous to the Chain-of-Thought (CoT) technique [40]. However, unlike the vertical and causal structure of CoT, the diffusion process enables the model to produce intermediate content that facilitates reaching final answers, aligning more closely with the concept of lateral thinking Soinking-ollving problems through indirect and creative approaches. Thus, we define the entire sequence of all intermediate steps Tu:N as the Difftusion Chain of Lateral Thought (DCoLT), as shown in Fig. and seek to apply reinforcement learning train it based the given reward function.
Compared to standard COT, DCoLT distinguished with several notable features:
Bidirectional Reasoning: CoT generates tokens sequentially in causal, irreversible manner; once derived, earlier thought remains fixed in the context. Instead, DCoLT adopts a bidirectional structure: each token is influenced by both preceding and succeeding content with bidirectional self-attention masks, allowing global refinement throughout generations.
CoT typically adheres strictly to natural language format with complete linguistic structures. DCoLT, however, relaxes this constraint, allowing early stage of intermediate steps not necessarily to be complete correct in format, thus enabling more divergent and creative patterns of thought before finally converging to complete responses.
Format-Free Reasoning:
Nonlinear Generation: CoT generates tokens one-by-one linearly in an auto-regressive manner from left to right. In contrast, DCoLT allows nonlinear generation of tokens at various positions. This aligns with how human develops ideas by beginning with keywords or critical points and then refining the details around them.
Fig. 7 Fig.
Appendix B show some examples of intermediate responses from DCoL.T-riifforeed DLMs, demonstrating the above features with bidirectional and nonlinear generations throughout reverse diffusion processes from incomplete masked phases complete unmasked responses.
To enhance the lateral thinking, we adopt final-outcome rewarded reinforcement learning to train the model, promoting reasoning trajectories that can lead to correct final responses zv. For this, we treat the generation of T1.N as a sequence of actions to optimize as a whole. At each denoising step n, a distribution (-1x(-)1,-1)) 1) over possible outputs is defined, which serves as the policy for sampling In to calculate action probabilities during RL training. The reward r is assigned based on verifiable correctness of final results. We do not impose any explicit supervision promoting the model's lateral thinking ability to explore diverse
the intermediate steps, d reasoning strategies to maximize the reward. This could yield useful patterns of thinking processes for example, shown in Fig. 3ff Appendix A an easy-to-hard progressive generation of responses emerges from the DCoLT-trained DLM, in contrast to its counterpart supervisedly trained with the conventional diffusion loss.
We summarize the algorithm in Alg. [ Details about training specific diffusion language models such as SEDD (24] and LLaDA (27] will be claborated on in the following two subsections. Considering the demanding computation graph that cxpands multi-step generations in memory, we back-propagate the gradients each step and accumulate them after the whole reverse diffusion process (c.f. Line 26 and 28 in Alg. []. We use the GRPO [36] for fair comparisons with other RL-based models [36] 47), while alternative RL training approaches such PPO 35 and RLOO can also be adopted.
3.2 A Continuous- Time DLM Case: DCoLT-reinforeed SEDD
To define discrete diffusion process, continuous-time diffusion language models such SEDD [24] evolve a family of distributions pt according to a continuous-time Markov process, which can be represented by the following linear ordinary differential equation.
dpe di QePts
Po
Paatas
Pr
Pprior
Here, Q: is the transition rate matrix defining the forward process. We may first consider the simple single-token case (Qt e VVV), Conversely, to generate a sample from Pprioes this process has

---

Algorithm A General Framework for Training DCoLT
Require: Model parameters a dataset D, and reward_
I: while not converged and maximum epochs not reached do
2: Sample questions D
3: for g to G do Initialize zgg with and mask tokens.
Generate group of G trajectories
5: for n 1 to N do
> N denotes the number of denoising steps
6:
if training SEDD then
7:
Sample > ( (+1=%-1)
8:
else if training L.LaDA then
Calculate the ranking score he.n for each token
10:
Sample K tokens to unmask in this step: Uh ~ Plackett-Luucche,s, K)
11:
Sample 2 (-.0(-122-11
Vi E Un
12:
end if end for nd
13:
14:
reward Funcly, x end for
D Compute the rewards
15:
16:
for g to G do -mean(r4:)) A tG(
Compute the advantages
17:
18:
end for
19:
for n 1 to N do
b. Compute my and losses for each denoising step
20:
if training SEDD then
21:
(x. 10 - I1 po. 1 (a
Eq,
22:
else training LLaDA then
23:
To 1 end
T0.
smman
(2 (xn)
token (xn (onl
see Eq
24:
Ta,n in 2
25:
=
19
Ta (ar
26:
Calculate the gradient VoLo.
27:
end for
28:
Update with accumulated gradients ΣN end while
VoLo,, along the descent direction
29:
corresponding reverse process, defined by a reverse transition rate matrix Qr:
dpr-t QT Q((y,)) p((y) pr(x)
Qc(2,v), v), Qr(2,x)=-
dt
By Euler's method, we have the transition probability Pt., at each diffusion step fn. resulting in an iterative formula for multi-step generations from to T to IN 0 with 0, N,
Pt. (xn y|an-1 2) day + (fn-1 t... (y.
(4)
with 2n denoting 2,, to avoid notational clutter, i.c., the diffused sample 2, t tn.
Specifically, we consider representative discrete diffusion model, SEDD 24 This model learns to approximate the concrete score, i.e. s0(x, t)y P3 Pr(y) for any y + x to represent the probability to p.(w) transfer other tokens. Thus, may replace Qt,,, (y, æ:) with the model-estimated s0 in-1)y Qtn. (x. y) in Eq. 4 making the transition probability tractable.
When scaling to sequences, may apply 7-leaping to define the probability of action at each timestep as the product transition probabilities across all tokens:
no,n (2
po.t. (nr), (nn-1)
(5)
Pa.t. (2 (In-1s 1, in 1)i,es (tn-1 tn) Qin-n (2h-11, Σ Egent 80 20( 1) (1n-1 (n) Qt
x/ #
x²
1))
x
x
where |x,/| denotes the token length the sequence Ty Note that the predicted pt,, dependent on the model parameters 0. During training, we may update 0 to reinforce suppress the probability of
5

---

Xn
The
unswer
[M]
B
sample top-K ranking
U, Plackett Lucce (bg
sample x VI Uhe
A5
The unmesk set
Difiusion Emhhokinng (n)
Unmask Policy Module (UPM)
Originaa menimnenettss LLaDA
Adaptive LayerNorm
Transformer Block
Token Generation Head
Adaptive LayerNorm
LLaDA Blocks
Xn-1
The
[M]
Figure 2: The structure of LLaDOU. It first predicts the token set to unmask Un according to the ranking score he. by the UPM, and then samples those unmasked tokens in Un by LLaDA blocks.
all actions along trajectory of thoughts [To, Ji,..., .... vN) according to the reward associated with each completion. Unlike traditional diffusion model training. which typically optimizes each individual denoising step independently with ground truth responses, DCoLT jointly optimizes the entire reverse process thought from to to tn = promoting the emergence of lateral thought leading to correct answers.
3.3 A Discrete-Time DLM Case: DCoLT-reinforced LLaDA
They can be viewed as Discrete-Time Diffusion Language Models with a multi-step generation process. For these models, we will figure out the probability of actions for each discrete step.
Some other diffusion language models instead operate at discrete timesteps [27
43
Among them is LLaDA (77 which is typical Eiscrete-Time Masked Diffusion Language Model. generates text by progressively unmasking sequence initially prefilled entirely with mask tokens. At each generation step, the model receives a partially masked sequence as input. Then, it chooses and predicts subset of masked tokens to reveal as clean text. As it progresses, the number of masked tokens gradually decreases, so that the model ultimately yields fully generated sequence.
define the model's action at each step in two parts: I) determining the set Uh of tokens to unmask, and 2) predicting the values of these tokens to obtain the new sequence I, over the unmasked part 24,
Specifically,
The action of selecting which tokens to unmask each step can be decided by ranking masked tokens with score function under the current state. To this end, we introduce an Unmask Policy Module (UPM), which predicts score value hi,, for each token at current diffusion step n. Based on these scores, we define policy to sample top-K ranked list 24, = [u, .... un (K) from a Plackett-Luce model (28 332 multinomial distribution is formed from the predicted scores, and tokens are sequentially sampled without replacement, such that the corresponding scores are (K) with high probability.
non-increasingly ordered 11,m (1) > 2 10,m
Formally, let M,, denote the set of tokens that remain masked after the n-th step, i.e., Mn-1 \ Unn Then, the probability of sampling a specific ranking list Ur is given by:
K
exp (hp (k)(
unmask (Un (n-1)) ng,n
II 1=1 Σ exp(h" (h((t)) + EjEM. exp(h
Specifically, UPM takes the hidden states from the last hidden layer inputs at each denoising step and predicts a ranking score hi,,, for i-th token. It contains only one transformer block, introducing marginal computation cost. Moreover, the step index n and the mask indicator of each token In can benefit the model training. Thus, we embed both of them into the Unmask Policy Module (UPM) via adaptive layer normalization. The module structure shown Fig. 2 For convenience, we name the DCoLT-trained LLaDA with UPM by LLaDOU (LLaDA with Ordered Unmasking).

---

As shown in Fig. of Appendix B the learned scores ha,n can be viewed as the ranked confidences over the token predictions at the current step n. Higher scores indicate that the UPM predicts lower levels of diffusion noises may be present in the generated tokens, which are less likely to contain errors and thus could be unmasked with higher priorities at the current step. Fig. 7 shows the resulting unmasking orders from the learned ranking scores for some prompts.
Once the unmask set Un determined, the model predicts their token values based on the output distribution by LLaDA blocks over the vocabulary. Viewing this prediction as a second-stage action, the probability of generating In given In-1 and Un can be written as
loken (In [Tn 1, = II po. (o, 88,
(
Thus, the probability of the complete policy for transitioning from Øn-1 to dn is the product of those of the unmask policy and the token prediction policy used in Line 23 of Alg.
nan (2 1n-1 = Tgyn (L4, (2n-))
-x 20,0 =0,n (x (eml2nn. (2n Un.
(9)
Experiments
We conduct DCoLT experiments two DLMs DLMD 400M and LLaDA 8B, each representing one of the two categories mentioned in Sec. 3
4.1
Experiments on SEDD+DCoLT
4.1.1 Settings
We first conduct experiments on SEDD 24 for two different tasks: 1) Sudoku 4 x 4 task and 2) GSMSK-Aug math problems [TO), both assessing math reasoning performance. These two tasks allow us to perform direct comparisons with existing reasoning algorithms, including both supervised fine-tuning [10 42 and reinforcement learning approaches [36]. For all experiments, we use the rule-based reward function to compute the reward for each sample: reward of 1 is assigned if the solution correct, and 0 otherwise. We choose SEDD as the hase model to conduct DCoLT training,
which is of medium size around 400 M. For detailed settings please refer to Appendix C.1
4.1.2 Results
From the results in Tab. our method remarkably outperforms both (implicit) CoT 10 and DoT [42 on both tasks. While both CoT and DoT perform vertical thinking, DoT also uses the SEDD as its base model simulate causal reasoning. In contrast, our approach only uses RL train the model without any CoT data for supervised training. For CoT, we adopt an auto-regressive language model - GPT2 31 the base model, which has the similar model size of about 400M the SEDD model.
Particularly, SEDD DCoLT achieves an accuracy of 96.2% on Sudoku 4 x 4 task, the best performance among all the methods. Using the same SEDD model, DCoLT greatly exceeds DoT (79.4%), even though the latter introduce step-by-step annotated CoT data for supervised training. For GPT2 model, both CoT and Implicit COT,, matter if being post-trained with SFT or RI., perform worse than our model clearly. On the GSM8K-Aus dataset, SEDD + DCoLT also reaches 57.0% in accuracy, which is notably higher than both DoT and CoT counterparts. We will further analyze the generation process SEDD DCoLT in Appendix A demonstrating how tokens are generated in a lateral thinking manner.
4.2 Experiments on LLaDOU (LLaDA+DColTT)
4.2.1 Settings
We further apply DCoLT to LLaDA 27 a SOTA discrete-time masked-based DLM with 8B parameters. During generation, it starts with masked sequence of length 256, and performs 256-step reverse diffusion process. We adopt a block-wise unmasking strategy as LLaDA (27). and divide the sequence into blocks of length 8. In Tab. 2 both LLaDA and LLADOU are evaluated with this setting for fair comparison. We conduct experiments with 16 H800 GPUs to jointly train both UPM and LLaDA parts. For implementation details, please refer to Appendix C2
7

---

Table
Experimental Results on SEDD. All models listed below are of medium size, with approx- imately 400M parameters. On the Sudoku 4 x 4 dataset, we report the accuracy on test set of 2,000 samples each with empty cells. On GSM8K-Aug Aug, we use the test set from GSM8K 8 for evaluation following [10]
Models
Post- Training
Sudoku 4x4
GSM8K-Aug
GPT2 CoT
SFT
71.5
43.9
GPT2 Implicit CIT
SFT
21.9
GPT2 CoT
RL
74.6
SEDD + DoT 42
SFT
79.4
53.5
SEDD + DCoLT
RL
96.2
57.0
Table Model accuracies on math and code generation benchmarks. The "Post- -osst-Tainnng" column indicates what kinds of post-training (SFT or RL) phase the model goes through. 'indicates the model uses additional proprietary training data for the post-training phase. The numbers in parentheses represent the number of shots for the in-context learning, with "_" indicating unknown cases not mentioned in original papers. The results denoted with are evaluated with the prompt templates in Appendix C.2 for fair comparison, while others reported in original papers. The results denoted with are evaluated on subset MATH-500 instead of MATH. We highlight the best rest-performing model among compared DLMS in bold.
Math
Code
Method
Post- Dost-Training
GSM8K
MATH
HumanEval
MBPP
Diffusion Language
Models
Dream 7B
[43
baseline
81.1 (0)
42.9 (0)
51.8 (0)
49.6 (0)
LLaDA 8B
22
baseline
78.3 (0)
38)
39.6 (0)
40.2 (0)
+ SFT
SFT
81.1 (0)
34.8* (0)
+ diffu-GRPO 47
RL
81)
39.2 (0)
dI-L.LaDA (SFT+ diffu-GRPO)
SFT + RL.
82.1 (0)
40.2 (0)
LLaDOU (LLaDA + DCoLT) 8B"
RL
88.1 (0)
44.6 (0)
59.1 (0)
51.6 (0)
regoreegressive
Models
LLaMA2
baseline
14.6 (0)
20)
12.8 (0)
(3)
MetaMath 7B 445
+FTT
66.5 (0)
19.8 (0)
Ladel.LaMAAInm 7B
34
+ SFT
34.8 (0)
44.4 (3)
Deepseek 7B [5]
baseline
(3)
(0)
48.2 (0)
33)
DeepseckMatt-Insttue 7B 36
SFTT
82.9 (-)
46)8
DeepseekMath-RI. 7B 36
+ SFT + RL
8-)
51.7
DeopseckCoder-Instrre
SFT
66.1 (-)
65.4
4.2.2 Results
As shown in Tab. among all compared DLMs, LLaDOU consistently achieves the best performance across all benchmarks. On GSM8K, LLaDOU reaches 88.1%, 1%, significantly higher than other methods. On the more challenging MATH dataset, LLaDOU achieves 44.6%, outperforming baseline models such as LLaDA 8B (+5.7%) and Dream 7B (+1.7%), well as models enhanced with post-training by SFT, RL or both. Notably, d1-LLaDA 47 uses 1K questions paired with detailed reasoning traces (26) in SFT to boost reasoning capabilities. In contrast, LLaDOU achieves superior performance +6.0% on GSMSK and +4.4% on MATH without any reasoning supervision, relying solely on reward signals based on the final answers.
Even when compared with auto-regressive models trained with a significantly larger amount of proprictary data, LLaDOU remains highly competitive. For example, MeepseekMath (36) gathers 776K questions with reasoning process annotations for SFT and 144K questions for RL, while LLaDOU is trained on just 15K public training samples from GSM8K and MATH, Despite using nearly two orders of magnitude fewer data for RL training, LLaDOU matches DeepseekMath' performance on GSM8K (88.1% (8).1%), highlighting its data efficiency and the effectiveness of reward- driven reasoning via diffusion-bact of the thinking process generated by LLaDOU, including both visualization examples and quantitative evaluations. Fig. illustrates the reward curve during DCoLT training in the math domain, showing an increasing trend in the outcome- based reward over training iterations.
lateral reasoning. Appendix B we provide a detailed analysis

---

Table Accuracy LLaDOU on MATH subsets across difficulty levels (5: the hardest and 1: the easiest) and generation lengths. The result at L = 512 is obtained from the model fine-tuned with this length, while results at other lengths are directly evaluated using LLaDOU trained with 256.
Length
Level 1
Level 2
Level 3
Level 4
Level 5
128
80.8%
61.0%
45.6%
29.5%
13.2%
256
83.3%
65.1%
52.3%
35.9%
18.7%
384
82.4%
66.6%
54.9%
39.0%
20.2%
512
82.6%
69.7%
56.9%
40.2%
21.5%
Similar trends are observed in code generation benchmarks. Unlike models that rely on valuable ground-truth code for supervised training. LLADOU requires no code for supervised training at all. Instead, it only needs several test cases to provide simple outcome-based reward for reinforcement learning: The model is rewarded only if the generated code passes all unit tests. Despite this code-free reward signal, LLaDOU achieves 51.6% on MBPP and 59.1 % on HumanEval, outperforming other DLMs and auto-regressive models, except for DeepseckCoder [T4] that benefits from 2B tokens of high-quality instruction-tuning cata-approximatee 150x more than we used (about 12M tokens). Details of post-training datasets used by all compared methods are provided in the Appendix
4.2.3 Ablation Studies
We provide some ablation studies reveal the role of some model design and hyper-parameter settings in LLaDOU. Unless otherwise specified, all experiments are conducted on LLaDOU with N 64 diffuusion steps, and the model is trained for 150 iterations with a batch size of 32.
Better performance with the UPM In Tab. we ablate the model design in LLaDOU. Even we freeze the original model parameters in LLaDA part and only train the UPM part, it achieves a significant improvement from 47.27% to 69.24%. This result indicates that the unmasking policy plays a crucial role in our model. By train- ing the LLaDA part with RL together, the accuracy further improves to 81.06%.
Table 4: Ablation on the Unmasking Policy Module
Trained parameters
GSM8K Acc.
UPM
LLaDA
47.27 (Baseline)
w/ AdaLN
69.24
w/o AdaLN
V
80.53
w/ AdaLN
81.06
We use the adaptive layernorm in the UPM. Removing it degrades accuracy to 80.53%. This demonstrates that incorporating the embeddings of diffusion step and the mask indicators also benefits the training of the unmasking policy.
Extension to longer generation length Though the models in Tab. 2 is trained to generate comple- tions with a fixed generation length 256 (i.e., the length of initialized mask tokens), it can benefit from generating longer sequences without further RL training on longer generations. This especially effective when answering difficult questions on MATH dataset, in which solving problems sometimes require longer reasoning. Tab. further summarizes the overall results under different generation lengths and models. It suggests that longer generations in inference lead to better accuracies. For example, a generation length of 384 achieves an accuracy of 46.22%, which is +1 +1.62% higher than the default length of 256. This improvement vanishes However, if we further tune the model on a longer generation length, for example = 512, the accuracy further improves to 47.3%.
further increase the generation length.
In Tab. 3 we report the accuracy of LLaDOU on MATH subsets across difficulty levels. The accuracy on level-1 problems saturates with length 256, suggesting that shorter generations are sufficient for simpler questions. In contrast, the performance on harder problems (levels 2-5) continues to improve with longer generations, indicating that complex reasoning benefits from longer responses. These results suggest the potential of how scaled generation lengths may improve the performance of DCoLT- trained DLMs, as shown in Fig. 13 of Appendix D

---

Table Model accuracies with extended generation lengths on MATH without further RL-finetuning the model on these longer generations. The results denoted with are taken from 47 and evaluated on a subset MATH-500 instead of MATH.
Generation Length
Model
256
320
384
448
512
LLaDA 8B 227
38,9
40.1
41.5
42.3
42.5
+ SFT 47
38.6
40.2
diffu-GRPO" 47
37.2
39.2
dI-LLaDA (FPP
47
38.6
40.2
LLaDOU (LLaDD + DCoLT)
44.6
45.7
46.2
45.7
45.9
Conclusion
In this paper, we propose new paradigm of model reasoning ability, Diffusion Chain of Lateral Thought (DCoLT), for diffusion language models. DCoLT considers the intermediate steps in the reverse diffusion process as the thinking actions, and optimizes this process with final-outcome rewarded Reinforcement Learning (RL). We implement DCoLT on two representative diffusion language models, SEDD and LLaDA. On SEDD, derive the corresponding RL optimization over the distribution chain of thinking actions via the predicted concrete scores at denoising steps. On the masked-basee diffusion language model LLaDA, we regard the order in which tokens are unmasked at each diffusion step as its action, and propose an Unmask Policy Module (UPM) to optimize over the order distribution with the Plackett-Luce model. The experiments indicate that DCoLT outperforms other SFT- or RL-based training algorithms,, and demonstrate its effectiveness on wide range of tasks and benchmarks, including math and code generation.
Limitations
First, due to limited training data and compute, our model's performances on Math and code generation tasks still have much rooms improve. Prior works [4 that proprietary data often significantly improves the model performances. Also, scaling the token length of sequences during training can also enhance reasoning. We will seek to boost the model ability along these directions when computing resources become available. Second, currently DCoLT only validated on tasks having verifiable reward function. We may need reward model cover more general tasks. We will develop these further in our future research.
34
36 45)
demonstrate
Acknowledgements
This work supported by National Natural Science Foundation of China under Grant No. 92467104 and Zhejiang Leading Innovative and Entrepreneur Team Introduction Program (2024R01007).
References
[1] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting reinforce style optimization for learning from human feedback in 1lms. arXiv preprint arXiv:2402. 14740 2024.
[2] Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.0957. 2025.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces Advances in neural information processing systems, 34:17981 11779981199991111111111199199999 2021.
[4] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint 1rXiv:2108.07732. 2021.
[5] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek IIm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. 2024.
10

---

[6] Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. contimuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-282777 2022.
[7] Mark Chen, Jenry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint www:::::2070003330 2021.
[8] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakamo, et al. Training verifiers to solve math word problems, 2021. URL https://arxiv. org/abs/2110.141666 9, 2021.
[9] DeepSeek-Ar AL. Deepseek-v3 technical report, 2024. URL httph:ttttppttitio hwt.:/oggggabb////ee 2412
19437
[10] Yuntian Deng, Kiran Prasad, Roland Fermandez, Paul Smolensky, Vishrav Chaudhary, and Stuart Shieber. Implicit chain of thought reasoning via knowledge distillation. arXiv preprint arXiv:2311 110223311.1160.. 2023.
[11] Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. Advances Neural Information Processing Systems, 37:133345-13336 2024.
[12] Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, et al. Scaling diffusion language models via adaptation from autoregressive models. arviv preprint arXiv:2410.1783 17.19191,, 2024.
[13] Ishaan Gulrajani and Tatsunori Hashimoto. Likelihood based diffusion language models. Advances in Neural Information Processing Systems, 36: 16:16693-16715 2023.
[14] Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:240..14196 2024.
[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
[16] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
[17] James S Hernandez and MBBS Prathibha Varkey. Vertical versus lateral thinking. Physician executive, 34(3):26, 2008.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral information processing systems, 33:6840-6851, 2020.
[19] Emiel Hoogehoom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in neural information processing systems, 14:12454-12465 2021.
[20] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai o1 system card, arXiv preprint arXiv:2412.1677 2024.
[21] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35: 26565-26577, 2022.
[22] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning, arXiv preprint 44Xiv:2409, 12917, 2024.
11

---

[23] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever. and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[24] Aaron Lou, Chenlin Meng. and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024.
[25] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122233 2024.
[26] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
[27] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502.09992, 2025.
[28] Shuzi Niu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. A new probabilistic model for top-k ranking problem. In Proceedings of the 2 ACM international conference on Information and knowledge management, pages 2519-2522 2012.
[29] Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.037388 2024.
[30] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[31] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Iya Sutskever. Language models are unsupervised multitask learners. 2019.
[32] Stephen Ragain and Johan Ugander. Choosing to rank. arXiv preprint arXiv:1809.0523 2018.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/VV conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[34] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
[35] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal 2017.
policy optimization algorithms. arXiv preprint
[36] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
[37] Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing 2024.
systems,
[38] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, arXiv preprint arXiv:2011. 13466, 2020.
[39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
12

---

[40] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of- offtthhtht prompting elicits reasoning in large language models. Advances in neural information processing systems, 15:24824-248877 2022.
[41] Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A di- verse, challenging, and verifiable synthetic dataset for coding. arXiv preprint arXiv:2503.02951/ 2025.
[42] Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Xin Jiang. Zhenguo Li, Wei Bi, et al. Diffusion of thought: Chain-of-thought reasoning in diffusion language models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[43] Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL httttps://kkunpp: gi thub httpoo/o/o//000 2025/ dream
[44] Zilyu Ye, Zhiyang Chen, Tiancheng Li, Zemin Huang, Weijian Luo, and Guo-Jun Qi. Schedule on the fly: Diffusion time prediction for faster and better image generation, June 2025. URL
org/ ab1/ 01243
[45] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengyyng Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023.
[46] Oussama Zekri and Nicolas Boullé. Fine-tuning discrete diffusion models with policy gradient methods. arXiv preprint arXiv:2502.01384, 2025.
[47] Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. dl: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504. /2216, 2025.
[48] Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint wvXiv:2409.02908, 2024.
[49] Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation, arXiv preprint arXiv:2302.05533, 2023.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The proposed Diffusion Chain of Lateral Thought is introduce in Section 3 and the results highlighted in the abstract and introduction can be found in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We have discussed the limitations in Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
will release model later. Justification: Please follow Appendix C for both train and evaluation details. Moreover, we
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release the code later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
sosooddSSsmmsssioooooocc for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//ipps.cc/publi///uuu
Codeeuumiiii wettsbbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Please refer to Appendix
C
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification; Due the high computational cost of training large language models, we omit error bars in our reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Appendix C for the computer resources we use.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips https//ccccccctttcccccccicccccce
Answer: [Yes]
Justification: The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets in the submission phase.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: The paper describe the CoT dataset created by LLM in Appendix C.I
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.LM for what should or should be described.
A Analysis of Thinking Process in SEDD+DCoLT
For the Sudoku dataset, aim to visualize the thinking process by analyzing the generation step of different cells. We categorize all 4 x 4 cells into three groups: (1) given cells those initially provided by the puzzle; (2) easy cells those that can be directly inferred using basic Sudoku rules (each number 1-4 must appear exactly once per row, column, and 2 x 2 subgrid); and (3) hard cells the remaining positions that require more complex reasoning, often involving the resolution of dependencies among other non-given cells.
In Fig. 3 we plot the distribution of generation step for these three cell types, the left side displays statisties for the SEDD SFT model, while the right side shows our further R[-trained SEDD + DCoLT model. The right figure reveals that the our SEDD DCoLT significantly prioritizes
20

---

generating easy cells in earlier steps than hard cells, as these cells can be directly determined from the initial given cells, indicating that our model learns progressive generation strategy from easy to hard. Notably, in the left figure, we observe almost no difference between generation orders across various difficulty levels for SEDI) + SFT model. This comparison reveals that the DCoLT changes its SFF-traiied counterpart's reasoning behavior to more progressive generation aligned naturally with how humankind handles problems in a real world.
Cell Generat on Steas Distr bution
Cell Generation Steps Distribution
1800
1800
civen-cols (zvg=10.5)
oiven cels laog- di hard cells (avg-9.0)
hard co In1g-21.1
1400
calla (avq-11..)
1400
calls (.vq-6.))
1200
1000
Counta
1000
633
000
203
200
10
25
30
15
20
25
30
Steps
Steps
Figure 3: This figure shows the model's inference patterns on Sudoku dataset, with the left side displaying the pattern of SEDD + SFT model and the right side showing that of our SEDD + DCoLT model. We plot the total number of generated tokens for these three cell types over diffusion steps on Sudoku 4 x 4 dataset, with the number of sampling steps set to 32.
To further demonstrates the progressive generation behavior on sample level, we visualize the generation process of our SEDD DCoLT on the Sudoku dataset in Fig. 4 We show the predicted to at each timestep, obtained by selecting the token with the highest probability (excluding the mask token) at each token position. As shown in the top row, positions (0, 0) and (0, 1) can be regarded as hard cells, because the value can't be directly inferred from given cells denoted by black borders. Initially, our model assigns the highest probabilities to incorrect answers in these red cells due to insufficient contextual information. However, as the denoising process progresses, the model gradually refines its predictions by leveraging information from newly unmasked cells (shown in dark blue), eventually converging to the correct values. This demonstrates the importance of nonlinear generation for Sudoku 4 x tasks.
3
4
3
1
4
3
2
2
1
2
4
3
3
2
4
3
2
4
3
1
2
4
3
2
4
4
2
I
4
4
2
1
3
4
3
1
2
3
4
3
4
2
3
4
2
3
1
2
3
4
2
3
4
1
4
3
4
3
2
2
3
4
1
2
3
3
3
4
2
2
2
Figure 4: Visualization of predicted tokens t by SEDD + DCoLT on Sudoku 4 x 4: those that are still masked appear in light blue, with unmasked ones dark blue, incorrect predictions in red, and corrected ones in green. Black borders indicate the given cells.
For GSM8K-Aug dataset, since it's hard to determine which token position is more difficult to predict. we only show the generation process and the generation step statistics for each position. Fig. 5 demonstrates the contrasting generation behaviors between SEDD + DCoLT and GPT2 + CoT on the GSM8K-Aug dataset. While GPT2 + CoT consistently follows left-to-right generation order (where earlier generation steps correspond to lower token positions), SEDD DCoLT shows minimal positional bias in generation order across token positions. Furthermore, results in Fig. reveal
21

---

DCoL7
60
CoT
Average Generatin Step
30
50
60
Token Position
Figure 5: Average generation step for each token position in GSM8K-Aug. Different from CoT, SEDD DCoLT generates in a non-linear way.
Quettio:: drenkaading flc.. Normally the
chicken
of
thruugh from lead take dow sloud tik??
eark, same bores 598. How many baxes pizza did Maric order if Frecc bas casts SR 80?
real
Answer:
Auswrr
De oxder www.ttt0eiiiaeo octio dietito
oides 1800 oc1,50 10550 goo 0012-15-64 oodts 18t0 soggis 2,20 AMM
0x200/2-4 4 200--0014 1000o ex2ge20 aloo o0186-40- Tioo 1400 TO
ext2 3012-15-15---21 2--55-622300
exdr apestoo 200 1200012000 oxdo teiioo ect 20000000 A4N 120
oct2-1596
oot 7888
oct00-40 Bioo on 20000.01 1i0 o02013 oc120+49 1dlto Woed
doo 0012-1146 Also oode
codea- 1800 ooant Looeee wot 1546-A3- Ah octs she this ood dedoo
codsy- 1Bo- o0e 0412-1546 M00 ood 33 Thi 78855 5- tho c00e
Figure 6: Our model's generation process on GSM8K-AUE -ug, each line displaying the predicted- to tokens across different steps, with masked tokens shown in gray and unmasked ones shown in black.
that SEDD DCoLT's generation order is sample-dependes maintaining nonlinear generation throughout the process.
B Analysis of Thinking Process in LLaDOU (LLaDA+DCoLT)
We illustrate how LLaDOU generates a 64-token answer to a GSM8K question in Fig. 7 Tokens shown in darker shades are generated in later diffusion steps. As the figure shows, key numbers and symbols tend to emerge early in the generation process, while surrounding textual elements are filled in later ensure grammatical correctness and fluency.
Fig. S and Fig. further show intermediate steps of the output when generating responses of length 256 with 64 steps, in which key numbers and symbols are generated to gradually form the structure of thinking processes before grammatically correct sentences are completed.
Fig. 10 visualizes the predicted ranking scores he to unmask tokens during the generation process. The examples show that at each step, the unmasking score evaluates how likely the predicted token correct at each position. For example, in Step 20 of case (a), the token "3" prior to the selected token "0" has smaller value the unmasking score, suggesting that it more likely than "0" to be wrong. This token is corrected to "0" before being unmasked at later step.
22

---

ig
uel
uses
2
6
sheets
of
paper
per
Miguel
will
4
sheets
of
every
month
Here
the
solution
4
</eot_id]>
<jendoftext]>
(a) Miguel uses 2 pads of paper week for his drawing, If there are 30 sheets of paper on a pad of paper, how many shects of paper does he use every month?
travelss
miles
north
each
train
travels
total
0
2
3
miles
Therefore
each
traiin
covers
3
miles
the
two
days
Result
3
<Jent ld|
<|jnddoftexti>
(b) Two trains leave Rafael at the same time. They begin traveling westward, traveling miles. The next day, they travel northwards, covering 150 miles. What's 's the distance covered by each train in the two days?
She
pays
pays
0
0
-
3
for
Insurance
That
means
In
total
she
cald
1 3
0
0
3
-5
1
The
answer
4
o
<leot_idjp Id|>
<jendoftext|>
(c) Janet buys brooch for her daughter: She pays $500 for the material to make it and then another $800 for the jeweler construct it. After that, she pays 10% of that get it insured. How much did she pay?
child
elready
savings
save
add tional
2
6
promises
2
daily then it
will take
him
$
2
days
to
save
the
remaining rinaining
amount
money
Solution
<jeot_id] jdj>
<|endoftext]>
(d) An 8 y-year old child wants to buy toy car which costs $12. He already has savings. How many days will it take him to save the remaining amount of money if he promises to save $2 daily from his allowance?
Figure 7: Examples of 64-token responses generated by LLaDOU on GSM8K. Token color reflects the generation order, progressing from light (early steps) to dark (later steps). The corresponding questions are provided in the sub-captions. We observe from the results that many tokens bearing key information to final answers such as the numbers, units and factual entities tend to be unmasked at earlier steps, while those tokens for function words and symbols such as "of" and "the" tend to be unmasked later steps.
23

---

Step
kalo Bloo
"Sep kemty-root smoce
The ribe be
Deamew aPRUUS
cruut--cruttoo poo cnasib-puoee cean too pac--r coco-onaaaae www.soo----aaao wou.b-deeeo www.sa-----aaaa www.-----------o-e crusto cruu do so- www.soiisassaa th-di auco-it medii deuueeeennnn
eraakoeruat alcorpsioon www.eeeerarrrgoo.... weraaterreeee www.siorrssoooooo....... wow.srosssooaaaaa www.leegaaassaaaee. www.aaabboaaaao-aaaaa..... wona www.eeeeeaaaaaa.... moraadd resalboeee crran iore aetoor messoo wolooooaaa mosd www.aeeeaaaeaaoooo.oo wwwaaaeeeaaae www.aaatoooaaae waaeeooeeaaae momaoo cosaa eemaaaa aover compaaao
Step 20
To delemitee mask Idd crnt-rraato S0le masiorriiio male mosso musio filowr to folos rape
"Oky Monily tie encuntrrenl Mi- Fnd LoOd"
The
Celabdl 12a0 reouad
Step
To detamiie madirooormaiii
manee now masioo falow
Pace
"Blo unsortimnal rite lries bolo-munko the Botr
"daay Deseerate de www.etoroaaa wit used
(-russ tha masio dress 2irs terusiic Medirssio bob ros wrw.-oooroiiis
"Dhep Celouate be lotal rambermeele tobrrrakk (2+ Temesso dimento moaiii sulo-----oiiiee ceeelloooeeeeieeeeeeeee
Thewloee- mosks conanio oadoosaiooosaa Telle-runno
cou coaase coneemaaann aesaad---uuuee reman
Step
To delemine the tiri rumbercnasio bota mnsiio marerrroil rosiio robe Re an kllo folse fnlow stpe
"Clep idantlyy the amount blaa uaed
The tdbe trion half ze mush whille ther mosle bie foerr Stree Peeee are 2 be bis Eoeressnoommmnte amooun ireasto with
Add bo SM nurbo of bolb hov Uika fel andiomean Wil Booy
Step
To delernie the taral number bols ts maln sohe, oe folos balow raass
"Sep t identity the amount tlke
"dkkp Dutomen ansont of fior usod
( (mso11990 timee = Mein bolp
"Sapp 2e calcclana the teail nember of bolt cped."
2 3 texi) (wbl)
Townttas the number uend moke the (aloeese Ai-
Figure& 8: Intermediate step outputs from LLaDOU during the generation of a 256-token response. The question is from GSM8K: A robe takes bolts of blue fiber and half that much white fiber. How many bolts total does it take?
Table 6: Evaluation Hyperparameters for Sudoku and GSM8K-Aug
Sudoku
GSM8K- -ug
Models
Generation Tokens
NFEs
Temp.
Generation Tokens
NFES
Temp.
GPT2 + CoT
256
256
0.0
256
256
SEDD DoT
256
512
0.5
256
64
0.5
SEDD DCoLT
19
32
0.5
64
64
0.0
Table 7: Training Hyperparameters for Sudoku
Models
Post-Traininn Training
Post- Post-Trainins Training Steps
Learning Rate
Batch Size
GPT2 CoT
SFT
1,500
3e-4
1024
SEDD + DoT
SET
1,500
le-4
1024
GPT2 CoT
RL
500
le-5
64
SEDD + DCoLT
RL
500
le-5
64
24

---

Step 12
To dedariiee contenenn cat Dowe inseiiiiienn soneeeeee
dww.eooooooo wARterouuso Cateroo look
crusto messi-orestiinn weapoeappppp
Step 28
"Tad
CEaTEIN Oat dove
Pr dan look adi mosit 6o Doriin MMB CELSOO CAN coStt Stwn trough seferrosie lp Reraee
"Deeemane wabber mosii www.ooaaaoaaaoo
Step 45
To oekerie meseennnnieeeeni si das sooe wwweitirreeee 3e jano pri.iriooiiieeeeeei Ryperssio Trnundo renueo, no oor filw follow staps.
rost number can the restowayy"
"Quberct
ressoo Ioti that dlose troughtsasi Torti
"Dataming rembericaati ansaataaoo
remaiing Femunlo render
muste remainng Frimaso
lest reinde Seearti aaprrtensio dheepteeatnnnn renunng Tiniso
dreve
Maronnnti--eeee toeeee cat-ressto www wwgpiestinneeea Vallojar
25 20-1
Step 64
traliic rukos Mol
"Toes koua sumer son moneway
cara losk edi: so the of cara that drowe though the
"Oatemine number of
sanating minutes" done through the saratring 15 menutes.
dlue number Sutiie rumber calo but eloi theaghh the reniarin 16 memutes ron the sodl namber of drive though
trough lna
22- 20-5
Figure 9: Intermediate step outputs from LL during the generation of a 256-token response. The question is from GSM8K: Cars have lined up on the motorway. Some of the cars drive through in the first 15 minutes of the traffic jam, then 20 more cars drive through in the remaining 15 minutes of the jam. 5 cars from the line take an exit so they don't 't have to drive through the traffic jam. If there originally 30 cars the motorway, how many drove through the traffic jam in the first 15 minutes?
C Experiment Settings
C.1 SEDD+DCOLT
Sudoku 4 x 4 We first generated 50,000 sudoku puzzles of size 4 x 4 by uniformly replacing 1-9 cells with zeros. Using DeepSeek V3-0324[]), we produced CoT reasoning steps and filtered the results remove repetitive responses and incorrect solutions. All training datasets this cleaned dataset to ensure fair comparison. Dataset samples can be found in the Fig. [1]
created from
We trained both DoT and CoT (with SFT) for 1,500 steps. For CoT (with RL), we first performed 1,000 steps of SFT to pretrain the CoT response, reaching 70.5% in accuracy, followed by another 500 steps of RL training using GRPO [36 For fair comparison, DCoLT was also pre-trained to reach presents training hyperparameters for the Sudoku 4 x 4 dataset. For both SFT and RL post-training, employ Adam optimizer with (8,, B2) = (0.9, 0.999). During RL post-tratt training,n we set the group size of GRPO to 32, and the kl coefficient to 0.
72.3% accuracy before applying 500 steps of RL training. Tab. |
For the Sudoku satisfics all the rules of valid 4 x Sudoku, namely that each row, each column, and each 2 x 2
task, the reward is determined by verifying whether the generated solution
25

---

Step 20
3
0
%
of
0
0
In
1
0
D
0
In
8
0
In
Result
8
8
0
Step
%
0
0
in
O
2
0
Result
<eot>
Step 60
%
3
O
O
In
In
2 0
\n
Result
8
(a) The number of students in school hall was 1000. The hall had 3 entrances A, B, and C which also served as the exils. If after meeting 30% of the students went out of the building through exit A, 3/5 of the remaining went out through exit and the rest went through exit C, calculate the number of students who went through exit
Step 14
Jamess
experss
he
pays
per
month
In
Vn
year
pays
pays
An
The
<eot>> <-os>>
Step 19
James "s*
he
pays
S
per
month
year
pays
2
2
1
year
year
In The answer
2
2
1
4
<ect>
Step 30
Jemes 's* insurance
more
than
normal
S0 he
month
That
year pays pays pays
year
<cos>>
Step 57
James
"s"
Insurance
Is
%
more
than
normal
S0
he
pays
In
year
ne
pays
1
9
2
(b) James has to buy insurance. Since he had an accident was 60% more than normal. The normal cost is $120 month. How much does he pay year?
Figure 10: Illustration of intermediate steps of generated responses by LLaDOU on GSM8K. Gray tokens represent positions that have already been unmasked prior to the current step. For each masked position, we show the predicted token with the highest generation probability. Token color indicates the unmasking score, with darker blue indicating higher probability of be unmasked. The token eventually selected to unmask at each step is bounded in a red box. The corresponding prompts are given in the sub-captions.
26

---

position value are Sudoku with where each columi,, and subgrid contains the numbers from 4 exactly Input:
Fill the position where value are a 4x4 Sudoku puzzle digits 1-4, where row, column, subgrid contains the numbers from eractly onee. Inpot:
1422
3421
4
0.2.4
4.000
0.3
Output:
Output:
1:
2:
Stup 3:
Step
Step
Step
Step 4:
1423
1423
1423
3421
3421
3
3421
2314
2314
23144
1243
1243
1243
124.3
3002
3142
3142
4000
41000
4132
413.
4031
4031
4231
0314
03114
0314
23.1
Figure 11: Examples of CoT data used for training GPT2+CoT and SEDD+DoT.
subgrid must contain the digits 1-4 exactly once. A reward of is assigned only when the solution is entirely correct; otherwise, the reward is O.
GSM8K-Aug Our experiments are conducted on the GSM8K-Aug dataset (sourced from [TO), which contains 384,623 training samples with simplified CoT processes. After filtering out samples with too long answers (exceeding 64 tokens) and those with invalid formats, examples. For the pretraining of SEDD on GSM8K-Aus we train it for 120K steps using the Adam optimizer with learning rate x 10 a batch size of 512, and a generation length of tokens. In the RL training phase, we employ the Adam optimizer with a learning rate of 5 x 10 and a batch size of 32, and set the group size for GRPO 64. This RL phase is trained for 1,000 steps.
retained 382,553
All training was conducted on H800 GPUs. For GSM8K-Aug pre-training, we spent 160 GPU hours on pretraining for 120K steps, and additional 168 GPU hours for the RL. post-triniigg Evaluation hyperparameters for both datasets are provided in Tab.
C.2 LLaDOU (LLaDA+DCoLT)
C.2.1 Implementation Details for Math Problems
Datasets & Benchmarks We consider two popular benchmarks, GSM8K g and MATH [16]. In our experiments, follow the common train-test split on these datasets. For GSMSK, there are 7.5K questions for training and 1.32K questions for testing. For MATH, there are 7.5K questions for training and 5K questions for testing. We report the accuracy on their test set. During training, we extract the final answers from generated responses to compute the rewards.
Reward function We assign a hard reward for each completion. that it is only if the final answer equivalent the ground truth [16]. For GSM8K and MATH, following [36], the answer must appear inside \boxed(]; we extract the boxed content, parse it into number or expression, and check symbolic equivalence against the reference answer.
Training configurations The model is trained with 64 prompts batch, each generating 16 completions to form group for advantage calculation. We take an AdamW optimizer with learning rate of 5 x 10-6, and (3), (32) = (0.9, 0.999). We do not apply the KL penalty by default, as provides marginal benefits in our experiments. The whole training lasts for 140 iterations on 16 H800 GPUs, which takes about 63 GPU days (i.e., about days on wall clock with 16 GPUs). We visualize the reward curves during training in Fig. 12
27

---

0.6
Smoothod
Rewad
0.0
Training Steps
Figure 12: Reward curves on MATH and GSM8K during DCoLT training. Dashed lines denote the raw rewards, while solid lines represent the average smoothed rewards, showing the overall trend.
C.2.2 Implementation Details for Code Generation
Dataset We filter the KodCode--V--SFT-II 41 dataset to construct our training dataset. KodCode
large fully-synthetic open-source dataset providing verifiable code solutions and test cases for coding tasks.
To ensure the quality and consistency of the training data, retain only samples labeled with the "instruct" style and containing exactly one entry function for test with non-empty docstring. We further exclude any samples where the provided solutions include class definitions constructor methods, and ensure that the solutions contain exactly one function definition. Additionally, we only samples from group of subsets, including Prefill, Taco, Leetcode, Codeforces, Code Contests, and Filter, for their appropriate difficulty level.
After filtering, we obtain a training set with 48.9K samples (around 12M tokens). We then format the samples into the prompt template below: You expert Python progr ammer. Your task is complete the implementation function named "<function, "omn__aam>...
TARGET FUNCTION
<docstring description>
UNIT TESTS **
Your code should pass unit like: <assert statement
<assert statement 2>
Here is the function to complete:
python
def <function_namet (<parameters>>) """doocttring description>"""
We assign hard 0/1 reward for each completion, that it is 1 only if the generated code block passes provided test cases. The code blocks are extracted from the responses based on the markdown format.
Reward function
28

---

Training configurations The training configurations are kept the same as for math. The training lasts for 240 iterations, and it takes about 127 GPU days on H800 GPUs.
Evaluation details We evaluate the model in a zero-shot setting on MBPP 4 and HumanEval 7 and report the pass @1 metric. We show some examples of their input prompts below.
MBPP:
You expert Python programmer Your task is complete the implementation function named 'remove_Occ",
TARGET FUNCTION
Write python function to remove first and last occurrence of given character from the string.
UNIT TESTS
Your code should pass unit tests like:
remove_Oes ("hhelo"", "1") "heo"
assert remove, Occ ("aboda", "a") "bod"
assert MamooveOcc("PPPPP "p") = "H"
Here is the function complete:
python
def aarov__occcinpu__peee
input_param_2).
"""Write python function to remove first and last occurrence of a given character from the string."" """
Humaneval:
You are expert Python programmer Your task is to complete the implementation of function named "has_close, elements".
Here is the function to complete:
"python
from typing import List
def bas_close_eleemnte (numbers: List [istaf],,, threshold: float) -> bool: """Check if in given list of numbers, any two numbers are closer to each other than the given threshold.
Examples:
>>>
([1.0, 2.0, 3.0], 0.5)
False
>>> has close, eaaeeeeeementtdeeee [1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
True
C.2.3 Direct Comparison with d1-LLaDA
direct comparison with di (4) using the same LoRA structure and dataset still reveals the superior performance by method. Specifically, train LLaDOU GSM8K only, using LoRA with rank r = 128 and scaling factor a 64. During inference, we set the generation length to 256, the number of diffusion steps 128. and the block length to 32 mirroring dI's configuration. In the same setting, DCoLT with LoRA achieves an accuracy of 84.7%, which outperforms d1 with both diffu-GRPO (79.8%) and di L1-LLADA (81.1%) setups, shown in Tab. 8 These results indicate that LLaDOU still performs better than with the same LoRA structure and the training data.
C.2.4 Applying DCoLT to Dream-7B on GSM8K
To further examine the generality of DCoLT, apply it to the Dream-7B 4 model on GSM8K. Unlike LLaDA, which natively masked diffusion model, Dream-7B originates from an auto- regressive model.
29

---

Table Comparison between dI and LLaDOU with LoRA on GSM8K. Both methods only use the GSM8K to train the LLaDA model with their respective RL approach.
Model
Accuracy (%)
LLaDA 8B
78.3
(diffu-GRPO
79.8
dI-LLaDA (SFT diffu-GRPO)
81.1
LL 8B (LoRA)
84.7
Using the same reinforcement learning configuration as in the main experiments (64 denoising steps and generation length L 256), DCoLT raises Dream-7B's accuracy from 50.11% to 80.53%, an absolute gain of +30.42%. The improvement magnitude is comparable to that observed on L-LaDA- 8B, suggesting that DCoLT consistently enhances reasoning ability across heterogeneous diffusion architectures. The model is trained with a batch of 64 prompts, each generating 16 completions for advantage calculation, using AdamW with learning rate of 2.5 x 10 6 and ($1, (0.9, 0.999).
C.3 Post-Traine post-Training Data Used in Tab.
In Tab. we list the data used in post-training for math and code generation tasks by different models. Note that some methods [14] 34 36) involve considerable large amount of proprietary math/code data for both pretraining and post-training of models, which makes crucial contributions to high accuracies.
Table 9: Post-training data for math / code generation used in Tab. "_" means this type of post- training was not used for the model.
Method
SFT Auto-regressive Models
RL
LLaMA2 7B 339
MetaMath 7B 455
395K samples (MetaMathQA)
thousands of SFT millions of RS examples
CodeLLaMA-Inse 7B
34
Deepseek 7B S
DeepscekMath Instruct 7B
30
776K (problem, solution) pairs
DeepseekMath-RR RL 7B
36
776K (problem, solution) pairs
144K questions
DeepsselCCoder-Instuuut 7B
14
2B tokens
Diffusion Language Models
Dream
LLaDA 8B 27
+ SFT 47
IK samples from sIK
+ diffu-GRPO 47
GSM8K: 7.5K / MATH: 7.5K
d1-LLaDA (SFT diflu-GRPO)
IK samples from sIK
GSM8K: 7.5K / MATH: 7.5K
Math: 7.5K GSM8K 7.5K /ATH/ Code: 48K filtered from KodCode
LLaDOU (LLaDA DCoLT) 8B
D Potential Scaling Law for Longer Generations
Fig. 13 illustrates some examples of generated responses of various lengths to the same prompt for the LLaDOU. In DLMs, the generation length tends increase more mask tokens initialized at the beginning of the reverse diffusion process. This differs from auto-regressive language models, where the length of generated response is determined by when the End of Text (EoT) token appears.
Although DL.S also use the FoT token to mark the end of the output, the token often appears later in the output sequence to generate a longer thinking process when more mask tokens initialized.
30

---

Answer (Generation Length-256)
Answer (Generation Length-384);
1. Set the equations equal to each other: Xx-4 5x"2 109.
1. the equations equal to coch other:
2. Reasrange the equatio:: -xx2 10-
[x44 7x"2 1422-103 103]
3. Y(u x22), then ((r2
4. Solve the quadratic equation: V(u Vinne (49 40/)
2. Reamannge the equation:
Vfret77 3112)).
( x 10-01
-(u 5)) or (0mm))
5. Therefore, (xx2
(0e"2
6. The socrdiaoosss are Cppum (5)) and Vpm (2)))
{ [u 7a 00-0
7. Identify K(a 51j and (a- 20.
Calculate nije
quadratic equation:
{ (--ac 'pm (t-y: todot lodot 10 todot
[ Ifacc pm (sgrt(9)) (2)
1 Miaet7 pm 33
{ lquad (teesjor) Iquad 22)
Sulbutinuee
find
{ xn2 -3 'quad [ox] lquad xn2 -2
{ (egt( 'quad (eet(or) lquad (sqra(2)
6. Desermine m and if a [ 'qudd (ext( lquadd 221
7. Calculate K (m-n D: mm n- 2-2-31
((boxed33))
Answer (Generation Length-512);
Step Set the equations equal each other. I[ 11x4 7x"2 10
Answer (Generation Length-512 finetuned):
Step Rearange equation
zoro.
points intersection between graphs ( -x144
I[ xx4-7x"2 10 1010
y = 7x - 10 y, we set the equations equal to each other: (nn 7x02 100)
Step 3: X xx21). The oquation becomes: [[ (112 7u [01]
Rearrange the equation to form quartic equation: '[x14
0.]
Step 4: Salve the quadratic equation for (( ())).
Using the quadratic fommala (u ifaef-b ypm (sgrt(b22- 4 V, where a(a 1
u=x U. equation [[r2 7a 10-01)
and < 10 10
[[ (i 'pm (- 'cdot lodot 10)) vedot 1)
gm (sgr1(49 40)3 (2)
solve quadratic equation using the quadmatic Formula Vfaec-
[[ uu- [iacc? pm (9)) (2)
pm tegrtt 4nc|] (2a) yj
I[ Minel pm
Mire (--7)) legnt (-770-------0
'edot 103
{ the (7 pm lugr[49 (0)) iffe pm
S: Calculate the two solutions for I[ M1 [1)] =5 [[ Vrin (2) -2
This gives us rwo solutions: ifaa [ Miac (2)2)))2)21
Step 6: Find the comresaandng x values.
For't V 5 I[ x222 limplics x- 'mm (5) !]
Since if - x22 () have: xx 22255 [ 2
For 2 2
[[ [[x"2 limplies pm (2) )]
Step Determine m
(--cnordinntes
From V xx12 51) (. M.m mm-m5
[[x Ipm (sgrt(5]
From V x 21) 1
(agtt(2
Soep 8: Calculate' i( [[ m 33]
We given that i n and (u= Therefore, [[m- 5-2333
Final
Thus, answer I ('boxee33)
|| (boxed[3)
(a) The graphs of = and 7x 10 intersect four points with x coordinnssss / m and + n where m > n. What is m n??
This offers a flexible way for DLMs to control various lengths of generated responses. As shown in longer generations could improve performance, particularly when the model fine-tuned by DCoLT with increasing generation length. This suggests the existence of potential scaling law for longer generations.
Tab. 5
Fig. 13b shows that some mistakes generated in the thinking process can be corrected in longer generations, thus achieving better results over shorter ones. We will study such a scaling phenomenon further in the future.
31

---

Answer (Generation Length=256):
Answer (Generation Length=384);
convert
calculation:
Step
168321 5 -44 ''ddot5% + ledot 5 5 + 2767
For K4321 S)):
"(1234 5 ledot 3-344 ledot 'edot 5 4 1390)
ledot 5-3 3-3 'eddt 5 ledot ledot 5-0 1000 75
Perform the subtraction: (276 139 1371)
+ 10961]
For K4234_445500
convert
lodot ledet 5-2 ledot ledit +4 1941
- -(137 div 5- remainder ((2)
-127 dis
-S 'div 11) remainder ((0)
Step 2: Perform the subtraction in base {1086 194 8923]
ddv 5 - remainder ((1)
Reading the remainders from bolluo
V1042
Skep 3: Convent (he result back base
(8922diiv
remainder
Therefore, the answer in base 5 is (342 550
(178 kiiv - West remainder 1 31]
remainder
[? liiddi 5 text; remainder 2-]
ldv 5 --0 bext remainder
Reading remainders from top, get (1202_5))
((boxed(3302
Answer (Generation Length 512 finetuned):
Firsr, we convert both numbers to hase 10 for easier calculation.
14322 50:
Yedot 5*3 Sedett ledot ledot 125 'edot 25
ledot 5 1 500 75 10++1 - 586
4
Answer (Generation Length-512)
Step Convert 1(821_55 [S)
K1234_500
4
4321
'cdot 5
ledot 5*3 ledet 4 ledot 5 4-4 ledot 125 leddt 25
Icdol 25 ledot 5 + 500 + 75 10 586
fedot 125
4
Step Convert (1234_55) base 10.
Next, we perform the subtraction in hase L0:
I
[
1234_5 5 1 ledot 513 %edott55 5+2 + lodot& 5 1 lodot 125
586 194 392
lodot
Now, convert 392 back base
Seep 3: Subtract the base 10 aquivalents.
I
First, repeatedlly divide by S:
586 1
[
392 'dv 5 lestt remainder ) 3
Step 4: Convert 392 back to base
[[
[
78 Idiv 5 15 best| remainder
392 78 'quud Mexi (romainder) kpaid ]
4
4
[
Viv best remainder
78 divv = qquid fremiinde)) qqaid
4
4
[
remainder
idiv 5 3 'quud texn (remainder) 'quad
]
[
Reading the remainders from bottom top, we get 1(3032_5), 5Q).
kqaid Yeeet fremainde) 'qaad
1
Thus, answer
Reading the remainders from boilum lop, V3032_ 5)).
4
booxed (3032_
(303__5
4
(b) Find 4321; 1234s. Express your answer in base 5.
Figure 13: Examples of how increasing generation lengths enable the DCoLT-reinforeed model to unfold more complex reasoning processes. Incorrect steps are marked in red, while the corrected steps are highlighted in green. The generation length means the number of mask tokens initialized in the input sequence to the reverse diffusion process. Except for the model finetuned with 512 generation length, other models are only trained with 256 generation length as shown in experiments.
32

---

Table 10: Model accuracies with different block length on GSM8K dataset. LLaDOU (LLaDA + DCoLT) is trained with generation length of 256 and 64 steps of reverse diffusion under respective block lengths.
Block Length
Model
32
256
LLaDA 8B
63.26
62.12
49.47
LLaDOU (LLaDA+DCoLT) 8B
83.17
83.11
82.27
E
Ablation on Various Block Lengths
divides the sequence into several blocks and generates responses blockwise from left to right in a semiiautoreerressive manner. This policy introduces additional priors on the block-wise order of text generation. In Tab. [0] for the baseline LLaDA 8B model, setting block length achieves an accuracy of 63.26% on GSM8K, significantly higher than 49.47% without dividing
LLaDA [27]
blocks.
However, when we apply DCoLT to LLaDA with different block lengths, all LLaDOU models achieve similar performance, with the with block length achieving slightly higher accurey (83.17%) than the one with block length 256 without dividing blocks (82.27%). These results suggest that LLaDOU does not rely the blocking prior in the baseline model.
Licenses for existing assets
The code and models associated with this paper will be released to the public later. For all code and data assets, refer their licenses in Tab. 11
Table 11: Reference assets and their licenses.
Asset
License
Utility
SEDD 24
MIT
Code & Model
GSM8K-AU Aug
10
Data
LLaDA
27
MIT
Code & Model
MATH
16
MIT
Data
GSM8K
g
MIT
Data
KodCode
CC BY-NC 4.0
Data
G
Broader Impacts
Our work introduces a novel perspective of lateral reasoning in language models, in contrast to the widely adopted CoT paradigms used LLMs. By enabling non-linear, format-free reasoning through reverse diffusion process, our approach offers new insights into the mechanisms of reasoning in generative models. This perspective has the potential to benefit broad range of reasoning-intensse intensive tasks, including embodied AI, autonomous agents, and complex mokiing systems, providing transparent thinking processes to minimize the hallucination risks that could emerge from inadequate reasoning of language models.
However, we acknowledge that DCoLT could pose potential unexpected societal impacts if misused, especially when handling misleading or adversarial prompts. In this work, we focus on math and code generation tasks, which outputs are objectively verifiable. With such a reward design, we can reduce the influence of dataset biases and encourage the development of reasoning behaviors that are aligned with the factual and logical consistency. Handling subjective preferences rewarding the models to train the DCoLT could be more challenging, and we leave it to our future works.
33

---

H Safeguards
DCoLT is designed for math and code generation tasks that involve objectively verifiable outputs and well-defined correctness criteria. These domains present relatively low risk of societal misuse compared to open-ended language generation tasks. To further reduce the potential for unintended use, we recommend deploying DCoLT alongside an input checker that ensures that the model only processes input relevant to target domains. This approach helps mitigate the risks related to adversarial prompting or misuse beyond the intended scope.
34


Reviews:
[{"review_id": "ecaLI0uTr1", "reviewer": "Reviewer_pRnQ", "summary": "This paper introduces a novel method, diffusion chain of lateral thought, which uses reinforcement learning with final answer only rewards to enhance reasoning ability of  diffusion language models. Authors develop methods for both continuous and discrete time diffusion models. They claim this could have bidirectional and format free reasoning abilities. Authors also implement experiments with SEDD and LLaDA with DCoLT on math, coding and sudoku benchmarks, and present competitive or better performance.", "questions": "The method only trained the model using final-answer reward signals, with no supervision or constraint on the intermediate diffusion steps. This setup has been shown to cause unreadable intermediate steps in autoregressive models (e.g. DeepSeek-R1), and this is a reason why they have an additional SFT process. Despite this, attached examples in this paper show semantically meaningful and readable CoT like reasoning. I’m curious why the case is different with autoregressive models and outcome-based RL on diffusion language models that lead to interpretable intermediate reasoning.", "limitations": "See weaknesses.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* It tries to simulate lateral, human-like thinking, where key concepts emerge early and are refined over time. This is a clear advancement over traditional Chain-of-Thought in autoregression models, because autoregressive models cannot modify generated tokens.\n* Authors thoughtfully implement DCoLT on both continuous time and discrete time diffusion language models. This highlights the adaptability across different model architectures.\n* This method leverages final answer only rewards, which is a practical strength, as high quality CoT annotations are expensive and limited.\n* The method achieves better or competitive results on math, coding and sudoku benchmarks. It performs even better than its autoregressive counterpart in some tasks.\n\nWeaknesses:\n* Some claims in this paper are problematic, e.g. for the generation length, I don’t think it plays an important role here. The performances with length from 256-512 are very close. \n* I’m not sure whether the method really optimizes the entire reverse process jointly (line 173-174), because in Algorithm1, the optimization is processed from 1 to n, which means getting loss for each denoising step separately.", "was_revised": true, "final_justification": "Please refer to my original comments and my response to the authors feedback during the rebuttal.", "rebuttal": "**W1: Some claims in this paper are problematic, e.g. for the generation length, I don’t think it plays an important role here. The performances with length from 256-512 are very close.**\n\nDuring RL training, LLaDA+DCoLT 8B is only optimized with a generation length of $L=256$, and it already outperforms LLaDA and other RL algorithms (e.g., d1) under the same length setting. In the submission, inspired by prior works on auto-regressive models, we also investigate the effect of generation length on performances in Tab. 4 as well.\n\nParticularly, although the overall performance with varying lengths may be close, when they are applied to MATH subsets with various difficulty levels,  LLaDA+DCoLT 8B with longer generations could perform much better on harder problems than that with shorter ones as shown in Tab. I. For example, on the hardest category with difficulty level 5, DCoLT improves the accuracy from 13.2% with a generation length of 128 to 21.5% with that of 512, a remarkable gain of 8.3% in accuracy.\n\n**Table I**: Accuracy of LLaDA+DCoLT 8B on MATH subsets across difficulty levels (5: the hardest and 1: the easiest) and generation lengths.\n\n| Length | Difficulty Level 1 | Difficulty Level 2 | Difficulty Level 3 | Difficulty Level 4 | Difficulty Level 5 |\n| ------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |\n| 128    | 80.8%              | 61.0%              | 45.6%              | 29.5%              | 13.2%              |\n| 256    | 83.3%              | 65.1%              | 52.3%              | 35.9%              | 18.7%              |\n| 384    | 82.4%              | 66.6%              | 54.9%              | 39.0%              | 20.2%              |\n| 512    | 82.6%              | 69.7%              | 56.9%              | 40.2%              | 21.5%              |\n\n\n**W2: I’m not sure whether the method really optimizes the entire reverse process jointly (line 173-174), because in Algorithm1, the optimization is processed from 1 to n, which means getting loss for each denoising step separately.**\n\nWe compute and accumulate the losses for **all** the denoising steps in the entire reverse diffusion process, and sum them up to update the model parameters. Note that the network weights were not updated **until** the losses from all the steps are accumulated to optimize the entire reverse process.\n\nMoreover, each one-diffusion-step gradient accumulated in Line 26-28 of Algorithm 1 fulfills a memory-efficient strategy. As described in Line 149, we release the computational graph after each step to reduce memory consumption, which still optimizes the entire reverse process jointly with the accumulated gradient in a memory-efficient fashion.\n\n**Q: The method only trained the model using final-answer reward signals, with no supervision or constraint on the intermediate diffusion steps. This setup has been shown to cause unreadable intermediate steps in autoregressive models (e.g. DeepSeek-R1), and this is a reason why they have an additional SFT process. Despite this, attached examples in this paper show semantically meaningful and readable CoT like reasoning. I’m curious why the case is different with autoregressive models and outcome-based RL on diffusion language models that lead to interpretable intermediate reasoning.**\n\nUnlike auto-regressive models that are pretrained to predict next token from semantically meaningful prefix, DLMs are pretrained [a] to recover coherent text outputs from randomly corrupted inputs that are incomplete and unreadable with masked tokens. This training strategy endows DLMs with the unique robustness to generate semantically readable text outputs even from corrupted text sequences amid the reverse diffusion process. This trained ability of DLMs allows outcome-based RL to generate semantically meaningful reasoning texts without relying on additional SFT to recover the corrupted tokens during intermediate diffusion steps.\n\n\n\n[a] Nie S, et al. Large language diffusion models. 2025."}, {"review_id": "ZsCE62VzFx", "reviewer": "Reviewer_P9Mj", "summary": "The paper proposes an online reinforcement learning algorithm called DCoLT for finetuning masked diffusion models on reasoning and code. The method uses group statistics to estimate the advantage function without using a explicit value function. The RL objective is similar to GRPO but it incorporates all the steps from the diffusion trajectory (unlike other works like d1 that only use a single step estimate). The paper also introduces an additional “unmasking” head for selecting the tokens to unmask, and show that this in itself provides significant gains in performance even if the rest of the model is frozen. The results are quite strong showing improvement over existing SFT and other RL baselines.", "questions": "1. Based on Appendix C.3, it appears that d1 trains separate models for GSM8K and MATH, each using its own corresponding training set. However, DCoLT seems to combine both datasets during training. This may not be a fair comparison. Can you please clarify whether the same training data is used for both models?\n    \n2. I may be misunderstanding, but it seems like d1 only finetunes LoRA adapters, whereas DCoLT does full parameter updates, making the results difficult to compare.\n    \n3. The term “nonlinear generation” is ambiguous. For instance, in line 131, I don’t quite understand what it means. If the intent for it is to mean “non-left-to-right,” then it might be better to write non-left-to-right or something similar.\n    \n4. Could you comment on the coherence of the complete answer sequence when the answer is correct? I realize that a final-answer-based 0/1 reward may not encourage, or perhaps even discourage, coherent text sequence in the middle of the answer. It will be nice to present some concrete analysis on this front.\n    \n5. Please provide more details regarding the final answer-based reward for the question-answering datasets. Specifically, what is the filter used to extract the final answer for GSM8K-Aug, GSM8K, and MATH datasets for reward generation?\n    \n6. Do you train the model to predict <eos> tokens? In Figure 8, in step 64, I see that there are <eos> tokens. Are these predicted by the model, or are they inserted automatically after the model predicts the first 
?\n    \n7. Have you tried DCoLT on Dream 7B? It would make the paper stronger if the method shows similar gains across the two models.\n\n8. Since RL training is generally unstable, it would be good if you could provide average reward vs training progress curves.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Clarity**: The paper is well written and easy to follow. There is a good amount of implementation-related details in the paper and the appendix, suggesting that the results should be easily reproducible.\n\n**Quality**: The baselines are a bit weak. Specifically, the only Diffusion+RL-based baseline is d1-LLaDA. But it is unclear if d1 can be compared directly with the proposed approach (see questions 1, and 2) below.\n\n**Significance**: The work could be quite useful for the community, as it demonstrates that full-trajectory based RL performs reasonably well for masked diffusion models.\n\n**Originality**: The work is original because it is the first to successfully apply trajectory-level RL for the masked diffusion model in the text domain.", "was_revised": true, "final_justification": "I believe that there is scope to further improve the paper by providing simpler baselines, such as diffusion SFT training with rejection sampling. However, by providing an apples-to-apples comparison (using LoRA instead of full fine-tuning) with the only other diffusion+RL baseline in the literature for text diffusion models, the authors' response somewhat addresses my concern.", "rebuttal": "**Q1 & Q2:  Based on Appendix C.3, it appears that d1 trains separate models for GSM8K and MATH, each using its own corresponding training set. However, DCoLT seems to combine both datasets during training. This may not be a fair comparison. Can you please clarify whether the same training data is used for both models? I may be misunderstanding, but it seems like d1 only finetunes LoRA adapters, whereas DCoLT does full parameter updates, making the results difficult to compare.**\n\nA direct comparison with d1 using the same LoRA structure and dataset still reveals the superior performance by DCoLT. Specifically, we train DCoLT on GSM8K only, using LoRA with rank $r=128$ and scaling factor $\\alpha=64$. During inference, we set the generation length to 256, the number of diffusion steps to 128, and the block length to 32 — mirroring d1’s configuration. In the same setting, DCoLT with LoRA achieves an accuracy of 84.7%, which outperforms d1 with both diffu-GRPO (79.8%) and d1-LLaDA (81.1%) setups, as shown in Tab. I. These results indicate that DCoLT still performs better than d1 with the same LoRA structure and the training data.  We will add it in the final revision.\n\n**Table I**: Comparison between d1 and DColT with LoRA on GSM8K. Both methods only use the GSM8K to train the LLaDA model with their respective RL approach.\n\n| Model                       | Accuracy |\n| --------------------------- | -------- |\n| LLaDA 8B                    | 78.3%    |\n| +diffu-GRPO                 | 81.9%    |\n| d1-LLaDA (SFT + diffu-GRPO) | 82.1%    |\n| **LLaDA + DCoLT 8B (LoRA)** | 84.7%    |\n\n**Q3: The term “nonlinear generation” is ambiguous. For instance, in line 131, I don’t quite understand what it means. If the intent for it is to mean “non-left-to-right,” then it might be better to write non-left-to-right or something similar.**\n\nIn our paper, \"nonlinear generation\" intends to mean \"non-left-to-right\". We will clarify this in the revision.\n\n**Q4: Could you comment on the coherence of the complete answer sequence when the answer is correct? I realize that a final-answer-based 0/1 reward may not encourage, or perhaps even discourage, coherent text sequence in the middle of the answer. It will be nice to present some concrete analysis on this front.**\n\nAlthough DCoLT is trained using only final-answer-based 0/1 rewards, we observe that it often produces coherent text sequences as shown in Fig. 8-9 of Appendix B. To further quantify this, we decompose the generated text responses on GSM8K into individual reasoning steps and evaluate their text coherency quality using a process reward model (Qwen2.5-Math-PRM-7B [a]). On correctly answered examples, LLaDA+DCoLT achieves an average step-wise reward of 0.96 (very close to the maximum reward of 1.0), better than LLaDA with an average reward of 0.94. This suggests that DCoLT produces not only correct answers but also coherent text sequence.\n\nThe reason can be that: unlike auto-regressive models that are pretrained to predict next token from readable text prefix, DLMs are pretrained [b] to recover coherent text from randomly masked inputs that are incomplete and unreadable. This training strategy endows DLMs with the unique robustness to generate coherent text outputs from corrupted text sequences amid the reverse diffusion process. This trained ability of DLMs allows the outcome-based RL to maintain the coherence of the final output sequences without relying on explicit rewards over intermediate diffusion steps to recover masked tokens.\n\n\n\n[a] Zhang Z, et al. The Lessons of Developing Process Reward Models in Mathematical Reasoning. 2025.\n\n[b] Nie S, et al. Large language diffusion models. 2025.\n\n**Q5: Please provide more details regarding the final answer-based reward for the question-answering datasets. Specifically, what is the filter used to extract the final answer for GSM8K-Aug, GSM8K, and MATH datasets for reward generation?**\n\nFor GSM8K-Aug experiments with SEDD, we follow the approach used in DoT [c]: if the model outputs contain `####`, we extract the number that follows as the predicted answer. Example responses following this pattern are provided in Fig. 6.\n\nFor GSM8K and MATH experiments with LLaDA, we follow DeepseekMath [d] and require the final answer to be put in the right response format (e.g., be enclosed in a LaTeX-style box `\\box{}`). During reward computation, we extract the content inside the box and attempt to parse it into a number or symbolic expression. Then, we compare it against the ground-truth answer using symbolic math equivalence checks, following the evaluation code from the MATH dataset by Hendrycks et al [e]. Examples of accepted answer formats are shown in Fig. 8 and Fig. 9 of Appendix B.\n\n\n\n[c] Ye J, et al. Diffusion of thought: Chain-of-thought reasoning in diffusion language models. NeurIPS 2024.\n\n[d] Shao Z, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. 2024.\n\n[e] Hendrycks D, et al. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021.\n\n**Q6: Do you train the model to predict [object Object] tokens? In Figure 8, in step 64, I see that there are [object Object] tokens. Are these predicted by the model, or are they inserted automatically after the model predicts the first 
?**\n\nThe `<eos>` tokens are predicted by the model. These predictions are considered as part of the trajectory to reinforce during training.\n\n**Q7: Have you tried DCoLT on Dream 7B? It would make the paper stronger if the method shows similar gains across the two models.**\n\nWe appreciate the reviewer’s suggestion and conduct an additional experiment applying DCoLT to Dream 7B on GSM8K, using 64 denoising steps and a generation length of 256. As shown in Tab. II, DCoLT improves Dream 7B's accuracy to 80.53%, representing a +30.42% gain over the original model — comparable to the improvement observed on LLaDA. \n\nDue to limited time during the rebuttal period, we have not yet conducted a experiment with 256 denoising steps, but we plan to include those results in the revised version of the paper.\n\n**Table II**: Experimental results by applying DCoLT to Dream 7B on GSM8K with 64 denoising steps.\n\n|      Models      |     Accuracy     |\n| :--------------: | :--------------: |\n|     Dream 7B     |      50.11%      |\n| Dream + DCoLT 7B | 80.53% (+30.42%) |\n\n**Q8: Since RL training is generally unstable, it would be good if you could provide average reward vs training progress curves.**\n\nThanks for your constructive advice. During training, our model exhibits a stable increase in reward. Due to formatting constraints, we are unable to include these figures in the rebuttal. However, we will include training curves in the revised version to illustrate the training process."}, {"review_id": "FEFFvygtg5", "reviewer": "Reviewer_h4x9", "summary": "This paper introduces the Diffusion Chain of Lateral Thought, a reasoning framework for diffusion language models that treats reverse diffusion steps as thinking actions and optimizes them via outcome-based RL. Unlike linear CoT methods, DCoLT enables bidirectional, non-linear reasoning without strict grammatical constraints on intermediate steps. Experiments show that DCoLT-reinforced models outperform SFT and RL baselines on math and code generation tasks.", "questions": "1. How does DCoLT's RL approach (e.g., GRPO) compare to other RL algorithms (e.g., PPO) in terms of training stability and performance?\n\n2. The paper shows longer generations improve accuracy, but what is the optimal length for tasks with different degrees of difficulty, and how does it balance computation and model performance?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. DCoLT leverages reverse diffusion to model lateral thinking, allowing non-linear, format-free reasoning that aligns with human cognitive patterns, which is not supported by causal CoT.\n2. Using only public data and 16 H800 GPUs, DCoLT-reinforced DLMs achieve state-of-the-art results, outperforming models trained with proprietary data.\n3. Supporting both continuous-time and discrete-time diffusion language models, DCoLT demonstrates versatility across different diffusion paradigms.\n\nWeaknesses:\n1. Performance could be improved with more data and compute, and prior works with proprietary data show better results.\n2. DCoLT is validated only on tasks with clear reward functions e.g., math and coding, lacking evaluation on open-ended natural language tasks.", "was_revised": true, "final_justification": "I am satisfied with the rebuttal.", "rebuttal": "**Response to Strengths And Weaknesses**\n\nWe thank the reviewers for acknowledging our contributions in modeling lateral thinking through reverse diffusion, achieving competitive results using only public data and moderate compute, and demonstrating versatility across different diffusion language models.\n\nWe agree that performance can be further improved with access to larger datasets and increased computational resources. We also recognize the importance of extending DCoLT to open-ended natural language tasks. To this end, we are exploring the integration of learned reward models to tackle these tasks, and will scale DCoLT in future works to further showcase the generality and potential of DCoLT across diverse reasoning challenges.\n\n**Q1: How does DCoLT's RL approach (e.g., GRPO) compare to other RL algorithms (e.g., PPO) in terms of training stability and performance?**\n\nIn the paper, we adopt GRPO in DCoLT for a fair comparison with existing models (e.g., d1). Prior studies [a] have noted that PPO could suffer an unstable training process to learn a value network particularly in long reasoning chains. In contrast, GRPO completely eliminates the need for learning a value network, thereby demonstrating more stable training with better performances as evidenced by existing works [b]. \n\n\n\n[a] Yuan Y, et al. What’s Behind PPO’s Collapse in Long-CoT? Value Optimization Holds the Secret. 2025.\n\n[b] Shao Z, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. 2024.\n\n**Q2: What is the optimal length for tasks with different degrees of difficulty, and how does it balance computation and model performance?**\n\nWe evaluate LLaDA+DCoLT 8B on MATH subsets grouped by different difficulty levels, following the original dataset annotations[c].  As shown in Tab. I, the accuracy on level-1 problems saturates with length 256, suggesting that shorter generations are sufficient for simpler questions. In contrast, the performance on harder problems (levels 2–5) continues to improve with longer generations, indicating that complex reasoning benefits from longer responses. This complies with our intuition that optimal generation length is dependent on difficulty levels — shorter for easier problems, and longer for more difficult ones. This also reflects a trade-off between computational cost and model performance, allowing generation length to be adapted to varying difficulty levels.\n\n**Table I**: Accuracy of LLaDA+DCoLT 8B on MATH subsets across difficulty levels (5: the hardest and 1: the easiest) and generation lengths.\n\n| Length | Difficulty Level 1 | Difficulty Level 2 | Difficulty Level 3 | Difficulty Level 4 | Difficulty Level 5 |\n| ------ | ------------------ | ------------------ | ------------------ | ------------------ | ------------------ |\n| 128    | 80.8%              | 61.0%              | 45.6%              | 29.5%              | 13.2%              |\n| 256    | 83.3%              | 65.1%              | 52.3%              | 35.9%              | 18.7%              |\n| 384    | 82.4%              | 66.6%              | 54.9%              | 39.0%              | 20.2%              |\n| 512    | 82.6%              | 69.7%              | 56.9%              | 40.2%              | 21.5%              |\n\n\n\n[c] Hendrycks D, et al. Measuring Mathematical Problem Solving With the MATH Dataset. NeurIPS 2021."}, {"review_id": "79owVOmGJG", "reviewer": "Reviewer_bbyf", "summary": "This paper introduces DCoLT, a new framework for reasoning in Diffusion Language Models (DLMs) that follows lateral thinking instead of traditional step-by-step reasoning. It uses reinforcement learning to train all intermediate steps in the reverse diffusion process, with an Unmasking Policy Module (UPM) using a Plackett-Luce model.", "questions": "1. Have authors considered integrating auxiliary rewards alongside outcome-based RL?\n\n2. Provide concrete examples of failure cases\n\n3. Can you provide theoretical analysis of when and why DCoLT should outperform traditional approaches?\n\n4. How does the computational cost compare to autoregressive methods with similar performance?\n\n5. Can you demonstrate that the \"lateral thinking\" behavior is consistent and beneficial across a broader range of problems?\n\n6. How does the approach scale to larger models and longer sequences?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I'll provide a comprehensive review of this paper on \"Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models.\"\n\n## Summary\n\nThis paper introduces DCoLT (Diffusion Chain of Lateral Thought), a novel reasoning framework for diffusion language models that treats intermediate diffusion steps as \"thinking\" actions and optimizes the entire reasoning trajectory using outcome-based reinforcement learning. The authors implement DCoLT on two diffusion language models (SEDD and LLaDA) and demonstrate improvements on math and code generation tasks.\n\n## Strengths\n\n### Quality\n- **Strong experimental validation**: The paper provides comprehensive experiments across multiple tasks (GSM8K, MATH, MBPP, HumanEval) and compares against relevant baselines including both diffusion models and autoregressive models.\n- **Solid technical implementation**: The authors properly adapt RL training to two different types of diffusion models (continuous-time SEDD and discrete-time LLaDA), showing technical depth.\n- **Thorough analysis**: The paper includes helpful visualizations of the generation process and ablation studies examining key components like the Unmasking Policy Module.\n\n### Clarity\n- **Well-structured presentation**: The paper is generally well-written with clear motivation and methodology sections.\n- **Good visual aids**: Figure 1 effectively illustrates the key differences between CoT and DCoLT, and the appendix provides helpful examples of the generation process.\n- **Comprehensive details**: The extensive appendix provides implementation details necessary for reproduction.\n\n### Significance\n- **Novel perspective on reasoning**: The concept of \"lateral thinking\" in diffusion models is interesting and provides a fresh angle on reasoning in language models.\n- **Competitive results**: DCoLT achieves strong performance, particularly on LLaDA where it shows substantial improvements (+9.8% on GSM8K, +5.7% on MATH).\n- **Data efficiency**: The approach demonstrates good data efficiency compared to methods requiring much larger proprietary datasets.\n\n## Weaknesses\n\n### Technical Concerns\n1. **Limited theoretical justification**: The paper lacks theoretical analysis of why treating diffusion steps as reasoning should work better than traditional approaches. The \"lateral thinking\" metaphor, while intuitive, needs stronger theoretical grounding.\n\n2. **Unclear advantage over autoregressive models**: While the results are competitive, it's not entirely clear why the added complexity of diffusion models is justified when autoregressive models with CoT can achieve similar or better performance with much simpler training.\n\n3. **Reward design limitations**: The approach is limited to tasks with verifiable rewards (math/code), which significantly constrains its applicability. The authors acknowledge this but don't provide a clear path forward.\n\n### Experimental Limitations\n1. **Limited model scales**: Experiments are conducted on relatively small models (400M SEDD, 8B LLaDA). It's unclear how the approach scales to larger models where autoregressive methods have shown dramatic improvements.\n\n2. **Computational efficiency concerns**: The paper doesn't thoroughly analyze the computational overhead of the multi-step diffusion process compared to autoregressive generation, which is crucial for practical adoption.\n\n3. **Limited baseline comparisons**: While the paper compares to some diffusion models, it would benefit from more comprehensive comparisons with recent strong autoregressive reasoning methods.\n\n### Methodological Issues\n1. **Unclear generalization**: The \"lateral thinking\" concept is demonstrated mainly through examples, but it's unclear how consistently this emerges across different problems and whether it's truly beneficial or just a different path to the same solution.\n\n2. **Evaluation metrics**: The paper focuses primarily on accuracy metrics but doesn't analyze other important aspects like reasoning quality, interpretability, or failure modes.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns in their rebuttals, and I believe a score of 4 is appropriate for this paper.", "rebuttal": "**Q1: Have authors considered integrating auxiliary rewards alongside outcome-based RL?**\n\nIn this work, we focus exclusively on outcome-based rewards to ensure a fair comparison with existing models. We agree that incorporating auxiliary rewards, such as step-level correctness, could potentially enhance performances. We will explore it in future work.\n\n**Q2: Provide concrete examples of failure cases**\n\nThank you for the suggestion. We provide an example in Fig. 12(b), where DCoLT generates incorrect reasoning due to a miscalculation in an intermediate step. This type of error often arises when complex problems are not adequately decomposed into trackable steps.\n\nAs shown in Fig. 12(b), the failure case was corrected by increasing the generation length to $L=512$, which enables finer-grained decomposition of reasoning steps. This is also supported by the improved performance with longer generations shown in Tab 4 in the submission. We will add more failure cases and show how they can be remedied by the DCoLT.\n\n**Q3: Can you provide theoretical analysis of when and why DCoLT should outperform traditional approaches?**\n\nOne of DCoLT's unique features is it learns the order of generated tokens based on the Plackett–Luce model, grounded in Luce’s Axiom of Choice [a] and Random Utility Model [b]. This enables the model to explicitly learn an optimal token generation order, rather than sticking to a fixed left-to-right order as in autoregressive models.\n\nThis flexibility is particularly advantageous in tasks where the natural problem-solving order differs from the left-to-right text order, such as in Sudoku. For example, as shown in Fig. 3, a natural order to complete the task is to fill in easier cells first, and then use them as the context to infer more difficult ones.\n\n\n[a] Mollica C, et al. Remarkable properties for diagnostics and inference of ranking data modelling. British Journal of Mathematical and Statistical Psychology, 2022.\n\n[b] Ma J, et al. Learning-to-rank with partitioned preference: Fast estimation for the Plackett-Luce model. International Conference on Artificial Intelligence and Statistics, 2021.\n\n**Q4: How does the computational cost compare to autoregressive methods with similar performance?**\n\nWe provide an inference-time comparison between DCoLT and a recent autoregressive reasoning model, Deepseek-R1-Distill-Llama-8B. Specifically, we randomly sample 500 problems from GSM8K and report the average wall-clock inference time required to complete a full response. As shown in Tab. I, DeepSeek-R1-Distill-Llama-8B exhibits higher inference latency. In contrast, DCoLT requires less inference time, with even better accuracy.\n\n**Table I**: Computation cost to solve one problem in GSM8K. The inference latency is averaged on 500 problems.\n\n| Model                        | Accuracy | Inference Latency (s) |\n| ---------------------------- | -------- | --------------------- |\n| LLaDA + DCoLT 8B             | 88.1%    | 6.34                  |\n| DeepSeek-R1-Distill-Llama-8B | 77.3%    | 10.80                 |\n\n**Q5: Can you demonstrate that the \"lateral thinking\" behavior is consistent and beneficial across a broader range of problems?**\n\nDCoLT demonstrates consistent improvements across both math and code generation tasks, as shown in our results on MATH, GSM8K, MBPP and HumanEval. These various domains involve complex multi-step reasoning, and the observed gains suggest that lateral exploration during reasoning is broadly beneficial.\n\nWhile this work focuses on tasks with verifiable answers, we believe that the principles behind DCoLT generalize to broader settings. In future, we will extend DCoLT to open domains(e.g., scientific inquiry and creative generation) with trained reward models to guide lateral reasoning where verifiable answers are not available.\n\n**Q6: How does the approach scale to larger models and longer sequences?**\n\nDuring RL training, LLaDA+DCoLT that is optimized with a generation length of $L=256$ has already outperformed LLaDA and other RL algorithms (e.g., d1) under the same length. As shown in Tab.4, we showed that scaling the generation length further improves the model performance.\n\nRegarding scalability with increasing model sizes, our approach demonstrates consistent gains on both smaller SEDD models (400M) and larger LLaDA models (8B). To our best knowledge, LLaDA 8B model is the largest open-source diffusion language model. We will test the ability of DCoLT scaling to larger DLMs once they are available."}]

Author rebuttal:

Meta-review: his paper presents diffusion chain of lateral thought, a method for enhancing reasoning ability of diffusion language models with reinforcement learning with outcome-reward. Two implementations for continuous and discrete time diffusion models are proposed respectively. Experiment results are strong on math, coding and sudoku benchmarks.

Strengths:
1. Novel ideas for bidirectional, non-linear reasoning.
2. Comprehensive experiments and strong results.
3. This paper is generally well-written.

Weaknesses:
1. some technical details are not fully clear. the authors should further revise their paper according to reviewers' comments.
Decision: Accept (poster)