Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
Authors: Jingyang Ou
Jiaqi Han
Minkai Xu
Shaoxuan Xu
Jianwen Xie
Stefano Ermon
Yi Wu
Chongxuan Li
Venue: ICLR 2026 Poster (2026)
Keywords: masked diffusion models
diffusion language models
reinforcement learning
GRPO
dLLMs
Primary area: foundation or frontier models, including LLMs

Abstract: Reinforcement Learning (RL) has proven highly effective for autoregressive language models, but adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges. The core difficulty lies in likelihood approximation: while autoregressive models naturally provide token-level conditional probabilities essential for token-level RL objectives (e.g., GRPO), dLLMs generate sequences through iterative non-autoregressive denoising steps that lack this factorization. To address this fundamental mismatch, we propose ELBO-based Sequence-level Policy Optimization (ESPO), a principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy. Our method incorporates per-token normalization of importance ratios and robust KL-divergence estimation to ensure stable large-scale training.  Extensive experiments on mathematical reasoning, coding, and planning tasks demonstrate that ESPO significantly outperforms token-level baselines, achieving dramatic improvements of 20-40 points on the Countdown task, while maintaining consistent gains on math and coding benchmarks. Our approach establishes sequence-level optimization as a principled and empirically effective paradigm for RL in dLLMs.

Published as a conference paper at ICLR 2026
PRINCIPLED RL FOR DIFFUSION LLMS EMERGES FROM A SEQUENCE-LEVEL PERSPECTIVE
Jingyang 1..3
Jiaqi Han",
Minkai Xu*, Yi Wu
Shaoxuan Xu1:2.3
Jianwen X
Stefano Ermon
Chongxuan Li
1 Gaoling School of Artificial Intelligence, Renmin University of China
2 Beijing Key Laboratory of Research on Large Models and Intelligent Governance
3 Engineering Research Center Geee--Genratinnn Intelligent Search and Recommendation, MOE
"Stanford University LLmbdda Inc Tsinghua University Correpsonding authors
jadu.unjjwwuy {ouj foujingyang, chongxuanni rucc. edu. cn; jxwuy @ma aalmmlll........ tsiigguua...... edu. on
ABSTRACT
Reinforcement Learning (RL) has proven highly effective for autoregressive lan- guage models, but adapting these methods to diffusion large language models (dLLMs) presents fundamental challenges. The core difficulty lies likelihood approximation: while autoregressive models naturally provide token-level condi- tional probabilities essential for token-level RL objectives (c.g.. GRPO), dLLMs generate sequences through iterative non-autoregressive denoising steps that lack this factorization. To address this fundamental mismatch, we propose ELBO- based Sequence-level Policy Optimization (ESPO), principled RL framework that treats entire sequence generation as a single action and uses the ELBO as a tractable sequence-level likelihood proxy. Our method incorporates per-token normalization of importance ratios and robust KL-divergence estimation to en- sure stable large-scale training. Extensive experiments on mathematical reason- ing, coding, and planning tasks demonstrate that ESPO significantly outperforms token-level baselines, achieving dramatic improvements of 20-40 points on the Countdown task, while maintaining consistent gains math and coding bench- marks. Our approach establishes sequence-level optimization as a principled and empirically effective paradigm for RL in dL dLLMs. Our code is available fattps://qithut
INTRODUCTION
Large language models (LLMs) (OpenAl 2023) have become cornerstone of modern natural lan- guage processing, achieving remarkable progress across math (Guo et al.) 2025), coding (Hui et al.) 2024), and planning tasks (Yao al. 2023). While autoregressive (AR) modeling has long dom-
inated this field, recent advances in diffusion large language models (dLLMs) have demonstrated
strong potential as an alternative formulation (Ou al. 2024 Shi et 2024 Sahoo al. 2024
Nic et al. 2025 Ye ct al.) (025). By modeling language generation as an iterative denoising process, dLLMS bypass the left-to-right dependency AR models and offer advantages in long context (Liu
et al. 2025b), multimodal (Yang et al. 2025 You et al. 2025a Li et al., 2025 Yu et al. 2025) and
fast inference (Inception Labs et al. 2025) DeepMind 2025) Song et al. 2025).
With the advent of powerful pretrained dLLMss, the next frontier lies in post-training (Ouyang al., 2022] to further enhance their capabilities. Among various post-training paradigms, reinforcement powerful approach that enables test-time scaling (Snell et al.) 2025)
learning (RL) has emerged
through verifiable rewards (Guo et al.) 2025). It has yielded substantial gains on reasoning tasks in such as math (Cobbe et al. 2021b), coding (Chen et al. 2021) and reasoning (Liu et al., 2023b). Motivated by this success, natural question arises: how can we extend reinforcement learning to dLLMs?
recent AR models (OpenAl]2025)
Applying RL to dLLMs, however. is nontrivial. Mainstream RL algorithms in language model- ing (e.g., GRPO (Shao et al. 2024)) assume left-to-right factorization of the sequence likelihood

---

Published as a conference paper at ICLR 2026
and rely on token-level importance ratios maly" la.y")
Feu (2 In contrast, dL.LMs generate sequences
non-autoregressively making such token-level conditionals either ill-defined computationally pensive. Prior attempts to address this discrepancy have resorted to heuristic approximations such
mean-field surrogates (Zhao et al. 2025) token-level ELBO contributions (Yang et al. 2025
Gong et al. 2025) or else computationally heavy trajectory-Ievel formulations (Huang et al. 2025
None of these approaches fully resolves the mismatch between autoregressive RL. objectives and the holistic generation process of dLLMs.
In this work, we address this fundamental conflict by introducing ELBO-based Seteaseeeeeeggnnnee Sequence-level Policy Optimization (ESPO), sequence-level reinforcement learning framework tailored for dLLMs. Our key insight is that dLLMs should not be foreed into an autoregressive token-level action space. Instead, we treat the generation of an entire sequence as single action, leveraging the ELBO as a tractable proxy for the intractable sequence liellikelihood. We further incorporate stabilization techniques essential for large-scale training, including per-token normalization of the ELBO ratio and robust KL-rregllarizatonn Our method eliminates the inconsistencies introduced by heuristic token-level approximations and enables stable, computationally efficient training.
Empirically, we validate the effectiveness of our design through extensive ablation studies, which confirm that combining sequence-level optimization with the ELBO objective provides a stable and principled foundation for reinforcement learning in dLLMs. Beyond ablations, we further eval- uate our method mainstream tasks spanning mathematics, coding, and planning. Across both LLaDA (Nie et al., 2025) and Dream (Ye et al. 2025), our approach consistently outperforms prior dLLM-RL baselines such dI (Zhao al.) 2025) and wdl (Tang al., 2025). with particularly strong gains on planning tasks that require global consistency.
In summary, we make the following contributions:
We provide a systematic analysis of why standard autoregressive RL objectives are incompatible with the non-autoregreesivv dLLMs, clarifying the limitations of existing heuristic approaches.
We propose ESPO, a principled sequence-level RL. framework that leverages the ELBO as a tractable proxy for sequence likelihood and introduces stabilized ratio and KL estimators for robust large-scale training.
We demonstrate through comprehensive experiments and ablation studies that ESPO yields sta- ble optimization and consistent improvements across math, coding, and planning benchmarks, surpassing prior dLLM-RL methods.
BACKGROUND
2.1 DIFFUSION LARGE LANGUAGE MODELS
Diffusion Large Language Models (dLLMs), or more specifically Masked Diffusion Models (MDMs), define a forward process that gradually corrupts the clean input by replacing tokens with the mask token M. Given the prompt x and the clean completions y, the forward process qe (3/ y. x) time is defined follows:
de (vel,, I) = II ge (uille and (e x))= = [ 1 1=1
y
(I)
M.
Unlike autoregressive models, the exact log likeliood log Te (y xz) in dLLMs is typically approxi-
mated via the evidence lower bound (ELBO) (Ou al. 2024 Shi et al.
2024 Sahoo et al.) 2024):
Lo(y)z) A 1) ( (x.x)
Σ
1 1gt
M] log (o By. x)
< log mo (2)
i
Zhu al. (025). Eq. 2 has equivalent, lower- variance variant that replaces the continuous masking ratio / with a discrete number of masked tokens 7:
As noted in Ou al. (2024): Nie et al. (2025).
I
(u) A E D.) E 1))))
Σ
1 jpt = M] log po (v' (1)),1
(3)
2

---

Published as a conference paper at ICLR 2026
where denotes the number of tokens masked (sampled uniformly). and y represents the corrupted
sequence obtained by masking tokens without replacement. While Eq. (2) and Eq. 33) only provide bound, the ELBO has been empirically shown to be tight and practically effective surrogate for
the intractable exact likelihood. This holds true for both likelihood- based evaluation (e.g., perplex-
ity (Lou et al. 2024 and training (e.g., DPO variants (Zhu et al. 2025)).
2.2
REINFORCHMENT LEARNING
Policy gradient methods have been shown to be highly effective for post-training LLMs (uyang al. 2022). Among them, Group Relative Policy Optimization (GRPO) (Shao et al. 2024) is widely adopted as it eliminates this need for value model (Schulman et al. 2017) by replacing it with a simpler Monte Carlo estimation: Given a prompt x. GRPO samples a group of G candidate completions Ji=1 from the old policy Te. Instead of estimating the baseline with a learned value function, it computes the relative advantage of each sample as its reward minus the group mean reward (Liu et al. (025d). Incorporating a KL penalty, the resulting optimization objective is:
(y)
JGRPO (T))
E EgnDD. (0) ..... (.... moa (11x)
1
m00(p A clipp(p -, -3DK (re) meel) -
(4)
G
1
where phi(1)
no T0() b. (k((t)))) is the token-level importance ratio between policies, and ne (2 (60 (r.y),10))))))))))
A R(x, )0 a Ei= R(x, y)( denotes the group-relative advantage. more compre-
hensive introduction to reinforcement learning is provided Appendix B.T
THE CHALLENGE OF THE TOKEN-LEVEL PERSPECTIVE IN DLLMS
The core challenge in applying reinforcement learning dLLMs stems from fundamental mis- match between the dLLMM'' probabilistic modeling and the assumptions inherent in standard RL. algorithms. Mainstream policy gradient algorithms, including GRPO as formulated in Eq. 44). are intrinsically designed for autoregressive models that factorize the sequence likelihood into a product of conditional probabilities To(y)a; a I no (y j y =²² This factorization naturally defines a sequence of actions, allowing the objective assign rewards the token level via the impor- tance ratio (2.y
Te (w* a.y However, dLLMs generate text non-autorrerressivlly refining complete sequence over iterative denoising steps. Consequently, the autoregressive conditional probability To(y"12,, (2c, <*) is ill-defined or hard to compute (see Appendix B.2 for detailed discussions), forcing existing methods to rely on heuristic proxies to bridge this gap.
Early attempts to resolve this incompatibility focused on finding suitable token-level substitute for the AR conditional probability T |x, (*). For instance, d1 (Zhao et al. 2025) employed a
mean-fiell approximation log po (v* [x)) as proxy for log (y |2, y ** This approach is inaceu-
rate as it ignores the context provided by other tokens in the sequence Recognizing this limitation, subsequent work such as UniGRPO (Yang et al., 2025) and Coupled-GRPO (Gong et al., 2025)
proposed more sophisticated proxy: the token's 's contribution to the ELBO, Lb (v)
(ylx)) & Fr Eeyerg (Velzar)
1 Molog Po (ve,
(5)
The (g(2) is better aligned with the nature of dLLM generation, since its computation involves predicting a masked token given both and the surrounding unmasked context from y. However, the ELBO, Lo(y)=) Σk= C/ (2)22) is only valid on sequence level lowerbound of log ()(y) An individual component C (2][z) has no formal probabilistic interpretation as a conditional likelihood. Therefore, the decomposition of ELBO at the token level and its heuristic substitution into the GRPO objective (Eq. 44) introduces unknown inconsistency.
"mmleemnntations may vary; for instance, UniGRPO's BLBO-like term omits the & coefficient from Eq. akin the simplified objective in DDPM (Ho et al. 2020
3

---

Published as a conference paper at ICLR 2026
As analyzed above, the core issue is not merely about finding better token-level proxy, but that the token-level decomposition itself fundamentally does not fit for dLLMs. Forcing a dLLM into a token-level AR framework rests
improper assumption. This motivates approach: instead of adapting the dI.LM model to fit the algorithm, we must adapt the algorithm to respect the holistic, sequence-level nature of the dLL.M model.
4 A PRINCIPLED SEQUENCE--EVELL RL FRAMEWORK FOR DLLMS
Motivated by analysis in Section we propose ELBO- -based Sequence-level Policy Optimization (ESPO) algorithm. which tailored for dLLMs. Our approach is built on a sequence-level action space, uses the ELBO as tractable proxy for the sequence log- logllikelihood, and incorporates crucial stabilization techniques for both the policy gradient objective and the KL-ddiergence regularizer.
4.1 THE SEQUENCF-LEVEI POLICY OBJECTIVE WITH ELBO APPROXIMATION
We begin by reformulating the RL objective to align with the nature of dLLM generation.
Sequence-Levv Level Objective. Instead of viewing each token as an independent action, we treat the generation of the entire sequence y as a single, atomic action. This naturally leads a sequence-level adaptation of the GRPO framework (Eq. (4) where the token-level summation is removed. The ob-
jective now depends on sequence-level importance ratio maly le) By substituting the intractable
me, (1)))
log-like ligllikelihodd log x (y(()) (z)) with its ELBO approximation C(y |r)), we obtain the sequence-level ratio for dLLM:
exp Lo (z)
P Dogq
exp(Coly(0))) Lo (x:))
(6)
exp Lase (2)10/z)))))))
Plugging this into the GRPO objective gives vanilla sequence- objective
C
Jovq (To) =E ( "...... ( no (:1z)
min 1 clipp( 1 e, 1, +1+))(())
(7)
G
While this sequence-level formulation correctly avoids the pitfall of splitting the ELBO at the token level, we found this vanilla formulation to practically unusable. The magnitude of the raw ELBO difference, ( (x) Lew (v (6) (x),, typically scales linearly with the sequence length L. The subsequent exponentiation results in astronomically large or infinitesimally small ratios, causing unstable optimization.
To address this instability, we draw inspiration from GSPO ZZeng et al. log-ratio by the sequence length L. This transforms the unstable, raw log-likelite likelihood difference into a stable, per-token scale. Our final, stabilized importance ratio is:
2025) and normalize the
exp (1
L EEchbol Lb (,)))))) (8)
Plgg exp (=(c)))))) (210)) (2))
Plugging this stabilized ratio into the sequence-level objective (Eq. enables effective training.
Empirical Validation. To validate our design choices of using a sequence-level action space with an ELBO approximation, we conduct complete ablation study on the Sudoku benchmark. We compare four variants: token-level actions with a mean-field likelihood, token-level with ELBO, sequence-level with mean-field, and our proposed sequence-level with ELBO. The precise mathe- matical formulations are detailed in Appendix C.I As shown in Fig. the results provide strong empirical support for our analysis:
"hhe KL. term omitted here for simplicity; detailed discussion is provided later in Section 4.2
4

---

Published as a conference paper at ICLR 2026
Rewards
Rewards
Edan-lodd- Nea Fide
Sapledd Mon
Saplord ELSO
0.6
0.4
0.2
0.2
1030 2000
4000
5000
500
1030
1500
2000
2400
5000
3500
4030
Training Steps
Training Steps
Figure 1: Training performance the Sudoku task under different action space (Token-level Sequence-level) and likelihood approxi- mations (Mean-field vs. ELBO). Our method (blue) combines a sequence-level action space with ELBO approximation, yielding the most stable and highest performance.
Figure 2: Training performance the Sudoku task with different KL-divergence estimators. The l2 estimator (blue) achieves stable and perior performance. The k estimator (orange) is highly unstable and collapses. while the 43 esti- mator (green) stagnates.
Mean-field is a poor proxy. Both mean-field variants (orange and green curve) fail to learn effectively, confirming that this approximation is fundamentally misaligned with the conditional denoising process.
Token-level ELBO is unstable. The Token-level + ELBO approach (red curve), while initially promising, suffers from high instability and eventual collapse. This highlights the inconsistency of breaking the ELBO's integrity across tokens.
Sequence-level ELBO is superior. Our proposed method (blue curve) is the only one to achieve fast, stable learning that converges to the highest reward, validating that the sequence-level action space paired with the holistic ELBO proxy is the correct and most effective approach.
4.2 STABLE KL-DIVERGENCE ESTIMATION
The complete GRPO objective includes KL-divergence term regularize policy updates against a reference policy. A common choice for token-level autoregressive models the h3 estimator [Schulman 2017). However, a direct application of the ha estimator our sequence-level objective is highly problematic. Its formulation, when approximating log log-likelsooods with the ELBO, is:
KL exxp exp (y (0
(o()))))) ) -
1 (c) - Ce(y |x))
As the formula shows, the K: estimator contains an exponential term, which reintroduces the unsta- ble problem similar to Eq. (6) at the sequence level. To cireumvent this exponential instability, we adopt the more robust l2 estimator (Schulman 2017), which is known to yield correct gradient for KL optimization (Tang Munos (0225)). Our practical and stable KL estimate becomes:
KL =
(Lo 3
(x))
(10)
Unlike the ky estimator, the by estimator is simple quadratic function of the ELBO difference. This polynomial form avoids the exponential term entirely, ensuring that the gradient signal from the KL regularizer remains stable and well-behaved, even for long sequences.
Empirical Validation. To demonstrate the critical impact of the KL estimator, we conduct an ablation study on the Sudoku task comparing the performance the kr, ko, and las estimators. More details can be referenced in Appendices C.2 and G.3 As shown in Fig. 2 the choice of estimator crucial for stable learning. The k3 estimator (green) fails to learn, with rewards stagnating at a low level, which is consistent with analysis of its unstable property. The kı estimator (orange) also highly unstable; while shows some initial progress, performance violently fluctuates before collapsing midway through training. In stark contrast, the h estimator (blue) enables
5

---

Published as a conference paper at ICLR 2026
Table Model performance on mathematics and planning benchmarks. For each task. we train a separate model. Countdown results with for LLaDA, diffu-GRPO, and wdl are from Zhao et al.
(2025 Tang et al. 2025), while other results are reproduced as detailed in Section 5.1
denotes
the improvement of ESPO over LLaDA or Dream model without reinforcement post-training
GSMBK(0)
()
Countdown
Sudoku
Model/ Seq Len
128
256
512
Avg.
128
256
512
Avg.
128
256
512
Avg.
128
256
512
Avg.
LLaDA
20.71
19.51
(Rth-GRPO(d))
74.6
78.1
81.2
78.0
34.9
36.6
41.7
57.7
33.21
31.31
37.11
33.91
26.7
24.1
15.9
22.2
wdl
77.2
80.8
$2.3
80.1
33.3
37.7
39.8
36.9
47.7
51.21
46.1
48.3
22.6
22.0
24.6
23.1
ESPO (ours)
82.3
43.4
81.6
82,0
92.7
+8.7
+6.1
+3.5
+6.1
+1.6
+3.8
+2.0
+2.5
+60.9
+62.5
+63.3
+62.3
+67.9
+68.5
+74.5
+70.3
Dream
75.8
81.3
80.7
79.3
38.2
45.7
48.0
44.0
8.5
7.8
17.4
11.2
9.3
2.1
14.0
8.5
(ifTI-GRPO(d))
77.0
81.9
81.7
80.2
39.4
46.9
48.9
45.1
27.3
27.7
57.5
30.8
64.4
69.7
51.1
61.7
wdl
45.8
28.9
29.5
ESPO (ours)
79.6
82.3
82.0
81.3
40.3
47.4
50.3
46.0
68.8
66.8
64.8
66.8
71.7
72.3
71.3
71.8
+3.8
+1.0
+1.3
+2.0
+2.1
+1.7
42.3
+2.0
+60.3
+59.0
+47.44
+55.6
+62.4
470.2
457.3
+63.3
stable and efficient learning, consistently improving and ultimately converging to the highest reward. This result empirically validates that the l/2 estimator the most robust and effective choice for our sequence-level framework.
5
EXPERIMENT
5.1 EXPERIMENTAL. SETUP
Models Tasks We apply our ESPO algorithm to two open-source dLLMs: LLaDA-SB-
Instruct (Nie et al. 2025) and Dream-7B-Instrucc (Ye et al. 2025). For reference, also report
evaluation results on LLa for comparison. Following prior work (Zhao et al., 2025 Tang
et al.
2025). we focus three categories of reasoning tasks: (i) Mathematics: GSM8K (oobe
et al.
2021a) and MATH (Hendrycks et al.) (02T), (ii) Coding: HumanEval (Chen et al. 2021
and MBPP (Austin et al. 2021b), EvalPlus (HumanEval+ and MBPP+) (Liu al. (223a) and (iii)
Planning: Countdown and Sudoku (Ye et al. 2024). For mathematical tasks, we train on the official training split of each dataset. For coding tasks, we follow Gong et al. (2025) and train on subset of AceCoder-87K (Zeng et al. 2025). For planning tasks (Countdown and Sudoku), we train on synthetic training data following Zhao al. (025).
Training To evaluate ESPO's effectiveness, we apply it directly to the models without additional task-specific SFT. For stable training, we adopt two standard variance reduction techniques: fol- lowing Zhu et al. (2025), we use antithetic sampling, which shares the same noise level and mask positions when estimating ELBO differences; and, inspired by Gong et al. (2025). we employ a
coupled-samplin scheme. All experiments use 2 Monte Carlo samples and policy update value
of μ 8. Additional details on variance-reduction strategies are provided in Appendix [ while further training configurations and extended results can be found in Appendices Eland
Following dI, we evaluate all benchmarks at generation lengths of 128, 256, and 512. For mathematics and coding tasks, we use the official evaluation scripts provided by the LLaDA and Dream repositories. respectively. For planning tasks, we use the evaluation code based on dl. To ensure fair comparison, we re-run those baselines whose reported settings differ from ours or whose results are missing for specific lengths (e.g., when length- 128 results were not originally reported).
Evaluation
5.2 BENCHMARK RESULTS
As shown in Table D and Table 2 our sequence-level RL algorithm, ESPO, consistently and significantly outperforms both the original models and prior teken-level RL baselines like diffu- GRPO (Zhao et al.) 2025) and wdl (Tang et al. 2025), while achieving performance on coding tasks that is comparable to 1... Notably, although our models are trained only on the sequence length of 256, the improvements generalize effectively other lengths.

---

Published as a conference paper at ICLR 2026
Table 2: Model performance on coding benchmarks. We train a single model and evaluate across multiple coding benchmarks (HumanEval and MBPP) at different sequence lengths. A de- notes the improvement of ESPO over LLaDA model without reinforcement post-training. ESPO consistently enhances the performance while even achieving competitive results compared with LLaDA-1.5, which was trained on a privately collected dataset at significantly larger scale.
(va()Evvlk()
MBPP(3)
Plus
Plus
Model/ Seq
Avg.
Avg.
Avg.
256
Avg.
41.5
37.0
)
25.6
36.0
50,0
37.2
22.0
29,9
37.2
29.7
34,8
36.6
38.0
36.5
31.8
34.8
39.4
35.3
wdl
34.7
38.4
38.4
37.2
29.9
29.9
32.9
30.9
38.0
37.2
34.4
36.5
31.8
32.8
35.6
33.4
ESPO (ours)
28.1
42.1
50.0
40.1
24.4
36.6
42.7
34.6
47.4
44.6
44.2
45.4
38,9
41.6
42.6
41.0
A
+1.3
+4.3
+1.2
+2.3
+1.2
+6.1
+1.2
+2.9
+9.2
+7.6
+6.0
+7.6
+2.1
+4.8
+4.8
+3.9
LLaDA-1.5
29.3
39.6
51.9
40.3
23.2
32,3
45.1
33.5
39.6
39.9
38.8
39.4
38.8
40.4
37.3
38.8
Countdown
Sudoku
0.8
0.6
Rewards
Rewarts
0.6
0.4
0.2
0.0
1000
2000
3000
5000
6000
S0033
0.0
2000
5000
4000
5000
5000
3050
3000
Training Steps
Training Steps
Figure 3: Training dynamics with different methods for Countdown and Sudoku tasks on LLaDA-8b-Instruck
Dominant Performance on Planning Tasks. The most dramatic improvements remain in plan- ning tasks. On Countdown, ESPO consistently outperforms the strongest baselines by 20-40 abso- lute points, while on the Sudoku tasks, the gains are up to over 60 points depending on the sequence length. This strongly validates core hypothesis: sequence-level objective is superior for tasks requiring holistic consistency, property that token-level optimization fails to capture effectively.
Consistent Gains on Mathematics and Coding. On established mathematics and coding bench- marks, the gains are more modest but still consistently positive. Note that the pre-existing knowledge of the base models acts as performance ceiling, limiting the maximum achievable gains through RI. fine-tuning alone. Despite this, our method reliably enhances performance, surpassing all previous token-level dLLM-RL methods averaged over sequence lengths, and comparable with LLaDA-1.5. This demonstrates the broad effectiveness of our approach even knowledge-intensin tasks.
5.3 TRAINING DYNAMICS
Fig. shows the training reward dynamics on Countdown and Sudoku, highlighting our method's superior performance. The difference most pronounced on Sudoku, where our sequence-level method rapidly converges to near-optimal policy, while the token-level dl and wdl baselines completely stagnate at low rewards. Similar trends can be observed the Countdown task.
5.4 ABLATION EXPERIMENTS
Number of Monte Carlo Samples We investigate the impact of the number of Monte Carlo (MC) samples used to estimate the ELBO. For this ablation, all other hyperparameters match those in main experiments. As shown in Fig. increasing the number of MC samples consistently improves training stability, but the magnitude of the effect varies by task. For signal-rich tasks such Sudoku, using more MC samples substantially accelerates convergence Maovergecce-M--1 -MC1 converges slowly, while MC=2 or MC=4 leads to noticeably faster progress. In contrast, for sparse-reward tasks such Countdown, the benefits of additional MC samples are much smaller.

---

Published as a conference paper at ICLR 2026
Countdown
Sudoku
018
0.8
0.6
Rewards
Rewards
0.6
0.4
0.4
0.2
012
MD+2
MC-1
MC-4
MC-4
Training Steps
Training Steps
Figure 4: Ablation study on the number of Monte Carlo samples for Countdown and Sudoku. We evaluate training performance with different MC sample counts (1, 2, 4), showing the effect of increased sampling on reward optimization.
Countdown
Sudoku
Rewards
0.5
Rewards
0.6
0.3
12
72
0.4
02
12
1
02
11
0.0
1900
2003
5000
4000
5000
0.0
2000
4000
6000
8010
10000
Training Steps
Training Steps
Figure 5: Ablation study on the policy update values (14 for Countdown and Sudoku. The reward curves illustrate performance across a range of μ values. While smaller values (e-g.. 8, 12) lead to faster initial convergence on Sudoku, the method is robust and achieves similarly high rewards across all settings for Countdown and Sudoku tasks.
Number of different policy update values In these experiments, we varied while keeping all other settings identical our main setup. Our method demonstrates remarkable robustness to the number of policy updates (/)) per data collection phase, as shown in Fig. 5 On both Countdown and Sudoku, our approach successfully scales to large μ values. converging to a similarly high reward across all settings, though smaller values (e.g., 8, 12) exhibit faster initial convergence on Sudoku. We hypothesize that this robustness is linked task complexity. While the policy for relatively simple planning tasks can be updated aggressively, more complex domains like mathematics and coding may have more brittle reward landscapes. In such cases, a large μ could lead to instability, making smaller, more conservative update frequencies safer and more effective choice.
5.5 DISCUSSION OF TRAINING EFFICIENCY
In reinforcement learning for dLLMs, the total training time can be roughly decomposed into two components: data generation (sampling) and policy updates (training). For dLLMs, the generation phase typically dominates: each step requires non-autoregressive denoising over the full sequence, preventing extensive reuse of KV cache. In contrast, the number of Monte Carlo samples M used for ELBO estimation only affects the policy update phase, which is less computationalyy intensive.
To quantify the computational cost, we use FLOPPs as a hardware-agnostic proxy. For model with N parameters and sequence length D, the forward and backward passes approximately require Ctorward 2ND D and Chackward = 4N D FLOPs (Kaplan et al. 2020). respectively. We obtain the total FLOPS per sample with coupled sampling 2N DN + 6/MM (see Appendix for details), where is the sampling step. μ is the policy update values, and M is the number of MC samples.
We further validate this analysis coding tasks using the same training configuration in Sec- tion 5.I except that the number of MC samples M is varied from 1, 2, and 4, The resulting theoret- ical FL.OPs and empirical wall-clock training time are summarized in Table 3
8

---

Published as a conference paper at ICLR 2026
Table Training cost under different numbers of Monte Carlo (MC) samples for 100 steps on the coding task. Following the setting in Section 5.1 the training parameters are set as denoising steps K 256 and policy updates value μ = 8 Experiments are conducted under H200 GPUs.
MC Samples (M)
1
2
4
Wall-clock Time (hrs)
5.61 (100%)
6.78 (121%)
9.06 (161 %)
Theoretical FLOPs
608 D (100%)
704N D (116%)
896N D (147%)
First, we observe that the theoretical FLOPs capture the overall trend of wall-clock time, consistent with the fact that dLLM inference compute-boume process. The small discrepancy may be attributed to factors such as GPU utilization, memory bandwidth, and communication overhead.
Second, focusing on the relative growth as M increases, our approach shows a much more moderate increase in training cost. Since generation costs are fixed (dominated by K), increasing M only adds to the policy update term, which minor when M is within a proper range. For example, increasing M from 1 to 4 raises the total FLOPs by only about 47% in our case, consistent with the observed wall-clock time. In contrast, for ELBO- based DPO algorithms such as VRPO (Zhu et al. 2025) their training time scales almost linearly with M (e.g., a 4x increase from M to M
6 RELATED WORK
Diffusion language models. dLLMs have recently emerged as a powerful approach for sequence
modeling, leveraging discrete diffusion (Austin et al. 202 Campbell et al., 2022 Meng et al.
2024 Zhao et al., 2024), or in particular masked diffusion models (Ou et al. 2023 Lou et al.
2024 Sahoo et al. 2024 Shi al. 2024). Distinct from autoregressive models (OpenAl] 2023)
that heavily rely on the left-to-right causal factorization of the sequence, dLLMMs instead operate
directly the sequence-level for each diffusion step and enable flexible order sampling (Kim al.)
2025) and parallel decoding (Ariolaa et al. (2025). While they have shown great promise in various
domains (Kwon et al. 2025 Ma et al. [2225] Liu et 2025c; Wu et al. 2025 Hu al. 2025 You
itt al. (0225) when scaled up (Nie et al. [2024]] Prabhudesai et al.) 2025 Ni & the team 2025 and
their adaptation downstream tasks with specific reward signals still has room for exploration.
RL for language models. Reinforcement learning (Schulman et al. 2017) has proven effective in enhancing language model performance, particularly when rewards can be obtained via automated verifiers, a paradigm known as reinforcement learning with verifiable rewards (RLVR) (OpenAT 2024). Methods such as GRPO (Shao et al., 2024, Guo et al.) 2025) and related works (Luo al.
2025 have improved reasoning capabilities and achieved strong performance across diverse tasks. However, directly applying these techniques to dLLMs challenging, primarily due to the difficulty of likelihood computation.
RL for diffusion language models. Except for continuous diffusion models (Wallace et al. 2024 Black et al. 2024: Liu et al. 2025a; McAllister al.] 2025), several works have explored leveraging RL to enhance the performance of diffusion language models. For general discrete diffusion mod- els. /Zhang et al. (2025) proposes a framework by target concrete score matching Venkatraman al.)
(024) proposes Relative Trajectory Balance to achieve unbiased sampling of posterior distribution, and Zekri & Boullé (2025) proposes score entropy policy optimization. For masked diffusion mod- els. LLadou (Huang et al. 2025) uses RL and trains an additional module to predict the position for decoding. It models each denoising step as an action, stores the entire trajectory, and optimizes the conditional probability of each intermediate state. However, in offline settings, optimizing over multi-step trajectories requires repeated forward and backward passes growing linearly with the number of denoising steps, which is prohibitive for large-scale models. Various works (Zhao et al.) 2025 Tang et al. 2025) have proposed mean-field surrogates of the likelihood that are efficient to compute to enhance efficiency. Yet, the approximations significantly trade off with quality and hin- der the multi-step generation capability of the model. Our work instead develops a discrete diffusion RL objective from a principled sequence- level likelihood evaluation via efficient MC estimate, dif- ferent from Yang et al. (2025): Gong et al. (2025) that evaluates the likelihood on the token-level heuristics.

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
In this work, we identified the fundamental incompatibility between autoregressive RL. objectives and the non-autoregreessive nature of dLLMs, and proposed ESPO, a principled sequence-level rein- forcement learning framework that leverages the ELBO as a tractable proxy for sequence likelihood. By treating sequence generation as a single action and introducing stabilized importance ratios and KL regularization, ESPO eliminates inconsistencies in prior token-level approaches and enables ro- bust and efficient sarge-scale training. Extensive experiments on math, coding. and planning bench- marks demonstrate that our method consistently outperforms existing di dI..M-RI. baselines. Our results establish sequence-level optimization as a principled and empirically effective paradigm for RL in diffusion language models.
ACKNOWLEDGMENTS
This work was supported by the Beijing Major Science and Technology Project under Contract no. Z251 100008425002; the National Natural Science Foundation of China (Nos. 62522609, 924701181 18); the Beijing Natural Science Foundation (No. L247030); and the fund for building world-class universities (disciplines) of Renmin University of China.
Personally, thank Jiajun Liu and Shusheng Xu for their generous assistance throughout this project.
ETHICS STATEMENT
This work is purely methodological and does not involve human subjects or sensitive data. Ex- periments are conducted publicly available benchmarks, and results reported in aggregate. We highlight that while our techniques improve efficiency and accuracy, they should be applied responsibly avoid potential misuse.
REPRODUCIBILTTY STATEMENT
We have open-sourced our code in https: ctt.://GG//.AAAAC.C...... Comprehensive explanation and details of our theory and experiments can be found in Appendices Cto to
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025.
Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang and Jennifer Wortman Vaughan (eds.), Advances Neural In- formation Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 17981-17993, 2021a.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, al. Program synthesis with large language models. ArXiv preprint, abs/2108.00733 2021b.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Tom Rainforth, George Deligiannidis, and A. Doucet. A continuous time framework for discrete denoising models. In Advances in Neural Information Processing Systems, 2022.
10

---

Published as a conference paper at ICLR 2026
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In IEEE/CVP Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans. LA. USA, June 18-24, 2022, pp. 11305-11315, IEEE, 2022. doi: 10.1109/ CNPPR52688.0022.01100
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, 0iss/207.00337/ 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv preprint, abs/2110.14168. 2021a.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2110. 2021b.
DeepMind. Gemini diffusion, 2025.
URL
gemi mimrini--nn difiiiiddffffio...
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code gen- eration, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv preprint, abs/2501. 12948, 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florinn Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
33: Annual Conference on Neural
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S. Abdelfattah, Jae-sun Seo, Zhiru Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided diffusion. ArXiv preprint, abs/2505.21 21675700..447...55 2025.
Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, and Guo-Jun Qi. Reinforcing the diffusion chain of lateral thought with diffusion language models. ArXiv preprint, abs/2505. 2025.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang. Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-codes technical report, 2024.
Inception Labs, Samar Khanna, Siddhant Kharbanda. Shufan Li, Harshit Varma, Eric Wang. Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. Mereury: Ultra-fast language models based on diffusion, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
11

---

Published as a conference paper at ICLR 2026
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. 0rXi::25002.6668,, 2025.
arXiv preprint
Taeyoun Kwon, Junhyuk Ahn, Taegeun Yun, Heeju Jwa, Yoonchae Choi, Siwon Park, Nam-Joon Kim, Jangchan Kim, Hyun Gon Ryu, and Hyuk-Jae Lee. Whisfusion: Parallel asr decoding via diffusion transformer, 2025.
Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai- Wei Chang, and Aditya Grover. Lavida: A large diffusion language model for multimodal understanding. ArXiv preprint, abs/2505. 16839, 2025.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023a.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint FrXXiv2505.054700 2025a.
Xiao Liu, Hao Yu. Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang. Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang- Agentbench: Evaluating lims agents, 2023b.
Xiaoran Liu, Zhigeng Liu, Zengfeng Huang, Qipeng Guo. Ziwei He, and Xipeng Qiu. Longllada: Unlocking long context capabilities diffusion Ilms, 2025b.
Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. 2025c.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi. Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl -i-zero-like training: critical perspective. In Conference on Language Modeling (COLM), 2025d.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning, ICML, 2024.
Michael Luo, Sijun Tan, Roy Huang. Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa. and Ion Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level, 2025. Notion Blog.
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. ArXiv preprint, abs/2505.1578.. 2025.
David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa. Flow matching policy gradients. arXiv preprint arXiv:2507.21053, 2025.
Chenlin Meng, Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: General- ized score matching for discrete data, 2023.
Jinjie Ni and the team. Diffusion language models are super data learners, 2025. Notion Blog.
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models text. ArXiv preprint, abs/2410.18514 2024.
Shen Nie, Fenggi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen. and Chongxuan Li. Large language diffusion models. arXiv preprint arXiv:2502. 09992, 2025.
12

---

Published as a conference paper at ICLR 2026
OpenAl. Gpt-4 technical report. ArXiv preprint, abs/2303.08774,
OpenAl, Sep
2024.
URL
https::/openai.com /ndxxx
learning -a--t------- rewson--nnn--o weth-witttt-11
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan
Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. ArXiv preprint, tbs/2406.003336 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems 35: Anmual Conference Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffu- sion beats autoregressive in data-constrained settings, 2025.
Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language mod- els. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: An- nual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
John Schulman. kl-approx blogpost.
http://jocch..ne//lll///------
html 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, Nod/100.00637, 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300,, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnau Doucet, and Michalis K. Titsias. Simplified and gener- alized masked diffusion for discrete data. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Uirich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural formation Processing Systems 38: Anmual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth In- ternational Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: www.oepenvvveeveeeo
/ ew net./ www.:nee/ffformooom formmmrrrmmm// wow.mmmmddd4www2.cc...mmm 4=4wWww2..22222
Yuxuan Song, Zheng Zhang, Cheng Luo. Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang. Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Yonghui Wu, and Hao Zhou. Seed diffusion: large-scale diffusion language model with high-speed inference, 2025.
Xiaohang Tang. Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. wdl: Weighted policy opti- mization for reasoning in diffusion language models, 2025.
Yunhao Tang and Rémi Munos. On a few pitfalls in kl divergence gradient estimation for rl. arXiv preprint arXiv:2506.09M 2025a.
Yunhao Tang and Rémi Munos. On few pitfalls in kl divergence gradient estimation for rl, 2025b.
13

---

Published as a conference paper at ICLR 2026
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, Alexandre Adam, Jarrid Rector- Brooks, Yoshua Bengio, Glen Berseth, and Nikolay Malkin. Amortizing intractable inference in diffusion models for vision, language, and control. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Process- ing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Leandro Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https://gi
com hwtp://gggggggggg/////as 2020.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 8228-8238, June 2024.
Ronald J Williams. Simple statistical gradient-followw algorithms for connectionist reinforcement learning. Machine learning, 8:229 256, 1992.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao. Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast dllm: Training-free free acceleration of diffusion I1m by enabling kv cache and parallel decoding. ArXiv preprint, abs/2505. 22661...............222....11288 618, 2025.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. ArXiv preprint, abs/2505.15809 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023.
Jiacheng Ye, Jiahui Gao. Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong.
Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2024.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong, Dream 7b, 2025.
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint 16X5 2025a.
Zebin You, Jingyang Ou, Xiaolu Zhang, Jun Hu, Jun Zhou, and Chongxuan Li. Effective and efficient masked image generation models, 2025b.
Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding. ArXiv preprint, htts/2505.16990, 2025.
Oussama Zekri and Nicolas Boullé, Fine-tuning discrete diffusion models with policy gradient methods. ArXiv preprint, abs/2502.01384, 2025.
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. ArXiv preprint, abs/2502.01718, 2025.
Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang, James Thornton, Zijing Ou, Joshua Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. ArXiv preprint, abs/2504.16443 2025.
Lingxiao Zhao, Xueying Ding. Lijun Yu, and Leman Akoglu. Improving and unifying discrrtee&coomiinous-time discrete denoising diffusion. ArXiv preprint, abs/2402.03301,, 2024.
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. ArXiv preprint, 0bs/2504.12216,, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong- Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy op- timization, 2025.
14

---

Published as a conference paper at ICLR 2026
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. ArXiv preprint, abs/2505.19225, 2025.
A THE USE OF LARGE LANGUAGE MODELS
During the writing of this work. Large Language Models (LLMs) were used as auxiliary tools to assist with language polishing, grammar checking, and improving the readability tables and figure captions. We will take full responsibility for all parts of the paper.
B FROM GENERIC RL PRINCIPLES TO DLLM RL FORMULATIONS
We focus on the output reward setting, where the reward R(x,, y) is evaluated only upon the completion of the entire sequence y. This is the mainstream setup for reasoning tasks (e.g., Math, Coding) in post-training.
Scope of Derivation:
B.1
GENERIC RL ORJECTIVE AND GRADIENT DERIVATIONS
The standard RL objective defined to maximize the expected accumulated reward:
(TT) = Ey-ro s) [[(r,, y)]
(11)
Using the log-derivative trick and introducing Importance Sampling (IS) to evaluate updates under the behavior policy Told, the gradient can be derived as below (Williams, 1992):
To(ulæ))
VoJ (TTO = V Byr-saalllaee
R(x, y) (y(a))
(12)
me (y((a))
= E (
Vo log no
(13)
[y)a(y(
Kp (y x))
Ve log To (rll))))(e
(14)
Told (ylz))
1S Ratio
where A(z, y) denotes the advantage function (typically R(x, v) - b(x)) introduced for variance reduction.
For Autoregressive models, the sequence likelihood factorizes as πyyl²) II (, vt).
Substituting this into Eq. (14 yields the full sequence-level gradient:
1 Il Motal y
1I To (o
Eys-ras
V log To A(xo,v))
(15)
unstable
The cumulative product term (in blue) introduces severe variance and numerical instability (explod- ing vanishing gradients) as d increases, rendering Eq. (15) impractical for long sequences.
To address this, Token-level RL. methods (e.g.. PPO (Schulman et al., (017). GRPO (Shao et al. 2024)) approximate the full ratio by decomposing it into per-token ratios. While practical imple- mentations utilize clipping mechanisms or KL penalties to constrain policy updates and ensure the validity of this approximation, we omit these regularizers here to focus on the core gradient structure:
d To x
JPPONGRPO Eys-rau
Vo log To(y" 1z, A((,0)))
(16)
d
Tota
15

---

Published as a conference paper at ICLR 2026
Conversely, Sequence-level RL methods (e.g., GSPO (Zheng al. 2025) argue that applying off- policy correction at the token level for a sequence- reward is heuristic. They retain the sequence- level perspective but stabilize the ratio via length normalization. Similarly, omitting clipping and KL terms for simplicity, the gradient formulated as:
à
I (o(''> y
V JaSpo Byn-Tas
Vo log no ( ()((,)))
(17)
d
Tola 22, yrt))
B.2 ADAPTING RL OBJECTIVES TO DIFFUSION LANGUAGE MODELS
Applying the derived principles to dLLMs involves a critical choice regarding the action space and likelihood approximation.
The most direct approach is to formulate the RL problem at the trajectory level as a multi-step Markov decision process, where each denoising step constitutes action (Huang et al.) (025)- Here, a "trajectory" (Jr, 3o0) represents the evolution from pure noise y/ to clean data yo- In this case, we factorizes over diffusion timesteps i. as n To(Ut-1 v1) rather than token index i I[= T(y"(x, "*)
In this view, the RL gradient involves summing over all T timesteps, where T equals the number of sampling steps (typically T [50, 1000).. Taking the simplest online policy gradient as an example:
TTaj
E
Vo log no (3t- 1y), R(),y
(18)
Requires T gradient evaluations
While theoretically sound, this approach suffers from a prohibitive computational bottleneck due the nature of the generative process. Crucially, unlike Autoregressive models where all ditional probabilities log ( () can be computed in parallel via a single network forward pass (enabled by causal masking), dLLMs fundamentally require T sequential network evalu- ations to compute the full trajectory probability. Consequently, calculating gradients for V Vinm requires backpropagating through separate forward passes for every sample. This scales the com- pute cost linearly with T (e.g., 50x ~ 1000x more expensive than a single step), rendering it infeasible for large-scale post-training.
Given this constraint, we can optimize the final generated sequence directly using the ELBO proxy for the intractable log-likelihood. This computationally feasible because the ELBO can be efficiently estimated via Monte Carlo (MC) sampling, requiring only K forward passes. In practice the MC sample size is typically small (K ss 4), offering a direct trade-off between the quality of the likelihood proxy and computational efficiency. This leads the fundamental divergence between token-level and sequence-level formulations:
Lequence-Level Validity: The ELBO Lo(y|z) is mathematically rigorous variational lower bound for the joint sequence log-likelihood: (o(lx) < log (y)zz). Therefore, substituting Lo(y)a) into the Sequence-level gradient formulation (Eq. (17h) preserves the mathematical in- tegrity of the objective. The sequence-level ratio acts as a valid proxy for the global importance weight.
Token-Level Invalidity: In contrast, the token-decomposed ELBO term (y(1z) is not a valid approximation for the autoregressive conditional probability log ne rryyyll [z, x) required by the
Token-level PPO/GRPO formulation (Eq. (1)). The AR conditional Te(y [a, x*) strictly de- pends on past tokens y (unidirectional), whereas the ELBO term 2 (z)(z) relies on the bidirec- tional context of the denoising process. Forcing this mismatched proxy into Eq. (I6) introduces an unknown inconsistency, explaining the instability observed in token-level baselines.
In summary, ESPO is principled algorithm where the intractable likelihood is replaced by rea- sonable variational bound (ELBO), constituting the only valid RL formulation for dLLMs under computational constraints.
16

---

Published as a conference paper at ICLR 2026
C DETAILS FOR ABLATION STUDY IN SECTION
C.I ABLATION FOR ACTION SPACE AND LIKELIHOOD APPROXIMATION
To better analyze the objectives used in the ablation study ( Fig. D. we start by revisiting the GRPO
formulation in Eq. (4). For clarity, we focus on its form for single data sample (æ, y of length L. This yields the single-sample objective:
( = (+)
Σ min (t.00, 1 1,,+))))
(19)
where pA:(0)
mo 12.
b.n
auu (y b.
To highlight the core mechanism, we omit the clipping operator and obtain the simplified form:
J(,,y(110) (
A0
(20)
Building the simplified form in Eq. do not provide autoregressive conditional probabilities directly, prior work replaces or approximates pli(i) with diffusion- based likelihood surrogates. This gives rise to four variants used in our ablation:
20
extend the formulation dLLMMs Since dLLMs
Token-level + Mean-field (Orange Curve): This approach, introduced by di (Zhao et al. 2025). applies a token-wise importance ratio using the mean-field approximation.
A0 po pas (y (()z))
Á
J(x
Σ
exp (logpo()A.00) log Pa.. ml (v (000))))
L
L
k=1
(21)
Sequence-level Mean-field (Green Curve): This baseline extends the mean-field approach to the sequence level.
[Ir, 10 = Ato)
exp
log pe ((00)))))) - log Pe.. (200))))))
(22)
Token-level ELBO (Red Curve): This method replaces the mean-field term with the token's contribution to the ELBO, but problematically computes the ratio for each token individually. Gong al. (2025): Yang et al. (2025) follows this approach.
Jz A Σ exp (1)))))2)) Co. (1()))))))))
(23)
Sequence-level ELBO (Ours, Blue Curve): Our proposed method. It uses the ELBO as a proxy for the entire sequence log-likelihood and normalizes the log-ratio for stability, as de- scribed in Eq. (7) and Eq. (8)
00, Ate exp
(o()) Law (v (2/)))))
T,
L (b) -
= -(t exp
cb (10)
(24)
C.2 ABLATION FOR KL DIVERGENCE ESTIMATOR
The choice KL-divergence estimator is critical for stable sequence- optimization. We analyze the three common kl estimators. As demonstrated in Fig. of the main paper, only the k estimator enables stable and effective learning. We provide a detailed analysis of their pitfalls here, supported by new visualizations of their training dynamics in Fig.
17

---

Published as a conference paper at ICLR 2026
Pitfalls in the ky estimator. The hy estimator commonly adopted in GRPO is given by
TT(
(T)
KLAa(TT Tret) =
Byn-ro
log Teae
(25)
(err(
where and Tref are the target and reference policies. However, we argue that the k3 estimator has several pitfalls.
First, the 33 estimator introduces training instability. For a model with the output instead directly parameterizes the log policy, logits, the k3 estimator requires take the exponent:
(26) where [(y) approximates log π (ty). We empirically find that during RL training C(y) may incur instability which will be magnified by the exponent, i.e., the first term in Eq. 26 As shown in Fig. F righht column), the k: estimate and its gradient norm are characterized by extreme, infrequent spikes, which destabilize and derail the entire training process.
KILK3 = Ey jexp (Co(y) Lee(l)) + Co(y) (y)r
Besides, while the lg estimator itself is an unbiased estimate of KL-divergence, its gradient is not (Tang & Munos, 2025a). In fact, it is shown that its gradient is instead unbiased estimate of reverse-Kie KI. (Tang & Munos 2025a):
E VILLL {rae]
VKL( mo) + E VKIL(To: Tret)
(27)
Therefore, the optimization using k3 estimator will exploit gradients that indeed optimize towards the reverse-KL, which will empirically lead to different converged policy with bounded model capacity.
Pitfalls in the & estimator. Another alternative is the original k1 estimator for KL-divergence, which is given by
(To, Teet) = Byyway log (8())
(28)
Wref (v)
However, by taking the gradient w.r. 0 on Eq. 28 it is proved(Tang & Munos 2025a) that the gradient equals zero, which means the kjestimator has no signal of the KL constraint. This is empirically confirmed in Fig. (right) of the main paper, where the kn policy (orange) initially learns but then catastrophically collapses. Fig. 6 (left column) further illustrates this: the KI. estimate plummets into negative values, and the gradient norm grows uncontrollably, reflecting a complete loss of stability as the policy drifts unconstrained.
The li estimator. We propose to employ the ko estimator
[
KLaz(TO, Tret) = Eyy
(logTo(y) - log mrer(1))?
(29)
which leads to
2 (Colv) (x)r (ce()))))))))))
KL
E Ey-xo
(30)
under the log parameterization. The ky estimator takes the form of an MSE loss that does not involve an exponential term. As shown in Fig. ( column), both the KL estimate and its gradient norm exceptionally stable, smooth, and well-behaved throughout training. Furthermore, the gradient of the ky estimator is indeed unbiased (Tang & Munos 2025a):
E
V VKKTLa((T (T (Ter)
VKL Fret)
(31)
as opposed to the biased estimate in ky estimator.
This analysis, confirms that k is the only robust and theoretically sound choice for our high- variance, sequence-level optimization framework.
D VARIANCE REDUCTION
As discussed in Section 2.1 there are two equivalent formulations of the ELBO. Empirically, Eq.
exhibits lower variance than Eq. ( (Ou et al., 2024 Nie et al. 2025). Therefore, we adopt Eq. (3) estimator for the ELBO. To further stabilize training, incorporate two variance reduction
techniques: antithetic sampling through mask sharing (Zhu et al. 2025) and coupled sampling
(Gong et al. 2025), which we describe below.
18

---

Published as a conference paper at ICLR 2026
XL Earmmate
Batbsatt (83)
Exthoate (13)
24
Traiidng Styge
Traiiin Stppe
Tratiing Stig
(krm
Gradient Norr
Gradiett Norm
0.43
Figure 6: Comparison of training dynamics with different KL divergence estimators. The top row illustrates the KL estimates, while the bottom row displays the gradient norms for the k1, kg, and hy estimators. The 1j estimator (left) lacks effective constraints, leading to model collapse. The ka estimator (right) suffers from severe instability and gradient spikes. contrast, the l2 estimator (middle) demonstrates superior stability and consistent gradient norms throughout training.
Antithetic Sampling through Mask Sharing. Antithetic sampling through mask sharing is used when computing the difference between two ELBO terms, such as Co Co (y/z) in Eq. (8) and Co 1z) (v(=)) in Eq. (10). Concretely, share the sampled timesteps and masked positions between the Monte Carlo estimators of the two policies, thereby reducing variance through negative correlation.
Coupled Sampling, Coupled sampling was originally proposed for the ELBO in the form of Eq. GGog al. 2025), but we adapt to Eq. (3). Note that Eq. (3) equivalent to
B
1.2..... 1...,,(
L.) By mo (na) ylx)),
(32)
where
1sy = M] Moggoo 11, 1> 0,
lo (o(y),4,3)z) &
(33)
0.
= 0.
Based on this formulation, we introduce a complementary masking strategy. For each sampled mask Wt we construct complementary mask in such that the two masks partition the token set: every token masked in is unmasked in ÿt. and vice versa. We then average the two losses, which also equivalent:
Co(y, 1.)x) + Co(in I. 1.
E Bur.
(34)
2
This construction guarantees that every token contributes at least once to the learning signal. Further, the estimator achieves lower variance and yields more stable optimization objective.
E EXPERIMENT DETAILS
As described in Section 5.1 train separate models for each of the GSM8K, Math, Sudoku, and Countdown tasks using their respective training datasets. For code tasks, we train a unified model on AceCoder-874 87K (Zeng et al. 2025) and evaluate it on four benchmarks: HumanEval, HumanEval- Plus, MBPP, and MEPP-Plus For ablation experiments, we conducted only on LLaDA- Insttrrtttttt-88 model.
All models are trained with a maximum sequence length of 256, while evaluation performed at sequence lengths of 128, 256, and 512 to better assess length generalization.
19

---

Published as a conference paper at ICLR 2026
E.1 INFERENCE SETTING
For simpler planning tasks (Sudoku and Countdown), we unmask 2 tokens per denoising step, re- sulting L/2 total denoising steps (where L 256 is the sequence length). For other tasks, the number of denoising steps is set equal to the sequence length to improve performance.
Sampling strategies are model-specifie:
LLaDA: We employ low-confidence sampling (Chang al. 2022) combined with semi- autoregressive decoding strategy (Arriola et al. 2025] Nie et al. 2025), with block length set 32 for all tasks. Training and evaluation share most sampling settings, except that the temper- ature is 0.9 during training and O (greedy decoding) during evaluation, following the evaluation codebase (Nie al. 2025)
Dream: We use top negative entropy remasking and pure diffusion sampling, without semi- autoregressive decoding (Ye et al., 2025). Temperature is set to 0.9 during training and 0.1 during evaluation, while other sampling settings remain consistent.
E.2 TRAINING DETAILS
All reinforcement learning training is conducted using the TRL library (von Werra al. 2020
Crramtter-Efficien Fine-Tuning For GSM8K, Math, Countdown, and Sudoku, we apply LoRA (Hu et al., 2022) with rank r = 128 and scaling factor a = 64. For code tasks, full pa- rameter fine-tuning is used to maximize performance.
Optimization Policy update value is set to = 8, and the number of Monte Carlo (MC) samples M = 2 for computational efficiency. Models are optimized using AdamW (Loshchilov Hutter, 2019) with $1 = 0.9, 32 = 0.99. A constant learning rate schedule is used. For LoRA-based tasks, the learning rate
10-6 with weight decay 0.01 and gradient clipping 0.2. For code tasks, learning rate is 1 10-6, weight decay 0.1, and gradient clipping 0.8.
Batching We set the group size G in Eq. ( and total batch size according to the difficulty of the tasks: GSM8K, Countdown, Sudoku use G 6 and total batch size 96, Math uses G 16 and batch size 256, and code tasks use G = 10 and batch size 160. Gradient accumulation is applied to enlarge the effective batch size.
Training Steps Models evaluated when the reward curve converges. Code tasks trained for 2k steps, Math for 3k steps, LLaDA planning tasks for 10k steps, and Dream planning tasks for 8k steps.
Special Notes on Sudoku The Sudoku setup in Dream (Ye et al., 2025) differs from dl (Zhao (et al. 2025). The prompt and data format in Dream are inconsistent with di (see Appendix H.I.. Additionally, Dream restricts the generation length to 24 tokens, encouraging direct answers without intermediate reasoning. contrast, di uses longer generation lengths (128/256/512), allowing the model to produce both intermediate reasoning steps and final answers.
In the main text, we report results under the dl setting, as it better reflects the ability of models to perform multi-step reasoning when guided by reinforcement learning algorithm. For complete- ness, we also present model performance under the Dream setting in Table
Special Notes on Coding It is important to note that for coding tasks, we employed full fine- tuning, unlike the LoRA setting used for Math and Planning. In this regime, we observed significant instability with the baseline methods. Specifically, wdl shows to be highly sensitive to hyperpa- rameters; using default settings resulted in exploding gradient norms (reaching 10 ~ 10*) and immediate training collapse. While were able to stabilize wdl by carefully tuning the hyper- parameters, it still suffered from performance degradation on certain metrics compared to the base model, shown in Table In contrast, ESPO remained stable and effective.
20

---

Published as a conference paper at ICLR 2026
Table 4: Model performance on Sudoku under the Dream setting (generation length 24). Unlike the di setting, the Dream setting restricts generation to 24 tokens, encouraging models to output answers directly without intermediate reasoning.
Model
Accuracy (%)
Dream-7B 7a-ann-B---sstuuu
98.1
diffu-GRPO (dI)
96.0
ESPO (ours)
98.0
F DETAILS OF TRAINING FLOPS
Recent advances in dLLMMs (Wu et al. of KV cache, but it will lead the loss of downstream performance. Therefore, analysis focuses on the vanilla case without KV cache.
2025 Ma et al. 2025 Liu et al. 2025c enable partial usage
As we mentioned in Section 5.5 for model with N parameters and sequence length D, we use 4N D to approximate the forward and backward passes FLOPS
Cforward 2N D and Chackward
according to Kaplan al. 22200). Combining the costs of generation and policy updates, obtain the naive formula of total FLOPs per sample:
Fiota
K Crorward + μMM (Cfowward + Cbackward) = 2N + +pM)),
(35)
where K is the sampling step. μ is the policy update values, and M is the number of MC samples. When using the coupled-sampling technique ,we will double the FLOPS of policy updates step, which leads to:
Fieal = K Clowwadd + 2pM ( (Crorwad + Cbackward) = 2N -6pM),
(36)
Which indicates increasing M only adds to the policy update term (6μM in Eq. (36). When 6uM, the overall training cost grows mildly with M.
>
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 DETAILED TRAINING DYNAMICS OF SUDOKU
As shown in Fig. the training dynamics exhibit clear phase transition characterized by sharp rise in reward. During the initial stage, the model explores with low rewards and stable gradients. However, coinciding exactly with the sudden performance jump (around step 2500), we observe sharp spike in the gradient norm and a rapid surge KI. divergence. These signals indicate that the model has successfully discovered the strict logical rules of Sudoku, triggering rapid restructuring of its policy and a decisive shift away from the reference model to exploit the newly discovered solution mechanism.
G.2 TRAINING CURVES ON MATHEMATICAL BENCHMARKS
As illustrated in Fig. 8 the training reward trends for ESPO and the baselines are relatively stable and similar, indicating that evaluated methods can robustly fit the training distribution given the reward signals. However, observed in prior work (Liu et al. 2025d). the final reward the training set often exhibits a weak correlation with the accuracy on the held-out validation set due to the potential for overfitting, While the training rewards are similar, ESPO's 's superior performance on the test set (as reported in Table [ demonstrates that our sequence-level optimization leads to better generalization rather than memorization of the training data.
G.3 ABLATION STUDY ON KL ESTIMATOR WITH TOKEN-LEVEL BASELINES
To rigorously investigate whether the superior performance of ESPO is primarily driven by the robust k2 KL estimator rather than our sequence-leve framework, we conducted an additional ablation study the token-level baseline. The wdl method (Tang al. 2025) reformulates the original RI.
21

---

Published as a conference paper at ICLR 2026
Reward
Reward Sid
0.25
0.20
Training Steps
Training Steps
Grad Norm
KL
0.10
0.06
0.04
0.02
2000 3005
4200
3000
3000 Training Steps
Training Steps
Figure 7: Detailed training dynamics of the Sudoku Task on LLaDA-SB-Instre We track reward, reward standard deviation, gradient norm, and KI. divergence to analyze the learning process of our ESPO method.
Math
GSM8K
1.0
0.6
Rewards
0.4
Rewards
066
0.4
0.2
will
0.0
500
1000
1500
2003
2500
3900
0.0
500
1000
1500
2000
2500
3000
3500
Training Steps
Training Steps
Figure 8: Training dynamics with different methods for MATH and GSM8K benchmarks on L.L.DA-SB-nstrrue
objective weighted likelihood objective (similar AWR) and does involve an explicit KL estimation. Therefore, we only conducted ablations for diffu-GRPO (d1 Zhao et al. (2025) in the sudoku task, replacing its original unstable kg estimator with the robust k2 estimator in Table
It shows that simply applying the K2 estimator to the token-level baseline (dl + k2) only yields performance comparable the original baseline (dl + k3). Both variants stagnate low success rate (~~ 20 26%). In stark contrast, ESPO achieves a dramatic improvement (>> 80%).
22

---

Published as a conference paper at ICLR 2026
Table 5: Performance comparison on Sudoku (LLaDA-SB-Instruec) We compare the original dl (using k3), dl adapted with the 12 estimator, and our ESPO method.
Method Seq Len
128
256
512
diffu-GRPO (dl) k3 (Original)
26.7
24.1
15.9
diffu-GRPO (dl) K2 (Ablation)
26.3
23.1
18.8
ESPO (Ours) 1/2
92.7
84.7
80.5
H PROMPTS AND OUTPUTS
H.1 PROMPT TEMPLATES FOR SUDOKU
Below, we demonstrate the one-shot prompts used for the Sudoku task.
One-shot Prompt (d1-style)
Please solve the following 4x4 Sudoku puzzle. The puzzle provided 16-character string reading left-to-right, top-to-bottom, where '0' represents empty cells.
Rules:
Fill empty cells with digits 1-4.
Each row must contain digits 1-4 exactly once.
Fach column must contain digits 1-4 exactly once.
Each 2x2 box must contain digits 1-4 exactly once.
Example:
Puzzle: 0401 01110002110000
This puzzle grid looks like this:
4
0
1
0
0
3
3
1
Solution: 2431312412224312
The solved grid looks like this:
1
2
4 3 2
3
Important: Your solution must be COMPLETE 16-character string with only the digits 1-4, repre- senting your final solved grid.
Respond in this exact format:
<reason >ng> ng>
Your step-by-step solving process
</ </reasoning>
<answer>
[16-character solution string]
</ //answo>>>
Now, solve the following Sudoku puzzle: 1240300020140100
23

---

Published as a conference paper at ICLR 2026
One-shot Prompt (Dream-style)
immstaart I>system
You are an Al assistant skilled at solving Sudoku puzzles. < im.end]
<I limsstart >uueer
Fill the positions where the values are 0 in a 4x4 grid with digits 1-4 that each column, each row, and each of the four 2x2 subgrids that compose the grid contains all of the digits from to 4.
Input:
0321
0030
0400
2100
Output:
4321
1234
3412
2143
Input:
0001
2100
1430
3000
Output:
im_end|>
im.start >assistant
H.2 QUALITATIVE EXAMPLES ON SUDOKU TASK
LLaDA + d1 (Baseline)
<reasonings
To solve this 4x4 Sudoku puzzle, need to fill in the empty cells with digits 1-4 such that each row, column, and 2x2 box contains all digits exactly We start by examining the given digits and trying to them according to Sudoku rules.
1. The first row has the 0' and the 4's in the first and fourth columns. The second has the 4's in the first and fourth columns.
The third row has the in the second column.
The fourth row has the in the fourth column.
5. The fifth row has the 1 in the first column.
6. The fourth row has the in the first column.
The first row has the in the fourth column.
Using this information, we can start filling in the grid. We need to ensure that each row, column, and 2x2 box contains the digits 1-4 exactly once.
</reasoning>
<answer>
12340
40221
30114
</ </answer>
24

---

Published as a conference paper at ICLR 2026
LLaDA ESPO (Ours)
<reasoning>
To solve the 4x4 Sudoku puzzle, we need to fill in the grid with digits 1-4 such that cach row, column, and 2x2 box contains all the digits exactly once. We can start by filling in the cells with the given digits and check for conflicts.
First, let's fill in the grid with the given digits:
3204
0030
1400
2001
Next, let's try filling fill in the remaining with digits 1-4, ensuring each row, column, and 2x2 box contains all the digits once.
After trying different combinations, we find a valid solution:
3 3 14
412
1423
2341
This solution satisfies all the rules: each row, column, and box contains all digits 1-4 exactly once.
</reasoning>
<answer>
3214413214232341
</answer>
25

---

Published as a conference paper at ICLR 2026
H.3 QUALITATIVE EXAMPLES ON COUNTDOWN TASK
LLaDA+d1 (Baseline)
Target: 94 Numbers: [77, 73, 98]
<reasoning>
To creale an arithmetic expression that evaluates to 94 using the numbers [77. 73, 98], we need to find the correct combination of operations (+, *) that will inbalance the the larger number 98 and the smaller numbers 94. </reasoning>
<answer>
77473-98
</answer>
X(Incorrect, evaluates to 5523)
LEaDA+ESPO (Ours) - Target: 67 - Numbers: [72, 92, 47]
<reasoning>
To create an arithmetic expression evaluates exactly using the numbers [72, 92, 47J, we need to consider the operations +, -, *, and Let's start by examining the largest numbers and see they can combine to give us target 67.
add 92 and 72, we get 164, which is than 67. we subtract 92 and 72, we get 20, which
than 67.
If we add 92 and 47, we get 139, which is more than 67. 4. If we subtract 72 and 47, we get 25, which is less than 67.
combine 92 and 72, we get 164, which more than 67.
Combining 92 and 47 will the closest answer to 67. </reasoning
92
</answer>
(CCoreet, 20 + 47 67)
26


Reviews:
[{"review_id": "QjQK8MDpAa", "reviewer": "Reviewer_Twof", "summary": "The paper studies the problem of reinforcement learning (RL) to diffusion LLMs. Token-level objectives for autoregressive model (e.g., GRPO) require token log-likelihoods that are intractable to compute for diffusion models. The paper proposes ESPO, a sequence-level policy optimization method that treats generating the entire completion as one action and replaces the intractable sequence log-likelihood with an ELBO proxy. ESPO stabilizes training by (i) normalizing the ELBO log-ratio by sequence length and (ii) using a k2 (quadratic) KL estimator instead of the unstable exponential k3 estimator, alongside variance-reduction tricks (antithetic/coupled masking). ESPO is empirically evaluated with LLaDA-8B-Instruct and Dream-7B-Instruct on math (GSM8K, MATH), coding (HumanEval/MBPP), and planning (Countdown, Sudoku). ESPO consistently beats diffu-GRPO and wd1. Ablations show sequence-level+ELBO is the only stable formulation among tested variants and a training-cost analysis shows FLOPs/time grow mildly with Monte-Carlo samples because generation dominates compute.", "questions": "* Length normalization is shown to have unintended consequences on the reasoning performance for AR LLMs. I am curious if it impacts the long reasoning behaviors in your experiments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper explains the problem setup and why token-level importance ratios lack a valid probabilistic interpretation for dLLMs fairly well. The shortcomings of existing methods also makes the motivations quite clear.\n\n* I like the idea of moving to a sequence level objective and using an ELBO-based ratio avoiding heuristic token surrogates.\n\n* The performance of the method is tested across a variety of tasks and two different base models and shows consistent improvement over the baselines.\n\n* The paper is also generally quite well written.", "weaknesses": "* ESPO optimizes an ELBO difference, not the true sequence likelihood ratio but the paper does not quantify how ELBO tightness affects policy improvement or bias across tasks.\n* The paper misses some closely related prior work on RL fine-tuning of diffusion language models [1, 2]. I believe a comparison to these baselines would be critical.\n\n[1] Venkatraman et al., 2024. Amortizing intractable inference in diffusion models for vision, language, and control.\n\n[2] Zekri and Boullé, 2025. Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uzjR6kdBsp", "reviewer": "Reviewer_8CJA", "summary": "The paper proposes ESPO, an RL framework tailored to diffusion LLMs which treats the whole completion as a single action. Experiments on LLaDA-8B-Instruct and Dream-7B-Instruct across math (GSM8K, MATH), coding (HumanEval/MBPP and EvalPlus), and planning (Countdown, Sudoku) show consistent gains versus token-level d1/wd1 baselines.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Using a sequence-level action space makes the method very simple\n- Experiments show strong improvements over the baselines which do not treat the entire sequence as an action", "weaknesses": "- Seems like a straightforward application of GSPO to diffusion models novelty-wise\n- Not clear why per-token evaluation is necessarily bad for diffusion LLMs specifically - is it true for all LLMs (as GSPO claims) or just diffusion LLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IInhCGz50Q", "reviewer": "Reviewer_sYDr", "summary": "This paper proposes a reinforcement learning framework for training diffusion language models. The approach builds upon group sequence-level importance sampling ratios for diffusion language models and incorporates an additional KL regularization term (referred to as K2-type regularization by the authors). The method is evaluated empirically on several tasks: Countdown, mathematical reasoning, and coding problems, demonstrating performance improvements over baseline approaches.", "questions": "**Justification for sequence-level formulation:** The paper does not provide clear insight into why the proposed sequence-level formulation outperforms the token-level formulation. Is the advantage due to reduced bias, reduced variance in the policy gradient estimates, or both? A quantitative analysis comparing the bias-variance tradeoffs of both formulations would strengthen the paper's claims.\n\n**Hyperparameter selection:** How was the clipping parameter $\\epsilon$ in Proximal Policy Optimization chosen? Were separate hyperparameter searches conducted for the token-level and sequence-level formulations, or was the same value used for both? This is important for ensuring fair comparison between the two approaches.\n\n**Role of KL regularization:** The results in Figure 2 raise several questions about KL regularization:\n\nThe figure suggests that KL regularization enables a significant performance breakthrough in later training stages, which is unexpected. Typically, KL regularization is understood to improve training stability rather than drive performance gains. Can the authors explain this phenomenon?\n- In standard language model fine-tuning on reasoning tasks, KL regularization is often minimal or omitted entirely. Why is it critical in this setting?\n- For clarity: which type of KL regularization (forward, reverse, or the K2-type mentioned) is used in Figure 1?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper is well-written with a logical structure that makes the technical content accessible. The progression from problem formulation to methodology to experimental validation is easy to follow.\n\n- The experimental evaluation demonstrates notable improvements on both the Countdown and math coding tasks, suggesting the proposed approach is effective for the target applications.", "weaknesses": "-  The proposed method largely combines existing techniques from prior work (Zheng et al., 2025; Tang & Munos, 2025b) without introducing new algorithmic components or theoretical insights. The contribution appears primarily incremental, adapting established methods to the diffusion language model setting rather than developing new approaches tailored to the unique characteristics of these models.\n- While the empirical improvements are encouraging, the paper does not address the fundamental question of what constitutes an appropriate RL formulation for diffusion language models. The current approach treats the problem as one of better approximating log-probabilities, but a more principled direction would be to re-derive the policy gradient theorem specifically for diffusion language models from first principles. Critical questions remain unanswered: What is the proper form of policy gradients for diffusion language models? Does the \"log-probability\" term even appear in such gradients, or should the formulation take a fundamentally different form? Without addressing these foundational issues, the work risks building on potentially unsuitable assumptions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGAtBa4v4p", "reviewer": "Reviewer_b9P2", "summary": "This work proposes ELBO-based Sequence-level Policy Optimization (ESPO), a principled framework tailored for dLLMs. ESPO defines the entire sequence generation as an atomic action (avoiding token-level decomposition), employs the evidence lower bound (ELBO) as a tractable proxy for sequence-level likelihood, and utilizes a robust $k_2$ estimator for KL divergence (free from exponential instability).\n\nExtensive experiments on models like LLaDA-8B-Instruct and Dream-7B-Instruct across tasks (mathematical reasoning, coding, planning) demonstrate ESPO’s superiority: it achieves up to 60-point absolute improvements in planning tasks (e.g., Sudoku), stable gains in knowledge-intensive tasks (e.g., GSM8K, HumanEval), and superior training efficiency (only 47% cost increase when MC samples double).", "questions": "Please see questions in Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. Clearly identifies the mismatch issue under the dLLM framework, highlighting that existing RL algorithms struggle to compute token-level importance sampling, and proposes a sequence-level alternative.\n\n2. The experiments are thorough, including extensive validation and ablation studies across both math reasoning and agent tasks, demonstrating the effectiveness of the proposed method.\n\n3. The paper is well-organized, with clear presentation and coherent argumentation.", "weaknesses": "1. For the math reasoning experiments, could the authors provide training curves? The improvement over the baseline appears limited.\n\n2. Why are there no experiments on Dream-7B-Instruct with the d1 baseline?\n\n3. Why are the results on coding benchmark such as HumanEval and MBPP missing baseline comparisons?\n\n4. The performance on Sudoku shows a sudden change. Could the authors provide a deeper explanation or analysis?\n\n5. From Figure 2, the observed improvements seem mainly driven by $k_2$ estimator. Could the authors include wd1 and d1 with $k_2$ estimator on Sudoku as additional comparisons?\n\nI am willing to raise my score should the authors provide satisfactory responses and clarifications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)