Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
Authors: Kevin Rojas
Jiahe Lin
Kashif Rasul
Anderson Schneider
Yuriy Nevmyvaka
Molei Tao
Wei Deng
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
discrete diffusion
diffusion language models
Primary area: generative models
TL;DR: We introduce an improved reinforcement learning algorithm for diffusion language models

Abstract: Diffusion language models (DLMs) enable parallel, order-agnostic generation with iterative refinement, offering a flexible alternative to autoregressive large language models (LLMs). However, adapting reinforcement learning (RL) fine-tuning to DLMs remains an open challenge because of the intractable likelihood. Pioneering work such as diffu-GRPO estimated token-level likelihoods via one-step unmasking. While computationally efficient, this approach is severely biased. A more principled foundation lies in sequence-level likelihoods, where the evidence lower bound (ELBO) serves as a surrogate. Yet, despite this clean mathematical connection, ELBO-based methods have seen limited adoption due to the prohibitive cost of likelihood evaluation. In this work, we revisit ELBO estimation and disentangle its sources of variance. This decomposition motivates reducing variance through fast, deterministic integral approximations along a few pivotal dimensions. Building on this insight, we introduce **Group Diffusion Policy Optimization (GDPO)**, a new RL algorithm tailored for DLMs. GDPO leverages simple yet effective *Semi-deterministic Monte Carlo* schemes to mitigate the variance explosion of ELBO estimators under vanilla double Monte Carlo sampling, yielding a provably lower-variance estimator under tight evaluation budgets. Empirically, GDPO achieves consistent gains over pretrained checkpoints and outperforms diffu-GRPO, one of the state-of-the-art baselines, on the majority of math, reasoning, and coding benchmarks.

Published as a conference paper at ICLR 2026
IMPROVING REASONING FOR DIFFUSION LANGUAGE MODELS VIA GROUP DIFFUSION POLICY OPTIMIZATION
Kevin Rojas"; Jiahe Lin", Kashif Rasul,, Anderson Schneider" Yuriy Nevmyvaka', Molei Tao't, Wei Deng"1
!Georgia Institute of Technology ML. Research, Morgan Stanley [kevin. rojas, mt fggttch. edu, weideng 056 gmail com
ABSTRACT
Diffusion language models (DLMs) enable parallel, order-agnostic generation with iterative refinement, offering a flexible alternative to autoregressive large language models (LLMs). However, adapting reinforcement learning (RL. fine- tuning to DLMs remains an open challenge because the intractable likelihood. Pioneering work such diffu-GRPO (Zhao et al., 2025) estimated token-level likelihoods via one-step unmasking. While computationally efficient, this ap- proach is severely biased. A more principled foundation lies in sequence-level likelihoods, where the evidence lower bound (ELBO) serves as a surrogate. Yet, despite this clean mathematical connection, ELBO-based methods have seen lim- ited adoption due to the prohibitive cost likelihood evaluation. In this work, we revisit ELBO estimation and disentangle its sources of variance. This decom- position motivates reducing variance through fast, deterministic integral approx- imations along a few pivotal dimensions. Building on this insight, we introduce Group Diffusion Policy Optimization (GDPO), a new RL algorithm tailored for DLMs. GDPO leverages simple yet effective Semi-deterministic Monte Carlo schemes mitigate the variance explosion of ELBO estimators under vanilla double Monte Carlo sampling, yielding a provably lower-variance estimator un- der tight evaluation budgets. Empirically, GDPO achieves consistent gains over pretrained checkpoints and outperforms diffu-GRPP one of the state-of-the-ar baselines, on the majority of math, reasoning, and coding benchmarks.
INTRODUCTION
Large language models (LLMs) (Radford et al., 2018; 2019; Brown al., 2020; Achiam et al., 2023; Dubey et al., 2024; Team, 2025) have have revolutionized modern science by providing ex- ceptionally general-purposs representations and abstractions. Their training typically proceeds in
Kevin did this work while interning at Morgan Stanley. i Correspondence.
HumanEval
50
80.9
30.0
27.5
70
50.6
92.1
40
39.0
50
37.8
36.2
42.2
40
30
70+
32
22.
LLaDA
diffu-GRPO (d1)
GDPO (Ours)
Figure 1: Across reasoning, planning, and coding tasks, GDPO algorithm for diffusion language models - using the best of 128/256/512 generations - significantly outperforms the LLaDA base- line and prior RL methods such diffu-GRPO.

---

Published as a conference paper at ICLR 2026
two stages: a pretraining stage, where vast corpora are used to optimize the next-token prediction objective and endow the model with broad world knowledge and linguistic representations: and post-training stage, where alignment techniques adapt the raw model outputs for improved reli- ability and usability (Bai et al., 2022). Among alignment methods, reinforcement learning (RL.) (OpenAI, 2024) has emerged especially promising for post-tuning L.LMs on complex tasks with well-defined reward signals. DeepSeekMath (Shao et al., 2024) advances this direction by scaling and stabilizing RL through Group Relative Policy Optimization (GRPO), showing that RL can re- duce the computational burden of alignment (Guo et al., 2025; Team et al., 2025) while substantially improving LLMs on reasoning-inte inteensive tasks such as mathematics (Cobbe al., 2021; Hendrycks et al., 2021a), planning (Ye et al., 2025a; Liu et al., 2025a), coding (Austin et al., 2021b), and multimodality (Venkatraman et al., 2024). Nevertheless, key limitations remain: inference is slow, generation is constrained to left-to-right order, and early mistakes cannot be revised, often leading to error propagation.
By contrast, discrete diffusion models (Campbell et al., 2022: Lou et al., 2024; Shi et al., 2024; Gat et al., 2024; Sahoo et al., 2024; Nie et al., 2025; Ye et al., 2025b) - often referred to as Diffusion Language Models (DLMs) offer greater flexibility and versatility. DLMs support faster inference (Arriola et al., 2025; Khanna et al., 2025; Song et al., 2025), iterative refinement through token revisions and remasking (Wang et al., 2025), knowledge transfer via adaptation from autoregressive models (Gong et al., 2025a), and demonstrate superior performance to autoregressive L.MMs in low- data regimes (Prabhudesai al., 2025). These advantages have motivated growing efforts to adapt RL methods originally developed for LLMs to DLMs (Zhao et al., 2025; Zhu et al., 2025; Gong al., 2025b; Zekri & Boullé, 2025; Shankar, 2025; Huang et al., 2025). However, the absence of straightforward autoregressive structure complicates likelihood estimation at both the token and sequence levels- quantities that many RL objectives fundamentally rely on.
Previous works have extended GRPO (Zhao et al., 2025; Gong et al., 2025b;b; Tang et al., 2025) and considered different heuristics to approximate the token-level likelihood; such methods were computationally scalable, but their connection to the mathematical foundations of DLMS remain unclear. A different approach considered extending DPO (Zhu et al., 2025), however, their method required many network evaluations, which limits its applicability.
In this paper, we introduce Group Diffusion Policy Optimization (GDPO), a novel RL algorithm built to enhance reasoning for diffusion language models (DLMs). We analyze the variance de- composition of the sequence-level ELBO, which clarifies why prior double Monte Carlo estimators lead to a large variance issue and tend to be computationally expensive. Motivated by these find- ings, we propose fast yet effective integral approximation strategies hased on a Semi-detterministic Monte Carlo scheme. This approach enables GDPO to solve complex reasoning tasks for DLMs while provably reducing variance under tight evaluation budgets. Empirically, GDPO consistently improves upon pretrained checkpoints and outperforms state-of-the-art baselines, including diffu- GRPO (Zhao et al., 2025), across diverse math, reasoning, and coding benchmarks.
PRELIMINARIES
2.1 MASKED DIFFUSION LANGUAGE MODELS (MDMs)
Diffusion models (Ho et al., 2020: Song et al., 2021) have demonstrated remarkable success in continuous domains such as image synthesis, but their extension to discrete spaces like text re- mains less explored. To address this gap, discrete diffusion models (Austin et al., 2021a; Lou et al.. 2024; Shi et al., 2024; Sahoo et al., 2024) often referred to as Diffusion Language Mod- els (DLMs) inttrducc masking noise progressively corrupt sequences and are trained to model the marginal distribution of the induced reverse dynamics. In what follows, we formally define the forward and reverse processes along with the training objectives that characterize DLMs.
Forward process. Given a clean sequence data yo Tdata and timestamp t E [0, 1], the forward process Yt Tepo- (o) factorizes as
(o(vilvd) = Cat((1 - t)e,, + tews). Ty (bo) = II (oo
2

---

Published as a conference paper at ICLR 2026
where Evj is the one-hot vector that encodes the position of token tó and eM is the one-hot vector for the mask token M; and Cat(-) denotes the categorical distribution. Thus, each coordinate y/ is independently replaced by the mask token with probability and otherwise remains unchanged.
Reverse process. The reverse process aims to reconstruct the original sequence yo from a cor- rupted sequence yt- Given probabilistic prediction of the token yå- write down the transition for any 8 < t as:
(slo(Ual31 vc) = II gape(ellle vb), gere(dClue v6) = Cat(Cgt);
y + M, Cat (fem v0)), vi = M.
Denoising objective. With the linear noise schedule and dime-independent conditional probabili- ties (Ou et al., 2025), the reverse transition is often approximated by no (yó (y:), trained via simple training objective (Zhu et al., 2025):
-ooorPas Be [0.1] yy
1 M] log To (vblvn))
Notably, the loss function of diffusion language models provides a lower bound for the likelihood known as evidence lower bound (ELBO):
CElBO(blg) = E0- (yo
1 Σ
M] log To (111,. s)
log ()lg)
(2)
where q usually denotes a prompt and y is its answer. DLMs are conceptually similar to BERT (Devlin et al., 2019) in that both rely token masking train language representations. However, they differ in that BERT masks a fixed proportion of tokens and predicts them in a single step, while DLMs adopt a time-varying masking schedule and iteratively denoise from full corruption, thus yielding true generative model.
2.2 REINFORCEMENT LEARNING WITHOUT VALUE NETWORKS
Policy gradients (Williams, 1992) have become the workhorse for post-training large language mod- els (LLMs). Among them, proximal policy optimization (PPO) (Schulman et al., 2017) remains the most widely used. However, PPO's reliance on value network for advantage estimation inevitably increases both computational cost and training instability.
Group Relative Policy Optimization (GRPO) (Shao al., 2024) addresses this limitation by elim- inating the value network. Instead, it leverages Monte Carlo (MC) estimator constructed from mul- tiple sampled answers. For a given prompt g and candidate answers is:
the GRPO objective
Mal
1 G
CRROO (0)
E,, Ey Tall
min (riAA,, clip(r'; 1 - - -,1+))AA) 3KL (TT (Tef))
Ug
where the importance ratio and normalized advantage are defined as:
To (v v5 Tola U/gle 19 x* Ag Rg mean Ri,..., RG)
(v)
std R1,... RG)
with Rg = R(q, By) denoting the sequence-lewe reward. Notably, although likelihoods are defined at the token level, rewards are assigned only at the sequence level.
Despite their wide adoption, LLMs face key limits: slow inference, rigid left-to-right generation, and error propagation. DLMs mitigate these with parallel, iterative refinement, but their flexibility complicates likelihood estimation and challenges RL-based post-training.
Diffu-GRPO: A pioneering effort in fine-tuning diffusion language models made by Zhao et al. (2025), who proposed Diffu- an adaptation of GRPO for masked diffusion. They ap- proximated the sequence-level likelihood via fast but coarse mean-field network evaluation, and
3

---

Published as a conference paper at ICLR 2026
introduced practical scheme for this approximation. Specifically, their method starts by perturb- ing the input prompt q with random noise to obtain q'. They then consider fully masked sequence q @ M @ ... + M, where @ denotes concatenation and M is the masked token, they perform single-step unmasking to estimate
lo @ M..... M).
This approach has two key virtues: it yields likelihood estimates for every token, and it does so with only one forward pass of the network, ensuring computational efficiency. Empirically, Diffu- GRPO achieves consistent performance gains across wide range of tasks. However, the one- step unmasking in the mean-field manner introduces significant bias: since tokens are generated sequentially, important token correlations discarded.
IMPROVING REASONING VIA GROUP DIFFUSION POLICY OPTIMIZATION
GRPO improves computational efficiency and training stability by estimating advantages from group statistics rather than training value network. autoregressive LLMs, its effectiveness relies on two factors: (1) accurate sequence likelihoods, naturally supported by the left-to-right factorization, and (2) token-level importance ratios. Diffusion Language Models (DLMs), however, the order- agnostic generation paradigm (Ou al., 2025; Kim al., 2025) renders both sequence likelihoods and token-level ratios intractable.
3.1 REVISITING SEQUENCE-LEVER LIKELIHOOD FOR DLMS
To address these challenges, much of the field has focused on fast but coarse token-level approxi- mations. Token-level methods leverage per-token probabilities to provide fine-grained control over model updates and act as a stabilizing force during training. To make token-level training feasible despite intractable likelihoods, Zhao et al. (2025) introduced heuristic mean-field approximations. Although efficient and empirically effective, these methods fail to capture sequential dependencies and often overweight individual tokens. Gong et al. (2025b) later improved the approximation by incorporating two complementary random timesteps, yet a general solution remains elusive.
In contrast, sequence-level objectives provide more faithful training signals but are difficult to apply in DLMs, since the order-agnostic generation paradigm precludes exact likelihood evaluation (Ou et al., 2025). We review the evidence lower bound (ELBO) surrogate of the likelihood:
Σ lvt = M] log To(v'lu..)) s) 1=1
Ee-2d(0 By
< log Ttl44),
which provides a principled avenue for extending sequence-level RL methods to DLMs. However, the computational cost of this substitution remains unclear.
Variance-Cost Dilemma. Although more principled, sequence-level objectives face fundamental trade-off: accurate likelihood estimates demand expensive network evaluations, while cheaper ap- proximations suffer from high variance or bias. For instance, Nie et al. (2025) report needing up to 128 samples for reliable estimates, incurring prohibitive cost; Zhu et al. (2025) reduces this to eight evaluations, yet the overhead remains substantial, and systematic understanding of the variance is still lacking. This tension highlights a central challenge:
Designing estimators that are both efficient and low-variance remains an open problem.
3.2 DISENTANGLING VARIANCE IN ELBO
To tackle the variance-cost dilemma, begin by analyzing the different sources of variance in approximating the ELBO. From Eq.(2), two distinct sources of randomness emerge: (1) Random Time: sampling t, which determines the overall masking level; and (2) Random Masking: lecting which tokens are masked given that ratio, introducing additional variance. Our analysis disentangles the contribution of each source to the variance of the loss function.
To investigate this we leverage 1000 different prompts pulled from the Open Web dataset. Figure reports the mean and variance of the loss functions of time, along with the percentage of variance
4

---

Published as a conference paper at ICLR 2026
(a) Variance Decomposition
(b) Mean Loss vs Time
(c) Loss Variance vs Time
1.0
Mean 5D (m-18)
Mean 5D (mm18)
80
1000
17500
Pecerlig (4)
15000
96.2%
Mean:
800
20000
40
7500
5000
200
Fandam Time
Random Masking
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Time 01
Time 10)
We plot the mean and variance the loss functions as function of the noise level (a) We observe that most of the variance comes from picking the random time (b) The loss function follows simple, predictable shape across many prompts. (c) The loss variance varies highly at the end but stabilizes for most times.
Figure 2:
attributed to each source. detailed derivation can found in Appendix A. Several key observa- tions are given next.
Random time dominates the variance: As shown in Figure 2(a), the majority of variance arises from sampling the timestamps that control masking ratios. This is intuitive: varying the ratio dras- tically changes the mauting from nearly unmasked to fully masked-whice produces large disparities in the loss and inflates variance.
The loss curve exhibits a simple structure: In Figure 2(b), loss as a function time reveals smooth, clear structure. Approximating the ELBO boils down to computing the area under this curve, indicating that the problem is naturally suited for deterministic integral approximation to suppress the variance.
Variance across timesteps stable: As shown in Figure 2(c), variance peaks near = 0, stabilizes across intermediate masking ratios, and decays to zero inputs become fully masked. This aligns with observations from Zhu et al. (2025) and indicates that only a small number samples are needed for the inner integral.
3.3 VARIANCE REDUCTION VIA SEMI-DETERMINISTLE MONTE CARLO
To achieve low-variance estimates under tight evaluation budgets, we limit naive Monte Carlo sam- pling and adopt deterministic integration methods to avoid the slow MC convergence O(N 1(2).
1/2).
Deterministic time: Motivated by the observation in Figure 2(a), instead of considering the problem as double Monte Carlo problem, we consider it to be a time integral to eliminate the large variance caused by random time:
CELBO(UL4) = 1' Eggeomet-l))
1
Σ
I
lvi = M] log To (v x) dt I log (v)4))
(4)
t
i=1
Numerical quadrature: We further approximate this integral using standard quadrature with N points, then our estimate is of the form:
CFLBOVVLg)
y Mol Toly" (w/
(5)
1=1
where v!! ~ TI.. Tr., (-18) and W are the associated weights and the inner expectation approx-
imated using Monte-Carke estimates.
Due to the ceterministic-stochhee stochastic nature in the integration, we refer to it as a Semi-deterministic Monte Carlo (SDMC) scheme.
5

---

Published as a conference paper at ICLR 2026
This representation serves several advantages, firstly, it is guaranteed to approximate the ELBO when N and are large enough. Secondly, by fixing the time points in the first integral we significantly reduce the variance of our estimator. Thirdly, the loss function is plotted for several different prompts, the loss function has a simple shape, specifically, it is strictly increasing and convex, which makes it well-suited for integral approx- imations via quadratures. Furthermore as observed in Figure 2 (c) the variance remains rel- atively constant across many noise levels, which allows for stable computations and results.
observed in Figure (b), where
In practice, we opt for utilizing simple and ef- fective Gaussian Quadratures, which are known to have fast convergence rates (Dahlquist &
200
Double-MC
SD-MC
Björck, 2008). Furthermore, based our variance analysis we employ a single Monte- Carlo estimate for the inner integral. Conse- quently, the total number of network evalua- tions per likelihood computation matches ex- actly the number of quadrature points N.
Mean Absolute Error
150
125
75
To evaluate method in controlled set- ting, we draw 1000 samples from the Open- Web dataset and estimate sequence likelihoods using both Monte Carlo and Gaussian quadra- ture with varying numbers of function evalu- ations. For each sample, we repeat the esti- mation 16 times and measure both the devia- tion from ground-trute -ound approximation (com- puted with 1024 Monte Carlo samples) and the variance of the estimator. As shown in Fig- estimators consistently
50
25
8 Number of Function Evaluations (NFEs)
Figure 3: Estimation error and variance for Dou- ble Monte Carlo vs our Semi-deterministic Monte Carlo method. SD-MC achieves lower bias and variance, with most benefits obtained using only 2-3 points.
ure 3, quadrature- exhibit lower bias and variance, yielding accu- rate estimates of the ELBO. Notably, most of the gains achieved with few
quadrature points, which adopt in practice.
With the flexibility to reconcile efficiency and statistical accuracy, we are ready to introduce Group Diffusion Policy Optimization (GDPO), a policy gradient method tailored to diffusion language models with group statistics. Compared with Diffu-GRPO (Zhao et al., 2025), which employs ef- ficient but crude estimates of token likelihoods, GDPO uses sequence-level likelihoods, which made efficient through the SDMC scheme. This reformulation of the importance weights from the token-level to the sequence-level has the added benefit of preserving the semantics of the advan- tage estimates, and the ELBO-basd -base objective fits naturally within the discrete diffusion framework while retaining the merits of GRPO-style updates. Formally, the GDPO loss is defined as:
1 G
1
CGDPO = Ey ~
min clipprr e, + e) -KK (
Fyl
where the importance weights and advantage estimates are both done at the sequence level:
(x:) exp(CELBO (x) exp( Cold (Ng (x))
Ag = Rg mean (Rı,..., RG).
Here CEL represents the ELBO evaluated under the old policy and Rg R(g. Mg). and we utilize unnormalized advantage estimates to avoid the bias (Liu et al., 2025b). The use of the ELBO is in spirit similar to GSPO (Zheng et al., 2025).
3.4 OVERVIEW OF THEORETICAL RESULTS
We provide brief theoretical analysis for the proposed ELBO estimate that is based on Semi- deterministic Monte Carlo, with an emphasis on its asymptotic error bounds when the total number of Monte Carlo samples K and the number of integration points N become large. Note that

---

Published as a conference paper at ICLR 2026
Algorithm GDPO: Group Diffusion Policy Optimization for diffusion language models (DLMs), with ELBO estimated via a Semi-deterministic Monte Carlo (SDMC) scheme.
Require:
Reference model Teff distribution over prompts D, completions per prompt G, inner updates μ., quadrature points and weights {(tx: w2)
1: Initialize from a reference model wy f- Wref
2: while not converged do
3: Teau f T
4: Draw prompt sample q ~
5: Generate completions By ~ Tama (. q), E [G]
6: Estimate reward rg and advantage A (Te using (3.(3) for each Vy.
7:
for 1..... do
For each gradient update
8:
for = 1, 1 G do
9:
LELBO Ug (1) - 0 for n =
10:
do CELBO 1)) + CELBO(yal1) + Wn {(Tes yo q.th tp)
D Evaluate quadrature over time
12:
end for end for
14: Evaluate GDPO objective in (6) using
Optimize Te AdamW. 16: end for
17:
end while
18:
return .e
"classic" alternative to this estimator would be double Monte Carlo one, and its error bound (in the form of MSE) would scale at O (NRR). For our proposed estimator, the analysis points to the following results:
decomposition of the MSE; the mean squared error of the estimator can be decomposed into the sum of Monte Carlo variance and the square of the integration bias, which resem- bles the classic variance-bias decomposition of statistical estimators;
rates under very general conditions: the variance term scales O( N ); with generic integration scheme (e.g., the Riemann sum !), the squared integration bias scales as O( N faster rate with additional assumptions the log-likelihood: with N-dependent decay condition on its variance, the variance term would scale faster at the rate of O N );
quadrature rule for integration can make the bias practically negligible: when the inte- grand is twice continuously differentiable and therefore one can use quadrature to perform integration, the squared integration bias would scale either at the rate O x ) or O Ne depending on the exact integration scheme used. As such, this term becomes practically negligible and the variance term becomes the dominating one.
The upshot is that under certain regularity assumptions of the lieelikldoood, the proposed SDMC estimator can attain faster rate than a generic double Monte Carlo estimator due to the deterministic integration. All details are deferred to Appendix B.
Table Asymptotic Error Bounds in relation to Integration Points N and Monte Carlo Samples K.
Setting
Variance O(/NN)
Bias O(L/N²)
General conditions / Riemann sum
Additional assumption & smoothness/ Quadrature
O(L/2²K)
O(L/N*) O(1/N*)
As noted above, to perform integration using quadrature, the integrand needs be sufficiently smooth. To that end, properties of the integrand are further investigated. particular, by relat- ing the integrand in (5) to KL form, it can be shown that such KL form is indefinitely
In case of Riemann sum, the integration is approximated as 9())dt EM (le.). where c, any value within the nth interval: w, N when the intervals equally spaced.

---

Published as a conference paper at ICLR 2026
differentiable (namely, living in ())). Further, under some additional assumptions on the likelihood ratio associated with the data distribution and the forward diffusion process, one can prove that the integrand is convex and monotone in Such conditions can be understood conceptually as follows: masking more tokens corresponds to removing more information, and the penalty grows as masking increases. The implication of these results are two-fold: (1) given the structure of the integrand, the standard quadrature rules are well-suited for this problem; combined with results from the error bound analysis, this justifies the observation that our ELBO estimator exhibits faster convergence and lower variance than the double Monte Carlo one; and (2) the convex shape empirically observed in Figure 2(b) supports the theoretical claim under the additional assumptions, albeit verifying these assumptions can be non-trivial. Details for this part of the results are in Appendix C.
4 EXPERIMENTS
We conduct comprehensive set of experiments. As base model use MaADA-&B-Instruce Zhu et al. (2025) which a open sourced DLM that has been tuned to follow instructions, but no specific post-training. We investigate the effect of applying GDPO with our SDMC estimator.
Tasks: We conduct experiments on mathematical reasoning, planning and coding benchmarks. For (1) Mathematical reasoning: we the GSM8K (Cobbe al., 2021) which contains grade school math problems, and MATH500 (Lightman et al., 2023), containing 500 problems drawn from the MATH dataset (Hendrycks al., 2021b). (2) Planning: this includes two tasks: 4x4 Sudoku puzzles, and Countdown in which given 3 numbers and a target the model must use arithmetic operations to reach the target. (3) Coding: we use the HumanEval (Chen al., 2021), a benchmark consisting of 164 manually designed Python algorithmic challenges and sanitized MBPP (Austin et al., 2021b), which contains 257 crowd-sourced Python programming tasks.
Table 2: Model performance on Mathematics and Planning Benchmarks based on N quadrature points. Green is the best performing model.
GSMSK
MATH500
Countdown
Sudoku
Model
128
256
512
128
256
512
128
256
512
128
256
512
LLaDA 8B Instruct
68.7
76.7
78.2
26.0
32.4
36.2
20.7
19.5
16.0
11.7
6.7
5.5
GRPO
72.6
79.8
81.9
33.2
37.2
39.2
33.2
31.3
37.1
18.4
12.9
11.0
+SFP=
73.2
81.1
82.1
33.8
38.6
40.2
34.8
32.0
42.2
22.1
16.7
9.5
+wDI
N/A
80.8
82.3
N/A
34.4
39.0
N/A
51.2
46.1
NIA
25.2
24.2
+FT GDPO
78.39
82.79
84,99
33.2
39.6
41.4
64.45
74.22
80.86
27.69
26.42
26.17
4.1 MAIN RESULTS
GDPO consistently improves the checkpoint and outperforms diffu-GRPO Table 2 demon- strates the performance of the baseline model, well other token level approaches. Our results demonstrate that we can better improve the baseline than the token-based algorithms.
GDPO outperforms most baselines w/ SFT. As shown in Table 2, GDPO with quadrature points surpasses existing RL baselines after few epochs of supervised fine-tuning (SFT), highlighting its strong ability to enhance reasoning performance.
Table 3: Model performance on Coding with
N 3 quadrature points. Green best.
HumanEval
MBPP
Model
128 256 512 128 256 512
GDPO enhances reasoning in coding tasks. We fine-tune the model on the KodCode-Light-RL. 10K dataset (Xu et al., 2025), which spans wide range of coding problems at varying difficulty lev- els, each validated through unit tests. We found that N 3 led to slightly better results than N = 2, and report GDPO with N 3 in Table 3. The results demonstrate that GDPO with
LLaDA-8B- Instruct
27.4 35.5 37.8 36,2 41.2 40.4
diffu-GRPO
29.3 39.0
34.8 42.0 45.5 41.6
GDPO
26.2 39.6 39.0 43.6 50.6 47.1
8

---

Published as a conference paper at ICLR 2026
3 quadrature points consistently improves performance across most baselines. Most strikingly, the MBPP benchmark, RL fine-tuning achieves a substantial 10% accuracy gain over the pretrained model without SFT.
GDPO improves performance beyond the training sequence length. Building on observations by Zhao et al. (2025) that Diffu-GRPO enhances generalization to longer contexts, find that GDPO achieves this effect to an even greater extent. On 512-token sequences, GDPO consistently outperforms all baselines while largely preserving performance shorter lengths. We attribute this to the use of sequence-level likelihoods, which promote more uniform improvements across token positions, in contrast to token-level methods that retain generation- biases, noted by Gong et al. (2025b).
GDPO is computationally efficient Notably we are able to obtain these remarkable results training on only 2 H100 GPUs, although train- ing can be accelerated with 8 H100 GPUs. This remarkable as practitioners often have a lim- ited computational budget and GDPO opens the opportunities to such needs.
GDPO Test Accuracy different ELBO estimators
SDMC
SDMC-2
Double-MC-4
Test Accumai Counstown 23G
30-
Importance of the ELBO approximation: To demonstrate that accurately approximating the likelihood is of vital importance fix selves to the Countdown dataset. We pick this one observed the greatest increases in performances in such dataset which allows for easier interpretation of the results. We train models using the same set of hyperparameters, only varying the ELBO approximation method. We evaluate the test accuracy as a function of the training iteration every 500 iterations.
5000
8600
7000
8000
9000
10000
Training Iteration
Figure 4: Test accuracy with different training it- erations and ELBO estimators on the Countdown dataset.
As observed in Figure 4, estimators that are more accurate result in better improvements on the RL pipelines. Furthermore, the sheer num- ber of function evaluations is not enough to guarantee good results. For instance, SDMC-3 can significantly outperform the naive Monte Carlo estimator even when uses more evaluations. This demonstrates that accurately designing the estimator is of vital importance in GDPO.
CONCLUSION
We proposed Group Diffusion Policy Optimization (GDPO), an RL algorithm designed specifically for diffusion language models (DLMs). By leveraging sequence-level likelihoods through the evi- dence lower bound (ELBO), our approach disentangles and mitigates the major sources of variance that hinder prior methods. In particular, we replace the inefficient double Monte Carlo estimation with simple and fast Semi-deterministic Monte Carlo sampling, yielding a provably lower-variance and more computationally efficient estimator. Extensive experiments demonstrate that GDPO con- sistently improves over pretrained checkpoints and surpasses strong baselines such as diffu-GRPO across math, reasoning, and coding benchmarks. We expect even stronger performance with more powerful pretrained checkpoints, though this remains outside the scope of the present study. Taken together, these findings highlight GDPO as both a theoretically principled and practically effective paradigm for aligning DLMs. We believe that the of cemi-deterministie Monte Carlo sampling offers simple and viable path to handling the large variance issue in the sequence-level ELBO es- timation of Ms,,, and future work explore more effective deterministic-stochasse integration schemes with data-driven quadrature weights and locations to minimize the large variance further.
ACKNOWLEDGMENTS
We thankful Core whose purpose-built Al cloud platform powered all of experi- ments. KR and MT gratefully acknowledge the partial support NSF Grant 2516699, DOE Grants NA0004261, SC0026274, and Richard Duke Fellowship.

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08777 2023.
Marianne Arriola, Aaron Kerem Gokaslan, Justin T. Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block Diffusion: Interpolating Between Au- toregressive and Diffusion Language Models. The International Conference on Learning Rep- resentations (ICLR), 2025.
Kendall Atkinson. An Introduction to Numerical Analysis. John Wiley & Sons, 2008
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Struc- tured Denoising Diffusion Models Discrete StateSSpaces Advances Neural Information Processing Systems (NeurIPS), 194:17981-1777 2021a.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program Synthesis With Large Language Models. arXiv preprint arXiv:2108.0773: 2021b.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and al. Training a Helpful and Harmless Assistant with Reinforcement Learning From Human Feedback. arXiv preprint arXiv:2204.05882 2022.
Joe Benton, Yuyang Shi, Valentin De Bortoli, George Deligiannidis, and Arnad Doucet. From Denoising Diffusions to Denoising Markov Models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(2):286-301, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. Continuous Time Framework for Discrete Denoising Models. Advances in Neural Information Processing Systems, 35:28266-2822 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 2021.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training Verifiers to Solve Math Word Problems. arXiv preprint 10Xi:2110.144666 2021.
Germund Dahlquist and Åke Björck. Numerical Methods in Scientific Computing, Volume SIAM, 2008.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 1171-4186, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, and et al. The LLaMA 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024.
10

---

Published as a conference paper at ICLR 2026
Itai Gat. Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete Flow Matching. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, pp. 133345-133388 2024.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling Diffusion Language Models via Adaptation from Autoregressive Models. In The Thirteenth International Conference on Learning Representations (ICLR), 2025a.
Shansan Gong, Ruixiang Zhang, Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. arXiv preprint 1.XXi:25066206399 2025b.
Daya Guo, Dejian Yang, Haowei Zhang, Teams DeepSeek, and Wenfeng Liang. DeepSeek-RI Incentivizes Reasoning LLMs through Reinforcement Learning. Nature, 645:633 00-6638,,- 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving With the MATH Dataset. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2021a.
Dan Hendrycks. Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Mcasuring Mathematical Problem Solving With The Math Dataset. arXiv preprint arXiv:2 03.003774, 2021b.
Jonathan Ho, Ajay N. Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. In Ad- vances Neural Information Processing Systems (NeurIPS), volume 33, pp. 6840-6851, 2020.
Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, and Guo-Jun Qi. Reinforeing the diffusion chain of lateral thought with diffusion language models. arXiv preprint arXiv:2505. 104466, 2025.
Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. Mereury: Ultra-Fast Language Models Based on Diffusion. arXiv preprint 1rXi::2506.17298. 2025.
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, and Sitan Chen. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. In Proceedings of the 42nd International Conference on Machine Learning (ICML), 2025.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step. In The Twelfth International Conference on Learning Representations, 2023.
Sulin Liu, Juno Nam, Andrew Camphell, Hannes Stärk, Yilun Xu, Tommi Jaakkola, and Rafael G6mez-Bombareell Think While You Generate: Discrete Diffusion with Planned Denoising. In International Conference Learning Representations (ICLR), 2025a.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20788, 2025b.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Diffusion Modeling by Estimating The Ratios of the Data Distribution. International Conference on Machine Learning (ICML). 2024.
Shen Nie. Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
OpenAl. OpenAl ol System Card. arXiv:2412.16720, 2024. URL https: /ttps:///ppeeee hww./a.enaacoooooo/ com/ index/learning to- 1mml Accessed: 2025-09-22.

---

Published as a conference paper at ICLR 2026
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data. International Conference Learning Representations (ICLR), 2025.
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffu- sion Beats Autoregressive in Data-Constrained Settings. arXiv preprint arXiv:2507.18855 2025.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Under- standing by Generative Pre-Training. OpenAl Technical Report, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAl Technical Report, 2019.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusion Language 2024.
Models. Advances in Neural Information Processing Systems,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint
2017.
Shiv Shankar. Padre: Pseudo-likelihone based alignment of diffusion language models. In 2nd Al for Math Workshop@ @ ICMI. 2025, 2025.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.033300 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and General- ized Masked Diffusion for Discrete Data. Advances in Neural Information Processing Systems, 37:103131-103167. 2024.
Yang Song. Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations (ICLR), 2021.
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang. Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang, Mingxuan Wang. Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu, and Hao Zhou. Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference. arXiv preprint arXiv:2508.022 02113993. 2025.
Xiaohang Tang, Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. wdl: Weighted policy opti- mization for reasoning in diffusion language models. arXiv preprint arXiv:2507 08838, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and et al. Kimi K1.5: Scaling Reinforcement Learning with LLMs. arXiv preprint 250i /2599,, 2025.
Qwen Team. QwQ-32B: Embracing the Power of Reinforeement Learning, March 6 2025.
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marein Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, Alexandre Adam, Jarrid Rector- Brooks, Yoshua Bengio, Glen Berseth, and Nikolay Malkin. Amortizing intractable inference in diffusion models for vision, language, and control. In Neural Information Processing Systems (NeurIPS), 2024.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformer Reinforce- ment Learning, 2020.
Guanghan Wang, Yair Schiff. Subham Sekhar Sahoo, and Volodymyr Kuleshov. Remasking Discrete Diffusion Models with Inference- -ime Scaling. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
12

---

Published as a conference paper at ICLR 2026
Ronald J. Williams. Simple Statistical frolowng-followingg Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, 8(3):229-23 22222,9.,., 1992.
Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), 2025.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning. Interna- tional Conference on Learning Representations (ICLR), 2025a.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b: Diffusion Large Language Models. arXiv preprint arXiv:25088 15487, 2025b.
Oussama Zekri and Nicolas Boullé. Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. dl: Scaling Reasoning in Dif- fusion Large Language Models via Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS), 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group Sequence Policy Opti- mization. arXiv preprint, 2025. URL https:///// arxiv. orgg horg/aber abs/2501 8071.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, al. LLaDA 1.5: Muriance-Rdduce Preference Optimization for Large Language Diffusion Models, arXiv preprint 1rXiv:2505.192.. 2025.
13

---

Published as a conference paper at ICLR 2026
Supplementary Material for "Improving Reasoning for Diffusson Language Models via Group Diffusion Policy Optimization"
In this supplementary material, present additional details variance decomposition in $A, pro- vide a theoretical analysis of asymptotic error bounds in $B, justify key properties of the integrand in $C, describe the experimental setup in $D, comparison against Gong al. (2025b) in Š and include representative generated samples in $F.
A MORE DETAILS ON VARIANCE DECOMPOSITION
We start by studying the different sources of variance in approximating the ELBO, in the case of masked diffusion models the equation is given by:
Ey (-
Σ
1 l log Tolvo () log (vlg)
From looking at this expression we observe that there are two sources of variance, (1) Time Sam- pling: we must sample t which determines the noise level, and (2) Random Masking: we must sample which injects noise to the clean zo- We start by analyzing how each source of randomness affects the variance of an estimator. To simplify our discussion, we will rewrite the integrand in simplified way:
Eg [Z 24)] S log (vy4))
M] log To 1y,, q) is a random variable w.r.t. the joint distribu- tion of (t, 3t). We are interested in studying Var (t., y))) and how decomposes with respect to t, Vr. by the law of total variance we can decompose this as:
where Z(t, 91) == ! EL l
Var (Zt, 90))
E. (ar Z [t]]
Var, (E t)
Var given by Z/r
Var given by
The above expression indicates that we can understand variance as the sum of two components, where each term corresponds to a distinct source of variance. We argue that most of the variance comes from selecting the noise level t. When the noise level is t = O we range from fully unmasked fully masked, creating a large disparity in the noise samples and subsequently causing large swings in the variance.
To test this we evaluate each term for 1000 different prompts pulled from the Open Web dataset. We plot the mean and variance as function of time in Figure 2. This figure reveals very important facts:
Most of the variance is coming from randomly selecting the timestamps/ noise levels and not from injecting noise into the text.
2. The loss function when observed as function of time yields a simple structure.
For this reason, we advocate for fixed timestamps for likelihood approximation and embracing a Semi-deterministic Monte Carlo sampler instead of the naïve double Monte Carlo approximations.
B ANALYSIS OF ASYMPTOTIC ERROR BOUNDS
In this section, we analyze the error bound of approximating the ELBO via the proposed Semi- Deterministic Monte Carlo scheme, focusing on the training objective given in (2),)), namely,
I.
CELBO(ylg = Ex-24 Br
l = M] log To (o (y lv)
14

---

Published as a conference paper at ICLR 2026
Notation and definition. We proceed by first defining the relevant quantities that will be used in the ensuing technical analysis. Define
1 E 1 = [ log (B'l,, 9),
Z,
which is random variable with respect the distribution of yt, where yt me(-))) Let
g(t) == Eyrr, (l1b)
(10)
and the Monte Carlo approximation of g(t) be gMC (1) 1 Σ 2k=1 (Z where 2,4² iid draws
of Ze For convenience, we let [1r) == {i E {I.... ,L] y/ M) and o [[we)), the cardinality of ((ye). Note that Z₁ can be rewritten as Ze = +Eier()) log pa g).
With these definitions, the training objective of interest is given by
CELBO q) 1 (o(t)) = J g(t)dt,
(11)
and its approximation via Semi retemiiteriiiistic Monte Carlo scheme (namely, Eq.(5)) can be written as
CELBO = wn.Mc((t),
(12)
which is numerical approximation of the integral the Monte Carlo approximation (o = So dtcdt at points f,. n = 1. 1.... N. For the time being, we assume generic numerical approxi- mation of the integral, and using the notation involving W/y
The approximation error of (12( is given by
[[CLLO [ELBO|] E
Σ w. Vnngecenn
- - s(t)dt
(13)
wng (t.))t [. s(c)dt)
E
W. W.gge (In)
Σ wng(tn)
MC variance
integration bias?
this holds due to the independence of Monte Carlo samples, since
N
N
N
E
W.JMc (tn))
Σ
w., E.,.Eggett...
Σ Wng(tn))
and therefore the second term boils down to the integration bias. can be seen that this becomes the classical weeanssquaed-erroo of an estimator, which can be decomposed into the variance (from total of K Monte Carlo samples) and the hias (from the approximation of the integral) at N points l1," AN- The bounds of these two terms will be analyzed separately in the sequel.
B.1 THE VARIANCE TERM
To analyze this term, we start with the pointwise Monte Carlo error.
Lemma B.I. Assume the following conditions hold:
CI. Er, (ct) < c0, (r,(c,) o0;
C2. 38 > 0: 5 < Toly" Vt. ().tg) < - 8, for all 2
Then, the following holds
E g(t) - smm(()) = $03(1).
where d²(1)= : (z).
(14)
15

---

Published as a conference paper at ICLR 2026
Proof. To establish the results in Eq.(14) the following assumptions need to hold:
A1. Measurability: Z4 is jointly measurable in and yes
A2. Integrability: E((2,) < oo, A [0. 1];
A3. Finite variance: (² < 0o, At E [0.
By writing Z Zi EieI(.)) log To yo. 9). it can be verified that C1 and C2 are sufficient for Al-A3 to hold.
Remark B.1. Interpretation of conditions CI and C2 are given follows: CI requires the number of masked tokens be controlled, and C2 requires the conditional likelihood to be well behaved.
Coming back to the variance term, by Lemma B.1, at each time point (i.e., tn 's), the Monte Carlo estimate (t,) introduces random error whose pointwise variance is given by o² Consequently, the variance of the estimator LELBO is given by
N
27
N
E
(x. - Σ (t))
E
(g(tn) 9MC (tm()))
m==1
n=1
1=1
N
E
Wn
Σ
W
2
(1 (1n)
(15)
K
the last equality holds since for fn & tn. due to the independence of MC samples across n, we have that (un umm) 0 for m; further note that [u,,)) 0 and Eu? k (t,))
Proposition B.1 shows that under a refinement of condition C1, Eq.(15) is bounded and therefore its rate can be derived accordingly.
Proposition B.1. Suppose the following condition holds for (defined identically to that in Lemma B: Cl'. (ce S Cot? for some constant Co 0.
Assume also that Wn = O(1/N) and in are approximately equally spaced. Then, the following holds:
N
Co
N
1 N K
( tdt) (10)=0((1) 1 ) K N2 N..NN.
w (tn) 5
Σ 2 -0 ==1
Σ tn) =
K
K
N n=1
n=1
(16)
Remark B.2, Condition CI' guarantees that Jo d²) < cc, which then implies the finiteness of EN o as N + oo, provided that the weights wn corresponds to valid integration scheme; iee, Wn > 0, Σ Wi, = 1 and the mesh of the points t, becomes finer as N + cc. Empirically, CI' can operationalized by considering a masking scheme where token xi is masked independently across = -.... L with probability t. and cı ~ Bin( L.t). Finally, the assumption on weights w,,, = 0 1) all integration schemes (simple Riemann, midpoint, trapezoid, Simpson).
N and that t,, 's being roughly equally spaced are both fairly reasonable for
The next proposition shows that when an N-dependent decay condition is satisfied by the log- likelihood, then a rate faster than the one established in (1.( can obtained.
Proposition B.2. Suppose the following condition holds
Cl". Er, (ct) s Cueen for some positive constant Croken-
In addition, assume the tog-likelihoud satisfies the following, for some positive constant Crog-u::
t2
Var( log no (y lve, 4)) < Clog-tik N
16

---

Published as a conference paper at ICLR 2026
Then, the following holds
1 N
Σ K
2 (In) =
(17)
KN2
Proof.
Given the definition of 0² d²()) (t) (see, e.g., Eq. (14)), under condition C1", has
(t)
Goken Var log (o (vyllve99)) 9))
12
Then, for weights max, Wn = 0 (x) (which are the weights for standard integration rules) and in we get
Cle Loken Gog. lik
12
1 Co N'
o² (t.)
where Co Cloken Clog-lik like
Hence, we have a uniform bound on (bm) S uniform bound, we obtain
Co for every n 1, 1,..* N. Using the above
N
C
N
C.
Co
Σ
2 w
0 (1m)
N
w
N
N
=
N2
1=1
n=
Consequently, follows that
N
Σ
(t
=0
K
KN2
Some intuition behind the N-dependent decay condition is given as follows. Specifically, for the log-likelihood to satisfy such condition, one compatible specification is given by
t
log To 1y. 9) h(t)
+
VN" Cr. where E(G)) 0, Var(e:) S og for some dg >
Vt;
h(t) is smooth Lipschitz function. Under this specification, the fluctuation of the likelihood is small for small t (little masking) and increases for large t.
B.2 THE BIAS TERM
Next, we analyze the bias term induced by approximating the integral fo g(t) via numerical inte- gration schemes.
The next lemma provides the rate of the bias for the case where some quadrature rule used, under certain assumptions of g(t).
Lemma B.2. Suppose that g(t) [0, R a deterministic and continuously twice differentiable function (i.e., in C²). Further, suppose we use some quadrature rule (midpoint, trapezoidal, Simpson, etc.) to approximate its integral at points ( E [0, 1}; iee.
N
J g(t)dt mz Σ wng(ta),
(18)
where weights wn,' 's are chosen according the specific quadrature rule in use. Then, such an approximation scheme introduces deterministic integration bias, given by
Ex(g) =1 L' g(t)dt Σ (n,) l.
N
and it scales as follows:
for midpoint trapezoidal rule, as O {z));
17

---

Published as a conference paper at ICLR 2026
for Simpson, O x4)
Proof. These are standard results from numerical analysis; see, c.g., Dahlquist & Björck (2008); Atkinson (2008).
The above lemma directly yields the following result: for the integration bias? that appears in the decomposition of (13,, it follows that for midpoint or trapezoidal rule,
2 Σ (wn((t)) L e(()ar)) = 0( N
and for Simpson, the rate is given by O O(Ns).
Remark B.3. The above rates suggest that by applying a quadrature rule-provided that g(t) sat- isfies certain properties-the integration bias² has become practically negligible, relative to the variance term that dominates. However, if one uses a generic integration scheme instead, such as the Riemann sum, the integration bias would scale at the rate of O xxx) and thus the bias? O (Zzz)
As it can be seen from Remark B.3, the "gain" by considering a Gemi-Deterministic Monte Carlo scheme is partly built on the fact that the integration bias can be practically negligible, which hinges quadrature rules for integration. However, apply such rules, g(t) needs to
on whether one be continuously twice differentiable, and its properties are further studied in Appendix C.
C PROPERTIES OF THE INTEGRAND IN THE NUMERICAL QUADRATURE
For this section, we will denote po probability distribution on [M] and denote ps the marginal of the forward process under masked diffusion and h, the learned marginal. Our goal state some conditions under which the loss function would be well-suited for numerical quadratures. We start by noting that from Proposition of Benton et al. (2024) form:
write the ELBO in the following
CELBO(y19) 11.
Σ lM M) Te Me. 9)
Pi (b)
E ②
dt +C C
Te (3xe
d (L(p:ll dt
where & is the score matching operator introduced in Benton et al. (2024). This calculation allows realize that study the integrand is enough to study [L definition, the integrand s(() defined in the previous section satisfies g(l) = f'(() + C for some constant C. By Proposition (1.1) in Benton et (2024) we know that 2 0, therefore f'( 2
ft).. Note that with this
Notation: We first introduce the following notation: let d denotes the probability distribution marginalized over the entries not S. The following lemma states that we can write f(t) as polynomial with coefficients relating to the average KL under different levels of masking,
Lemma C.1. The KL divergence for two distributions over time evolving according to masked diffusion 8 as a function oft t and we can write:
D
KL(pol(=.) = Σ
1) --KK))=kK
b=
SCID sk
(D)
D
Σ
t
(1-
Ck
k
18

---

Published as a conference paper at ICLR 2026
where:
Ck
SCID
|S|=k
the average KT. divergence over sets of size k.
The proof relies on fundamental combinatorial facts and the definition of the forward pro- cess:
Proof.
Pr(x) (x))
(L(p(((T+)= = Σ
(r)
log
Pc(x) x1(
Σ
Σ
(x log
1= SC(D]
2:2,= M 18
SES
D
td t) D- pp (x)
Σ
(1- D- p (x) log
th t) 1- k T (x)
SC -.
3 IES
Pú (z) To (x)
S
Σ
Σ
(1- 1)
Σ
(x) log
7:z.=M M ES
1
D
Σ
Σ
11
(L 1
k=
SCID]
|S|=
Using this, we can rewrite the KL as:
(1-1)) ((1-t)--KKI))) (KL((E
SC
iSk=
D)+((1 D)
rd(1
Cl
k
k=1
which gives the result.
Given this lemma, it becomes clear that to understand the properties of the integrand, it is enough to understand the following polynomial:
D
f(t)
Σ km
Ck
*k (1 - (1-
Σ (()
This is Bernstein polynomial, and its properties have been studied before.
Forward differences and Bernstein derivatives. When looking at the derivatives of Bernestein polynomials, the differences of the coefficients appear, to this end denote:
Ack- Ck+1 Ck
19

---

Published as a conference paper at ICLR 2026
A A ck+2 2ck+1 + Ck (ct+2 Ck+1) (ct+1 cx) = Ax+1 A
A Ck++ 3cx+2 30k+1 Ck A A
Then it easy to check that: Lemma C.2 (Derivative formulas). For all E [0,1]
D-1 f(() D Σ
Ack (-))-
(19)
k=0
("(1) D(D Σ c Bi,D-2((),
(20)
D-3
(t) = D(D- - 1)(D 2)
Σ A Bs.
(21)
kmm
Because (1) 2 0 on [0, 1]. the signs of (() are controlled by the signs of the averaged forward differences A'c..
One-step representation via KL chain rule. Let F(S) = KL || To ). we are now able to obtain simple, intuitive explanations of the meaning each coefficient A For & S, define the (nonnegative) one-step gain
(,( F(S U (1)) F(S) Xx-r(3s [I(()(X,) X (o Xs))] 2 0.
The above quantity represents the average KL that obtain by masking an token given set. Intuitively we expect A₃ to be increasing, as masking more tokens reduces the amount of information that we have (This in fact, holds already without extra assumptions). Similarly, we obtain the iterated differences:
ACk = E SI (A)))),
(22)
iES
A ck
5 (S (1) (.)))
(23)
i
A = E [ U (j. A.SSU)) (SU (0))+ +A)))
(24)
i.j.4S
such quantities are positive, this will immediately imply that the derivatives in C.2 will be positive, implying important consequences the loss function. Intuitively, such coefficients being positive correspond to the idea that masking one token when K are masked results in more loss of informa- tion than masking token when L K are masked. We require these assumptions and explain the sufficient conditions for them to hold in the following Proposition:
Proposition C.1 (Sufficient conditions for convexity and increasing integrand). For all t E [0, 1]:
1. (Integrand is positive) For any Po. Qo. the set function F is monotone: $ S = F(S) <
F(T) (data processing under marginalization) Hence A:(S) Ack 0, and by Eq.(19 19) we have
('(1) (I.(P.)).)) 2 0
dt
2. (Integrand increasing in L) If the Increasing Conditional Divergence (ICD) condition holds:
A.(S) S ATT) for all S T C (i),
then 2cc by (23), and (20) yields
d? (I((1.)Q..) 2
("(()
di
20

---

Published as a conference paper at ICLR 2026
3. (Integrand has convexity in t) If, in addition, the second-order ICD condition holds:
(A (1)) A.)) 2 (5. A (())
for all distinct i, l and S C [D] \ {i.j. e). then e 2 0 by (2.(24), and (21(2 gives
d
(t) बe KL( 2
D EMPIRICAL DETAILS
D.1 TRAINING DETAILS
We leverage the codebase from Zhao al. (2025) which in itself leverages the TRL library (von Werra et al., 2020). For our hyperparameters we keep most of the default parameters from Zhao al. (2025) without any hyperparameter search. We leverage a Low-Rank Adaptation with rank r = 128 and scaling factor = 64.
Across all runs we utilize the AdamW optimizer with parameters 3, 0.9, 32 0.99 with a weight decay of 0.1 and gradient clipping at 0.2. We leverage flash attention and 4 4-bit quantization. We found that GDPO usually requires smaller learning rate than diffu-GRPO and otherwise it can result in diverging models. We save checkpoints every 300 or 500 iterations and stop runs when the reward function has plateaued, for the countdown dataset we observe that the reward function continuous increasing for very long time, while for the math dataset observed high- performing checkpoints around 9000 iterations, but report the best one, which happened later in training. We present the learning rate well other hyperparameters in Table 4. We present the reward function for the different datasets in Figure 5 and observe that GDPO can offer a steady growth before plateauing.
Math
GSMBK
Countdown
1.8
1.6
Reward
Reward
1.4
Rewaro
0.6
1.0
1.2
0.4
0.5
0.0
5000
10000
5500
20000
sóa
2ni0 2500 Gradient Update Steps
1000
Gradient ate Steps
Gradient Update Steps
Sudoku
KodCode
1.5
0.25
144
Reward
1.3
Reward
0.15
1.2
0.10
1.3
0.05
2500
5000
7300
10000 12500
0.3
1030 1200 2000 2000
Gradient Update Steps
Gradient Update Steps
Figure 5: Reward curves during RL training for the models reported in Table 2.
D.2 REWARD FUNCTIONS
To provide a self-contained exposition, we summarize the reward functions used RL-based post- training. The design incorporates task-specific incentives that promote formatting fidelity, logical consistency, and solution correctness, with their structure tailored to the demands of each task. This formulation is largely consistent with prior work (Zhao et al., 2025), and our exposition follows established conventions.
GSM8K The setup uses composite reward function composed of several components, including
21

---

Published as a conference paper at ICLR 2026
Correctness: Reward for the parsed answer aligning with the ground truth (+2.0 points).
Integer Answer: Reward for producing a valid integer as the parsed answer (+0.5).
Soft Format: Reward for responses in the pattern <reasoning>... w.t...../reaaoonnnn
<answer>.... .aaeerr...../ssseeeeee </answer: (+0.5 points).
Striet Format: Reward for outputs matching the strict format and line breaks (+0.5 points)
XML Format: Reward for correct placement of reasoning and answer tags, with +0.125 points per matched pair.
MATH500
Correctness: Reward for the parsed answer given in \boxed{] and consistent with the ground truth (+2.0).
Soft Format: Reward for the use of <answer> tags and \boxed, assigned as:
+1.00 point when answer tags are included and enclose \boxed.
+0.75 point when answer tags are included but without \boxed.
+0.50 point when answer tags are omitted yet \boxed is provided.
+0.25 point when both answer tags and \boxed are missing.
Countdown
Soft Correctness:
Reward for equations relative to the target value:
+1.0 point when the equation reaches target using only the provided numbers.
+0.1 point when the equation does not reach the target but uses the correct numbers.
+0 point when both criteria fail.
Sudoku
Cell Accuracy: Reward for the fraction of originally empty cells filled correctly.
Coding (Gong et al., 2025b)
Format: Reward for generated code adhering to the required Python code-block structure: 1.0 point when the code is enclosed in valid Python code block 'python " and the snippet parses without syntax errors.
0.5 point when the output well-formatted but contains Python syntax errors.
0.0 point when the format invalid no code block detected.
Code Execution: Reward for generated code that passes unit tests:
Score equals the fraction test cases passed by executing the code.
0.0 point when the code fails formatting or does not run successfully.
Table 4: Hyperparameters for different training runs
Model
GSM8K
MATH500
Countdown
Sudoku
KodCodde-Light-RI.
Learning Rate
5e -
7
3e 7
1e -
6
2e -
le
Number of
4
8
GPUs
Per-GPU-Batch
6
6
Size
Number of Quadrature Points
3
3
Number of iterations of best checkpoint
6000
17000
3000
11.500
3800
22

---

Published as a conference paper at ICLR 2026
Table Comparison of training iterations across different configurations
Method
Time (Hours)
Number of Iterations
NFEs per Step
diffu-GRPP
4500
GDPO (N=2)
3800
2
GDPO (N=3)
3500
3
Table 6: Comparison of model performance across different configurations.
GSMSK
Countdown
Model
NFE
128
256
512
128
256
512
LLaDA-8B-Instuue
I
68.7
76.7
78.2
36.2
41.2
40.4
+ diffu-GRPO
73.2
81.1
82.1
34.8
32.0
42.2
+ GDPO (N=2)
2
79.08
82.94
84.61
48.83
67.19
77.34
+ GDPO (N=3)
3
78.39
82.79
84.99
64.45
74.22
80.86
D.3 ABLATION STUDIES
Computational Cost Analysis We present analysis on the additional computational budget introduced by using multiple NFEs per training iteration. To do so, we limit the total training time fixed budget of 6 hours. We study this on the countdown dataset and use dl (Zhao et al., 2025) as our baseline. We use per-device batch size of and train 3 models using different number of NFEs per training step. We don't evaluate the test accuracy for this experiment different methods may require different hyperparameters to achieve optimal performance, Our main goal is to highlight the additional computational overhead introduced by using GDPO with multiple quadrature points.
We present the results of this analysis in Table 5. As expected, increasing the number of network evaluations results in less total training iterations. However, even the most expensive configuration N = we still achieve 77% of the iterations while obtaining the added benefits of GDPO and better likelihood approximation.
Effect of Quadrature Points We further conduct an ablation study in Table to analyze the impact of varying the number quadrature points when approximating the ground-truth likelihood. Increasing the number of quadrature points consistently improves approximation quality and training stability. Using N = 2 already captures most of the gains relative to the ground truth, while N 3 achieves the best overall performance, establishing state-of-the results among the evaluated configurations.
E COMPARISON WITH DIFFUCODER
One method that appears similar to ours is Gong et al. (2025b) which introduces coupled-GRPO an extension of diffu-GRPO (Zhao et al., 2025) that leverages multiple network evaluations to improve the per-token likelihood evaluations. Diffu-GRPO had originally used the following approximation to the token-level likelihoods given by:
logpo(y'lg @ MM.
Coupled- GRPO improves this by using complementary timestep pairs () such that t + f T and complementary masks. Each mask is disjoint and together they cover the entire sequence. Given the two sequences y/, y/ can approximate the token-level likelihoods as:
(logpo(y) @ we)
if y y
log Toly (q)
(lg 1 vz))
if y y
23

---

Published as a conference paper at ICLR 2026
This method has the virtue of evaluating each token under more realistic setting than the fully masked set up of Zhao et al. (2025). Furthermore, Gong et al. (2025b) showed that results in variance reduction while remaining computationally efficient.
There are two main differences between both methods. (1) To- ken vs Sequence Level proba- bility GDPO relies sequence level probabilities, while coupled- GRPO relies on token level like- lihoods. Although both meth- ods achieve variance reduction, it is over different objective. (2) Variable number of timesteps Coupled-GRPO uses 2 sampling times, while GDPO, when imple- mented through Gaussian Quadra- tures, can use an arbitrary N. Al- though it should be possible tend coupled-GRPO to more points, we leave this analysis to future work.
Table 7: Model performance on Mathematics and Plan- ning Benchmarks Green is the best performing model.
Countdown
Sudoku
Model
128
256
512
128
256
512
LLaDA-88-Instrudt
20.7
19.5
16.0
11.7
6.7
5.5
diffi GRPO
33.2
31.3
37.1
18.4
12.9
11.0
+RPP
34.8
32.0
42.2
22.1
16.7
9.5
N/A
51.2
46.1
N/A
25.2
24.2
tooupled-RRP
33.59
43.75
47.66
23.63
21.0
21.92
GDPO
42.97
67.19
66.41
25.05
24.17
25.10
We perform experiments analyze the difference in performance between GDPO and coupled- GRPO. We implement diffucoder for the countdown and sudoku benchmarks following the official implementation and present the results in Table 7. We that couple-GRPO is effective in improv- ing the results when compared to diffu-GRPO but GDPO obtains better results across all tasks.
F SAMPLE GENERATIONS
F.1 GENERATION ON SUDOKU
For the sudoku task leverage the following prompt and present samples at 512 tokens.
Sample Generations
Prompt
[|strtofteet| |x| heacer idl >user< end header id|>
Please solve the following 4x4 Sudoku puzzle. The puzzle is provided as a 16-character string reading left-to-right,
+
+ top-to-bottom, where '0' represents empty cells.
Rules:
Fill empty cells with digits
Each row must contain digits 1-4 exactly once
Each column must contain digits 1-4 exactly once
Each 2x2 box must contain digits 1-4 exactly once
Important: Your solution must be COMPLETE 16-character 4 string with only the aigits 1-4, representing your final
+ solved grid.
Respond in this exact format: <reasoning>
Your step-by-step solving process
</reascning>
<answer>
(16-character solution string with no spaces or separators]
</answer>
24

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
Solve the following Sudoku puzzle: 4320004330100004
Model Generation
To solve the 4x4 Sudoku puzzle, we need to fill in empty
cells with digits 1-4 such that each row, column, and 2x2
box contains all digits exactly once. We's start by analyzing the given puzzle and filling the grid step
- by step,
1. ++Ieentify the empty cells**:
The first row: 43 20
The second row: 00
The third row: The fourth row: 00 04
2. **Fil1 the first column**:
The first column already has 4, 0, 3, and 0. The only
+ digits left are 1, and
Fill the first column: 2 4
3. +*Fil1 the second column**:
The second column already has 3, 4, 0, and o. The only digits left
and
Fill the second column:
4
4. *+Fill the third column++:
The third column already has 2, 3, 1, and 4. The only
digits left are 1, 2, and 3.
Fill the third column: 1
5.
*Fi11 the fourth column**:
The fourth column already has 0, 0, 0, and 4. The only digits left are 1, 2, and 3.
Fill the fourth column: 1 3
Now, let's fill the 2x2 boxes:
The first 2x2 box: 2 3
The second 2x2 box: 1 2 4
The third 2x2 box: 1 2 4
The fourth 2x2 box: 1
</reasoning>
<>
</>
<answer>
123124
12444
123124
123124
</answer>< |eot_id| [edooftxxt]
Ground Truth
4321124334122134
25

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
Solve the following Sudoku puzzle: 3014002020004133
Model Generation
To solve the 4x4 Sudoku puzzle, we need to fill in empty cells with digits 1-4 such that each row, column, and 2x2
box contains all digits exactly once. We's start by analyzing the given puzale and filling the grid step
- by step,
1. +IIeentify the initial digits+4:
The first row is \"3014"".
The second The third row \"2000\".
\*0020)".
The fourth row is \"4130\".
2. **Fil the first column**:
The first column is \"3014\",
The second column The third column is \"2000\",
\*0020\"
The fourth column is \"4130\".
3. **Fil1 the second column**:
The second column is "00200".
The second column \*0020\".
The third column is \*00201".
The Sourth column is \"0130\".
4. *Fi11 the third column**:
The third column is "2000>""
The third column is "2000"".
The third column is \"2000\".
The fourth column is \"0030\".
5. **iil the fourth column**:
The fourth column The fourth column is \*4130\".
""4130"".
The fourth column is \*4130\"
The fourth column is \*0334\".
Now, let's fill in the 2x2 boxes:
The first 2x2 box \"3014\".
The second 2x2 box is \*2020\".
The third 2x2 box is \"2000)".
The fourth 2x2 bex is \*<134\".
Finally, we can verify that all rows, columns, and boxes are
4 filled correctly according to the Sudoku rules. </reasoning>
<answer>
1213142122112214133
</answer>< id|> |endoftext
Ground Truth
3214142323414132
26

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
Solve the following Sudoku puzzle: 0000100420013142
Model Generation
To solve the 4x4 Sudoku puzzle, we need to fill in the grid with digits 1-4 such that each row, column, and 2x2 box contains all the digits exactly once. Let's the stcp-by-step process:
1. +IIeettiy the missing digits**: start by identifying the
missing digits in the grid. In this case, the missing digits are 1, 2, 3, and 4.
**Analyze the grides: look the grid and identi fy the - positions of the missing digits. The this puzzle, the
- missing digits are in the first row, first column, and the first 2x2 box.
3. **Fil in the first row**: The first row must contain the
4 digits 1-4 exactly once. The first row should
3,
4 4.
4. +Fiil in the first columnes: The first column must contain the digits 1-4 exactly once. The first column
4 should be 1, 2, 3, **Fill in the first 2x2 box*:: The first 2x2 bex must
4 contain the digits 1-4 exactly once. The first 2x2 box should be 1, 2, 3, 4.
6. *Fil1 in the second row**: The second
must centain the digits 1-4 exactly once. The second row should be 2, + 3, 4, 1.
7. **Fill in the second column**: The second column must contain the digits 1-4 exactly once. The secend column
should be 3, 4,
**Fill in the second 2x2 box**: second bex must 4 contain the digits 1-4 exactly once. The second 2x2 box
+ should be 2, 3, 4, 1.
9. ++iil in the third row**: The third row must contain the
digits 1-4 exactly once. The third should be 1,
2.
By following these steps, we can solve the 4x4 Sudcku puzzle.
</reasening>
<answer>
123412341123413412
</ (nndfttttt
Ground Truth
4213132424313142
F.2 GSM8K SAMPLES
For the gsm8k task leverage the following prompt and present generations at 256 tokens.
27

---

Published as a conference paper at ICLR 2026
Sample Generations
Prompt
<|stttoffttt /xceexxxx><]p/
/><|start, header idl >user< header_ idl>
You are math expert. You will be given a question to solve.
7 Solve it stop by stop, Wrap the final answer in a S \bboxe(().
Respond in the following format: <reasoning>
Your reasoning here
</reasoning>
<answer>
\ \boxed(....)
</ </answer>
Sample Generations
Question
Janet \u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends
every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day the farmers' market?
Model Generation
determine how much Janet makes every day at the farmers! market, we need to follow these steps:
1. Calculate the total number of eggs laid by Janet's ducks per day.
Subtract the number eggs she eats for breakfast bakes for her friends. Multiply the remaining eggs by the price at which she sells them.
+
3.
First, we know that Janet ducks lay eggs per day.
Next, we account for the eggs she eats for preakfast and bakes for her friends: She 3 eggs for breakfast. She bakes eggs for her friends.
So, the total number of eggs she uses is: WM = A
The number of eggs left to sell is: WI 16 - 7 = WV
Janet sells each egg for $2, Therefore, her daily earnings are: WI 9 Vitimes 2 18 W
Thus, Janet makes $18 every day at the farmers' market. <eeasoning>
<answer> \ \boxed(18)
</answer>< eot > /enooffexttttttt//o/////
Ground Truth
18.0
28

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
Every day, Wendi feeds each her chickens three cups mixed chicken feed, containing seeds, mealworms and vegetables help keep them healthy. She gives the chickens their feed in three separate meals. In the feed.
morning, she gives her flock of chickens 15 cups In the afternoon, she gives her chickens another 25 cups + of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi's 's flock is 20 chickens?
Model Generation
>user< hnddhrrred_eedder_idd
You are a math expert. You will be given a questien te solve. Solve it step by step. Wrap the final answer in a ().
Respond in the following format: <reasening>
Your reasoning here </reasoning>
<answer>
W)boee(....)
</answer>
Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives 4 chickens their feed in three separate meals. In the morning, she gives her fleck of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups feed. How many cups of feed does she need give her chickens the final meal of the day the size of 4 Wendi's #lock is 20 chickens?
< <leot sttrti/////tttteeeeeee_aaa header ial halssassstttts/ lend header id/>
<reasoning>",
"generations":
To determine how many cups of feed Wendi needs to give her chickens in the final meal of the day, we first need to calculate the total amount
feed required for the entire day, Wendi has 20 chickens, and each chicken needs
4 3 cups of feed per day. Therefore, the total amount of
5 feed needed for the day is:
WM 20 \text chickens) Mtimms 3 \ltext cups/chicken) - 60 litext cups) WV
Next, we know that Wendi has already given her chickens 15 cups in the morning and 25 cups in the afternocn. So, the S total anount of feed given far is: / cups] 40 \ cups]
WM \itexe/ cups) WV
To find out how many cups needed for the final meal, subtract the amount feed already given from the total amount required:
W 60 \ttext cups) 40 Wttext ( cups) 20 \text cups)
\!
</reasoning>
<answer> \boxed (20)
</answe>><leo__idl>>>
Ground Truth
29
20.0

---

15 Published as a conference paper at ICLR 2026
Sample Generations
Question
John drives for hours speed of 60 mph and then turns
around because he realizes he forgot something very
important at home, He tries to get home in 4 hours but
spends the first 2 hours in standstill traffic. He spends the next half-hour driving at speed of 30mph, before being able to drive the remaining time of the
+ hours going at 80 mph. How far is he from home at the end of those 4 hours?
Model Generation
First, we calculate John total driving distance. He drives for 3 hours at 60 mph, sc he covers 3 * 60 180 miles. Then he spends 2 hours in standstill traffic,
he coesn't cover any distance during that time. Next spends half an hour at mph, so he covers 0.5
miles. Finally, he drives the remaining 1 hour at 80 mph, so he covers
80 80 miles. Adding these distances
together, we get the total distance driven: 180 15 + 80 turned around and the distance is + the same in both directions, the distance he drove away from hone is 270 / 2 135 miles. Therefore, he is 135 miles from home at the end of those hours.
275 miles. Since
</reascning>
<answer>
\boxed(135)
< //twee//eooLLLlkx///oooooooo///d
< axnoffterx|///emmddeeeexee/d//dd
< |enoffrsxs////mm00tttxxx/dddeeer
< |nooffeec///1ebbtttttxx////deeer
< exootteee////Ieeeottttaeeedddded
< [nnoffexxi/<</oroffeeee////eeeee
< [
< (
Ground Truth
45.0
F.3 COUNTDOWN SAMPLES
For the the countdown task we leverage the following prompt and we present generations at 256 tokens.
30

---

Published as a conference paper at ICLR 2026
Sample Generations
Prompt
ststttoftttt <xrttexx|<<<]] statttart header >cl>sser<] end header_idd
Using only the provided numbers,
arithmetic number, You may use the operations +,
expression that evaluates to exactly the provided target and / as needed, but each number must be used exactly ence. Think step-by-step, After reasening, provide only your final expression inside [boxed() tags without including equals sign or the target number. For example: (boxed + b Reppppd in the following format: <reasoning>
Your reasening here </reasening>
<answer>
\)boxe((...)
</answer>
Sample Generations
Question
Numbers: [49, 55, 53]
Target: 51
Model Generation
To find arithmetic expression using the numbers 49, 55, and 53 that evaluates to 51, aifferent combinations
start by considering operations numbers can use to - achieve the target. Let's s try a few possibilities:
1. **Addition and subtraction+s:
~V3 51 49 51\))
-55 + 53 51\
2. *Muuttppliaation and DiVisionAA:
(5555 53 49 = =5\\
(53 49 5)) = 511))
that the expressiens work in different ways. but - one simplest way is to use subtraction since subtraction
4 and addition are inverse operations.
Thus, one valid expression is:
WM 55 53 49 51 WV
This expression uses each number exactly once and evaluates 4 to the target number 51. </reasoning> <answer>
\boxed 53 49)
</answer>< poot_idd_ee I
Ground Truth
49, 55, 53 }. 51
31

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
Numbers: [38, 92, 52]
Target: 78
Model Generation
To find arithmetic expressicn using the numbers 38, 92, and that evaluates to 78, we can start by considering different combinations of operations numbers can use to achieve the target. Let's try a few possibilities:
1. *Addition and Subtracticnss;
52 78\1)
(38 + 92 52 78\))
2. **Multiplication and Divisicn*s;
(9192 - 52 38
eg 78\1)
(3192 38 52
eq 78\1)
(92 - 52 + 38
eg 78\1)
From the above combinations, we see that the expressicn (9
3 38 52( correctly evaluates 78. Therefore, the
3 expression that uses 38, 92, and 52 exactly once and 78 </ </rassnnigg>
<answer>
\boxed </ </answer>< eot idl Indoftext
52)
Ground Truth
38, 92, 52 ), 78
32

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
Numbers: [67, 47, 25]
Target: 5
Model Generation
create arithmetic expression that evaluates exactly the target number 5 using the numbers 67, 47, and 25, we need to consider the operations operations +,
and One's try to get to by using subtraction combinaticn of Using
S /. Each number must be used exactly once.
We need a number that when subtracted from 67 gives us 4 5.
This is impossible because 67 is 708 larger than 5. Next Using addition try:
Aading -27 to gives usb -27 32 5).
Next, using division try:
- Dividing each 5 by gives us 5 (25/2) 5/1
However, using multiplication combination:
Multiplying divided by -2 gives us Thus, the expression expression that25+67- 87 < /rrasoning>>
<answer>
\47 + 25 67)
conswer></cot_iie id Indoftext
Ground Truth
67, 47, 25 }, 5
F.4 MATH SAMPLES
For the the mathj task we leverage the following prompt and present generations at 256 tokens.
Sample Generations
Prompt
[ header_idl
You math expert. You will be given a question to solve. 4 Solve it step by step. Wrap the final answer in 4 (bboxed
Respond in the following format: <reasoning> Your reasoning here </reascning>
<answer> \bbxed(....)
</answer>
33

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
Convert the point (0, $ in rectangular coordinates to polar coordinates. Enter your answer in the form $ (r, \\thea), where år and $0 Wie Atheta
\ \pi..
Model Generation
Convert the point (0,, $ in rectangular coordinates to polar coordinates. Enter your answer in the form $ (r, Attheta)) where Sr > 0$ and $0 Wile \theea < 2 <reascning>",
_i
"generations":
The point $ (1, in rectangular the point lies on the = sys-axis, so &r 3.$ The angle $1lthetas is MV
the angle between $90*Vecire" and 9180"\eeire$ that is measured the positiveclockvise from - the positive $x8-axis).
</reasoning>
<answer>
\boxed (3, \/srac/\pp](21))
[utnep==eeeet.lulxx//eeeoxxxttpd >>
< |uoottextissseebbbtexx//ddeeeede
< [ntoefteees///ooooooooox///ddeee
< onfoffteext..//moooffffxxxxxxxnd
[gnnofteeee//xnnnootteeeeeeeeedd >>
< ( >>
< /xnootteeec|//eeooofffeeeex/dddd
endoftext
< [x
Ground Truth
Wllef 3, ANfrac(\lpi) (2) \rright)
34

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
The results cross-countzy team's training run are graphed below. Which student has the greatest average speed? [asy]
for ( int i = 1; i <= 7; ++1
{
draw 0) - (i,6)) 6));
for int i 1; i <= 5; ++i
draw ((0,1) i));
draw (--0.5,0) -- (8, linewidth (1));
( -0.500000000 (00 linewidth (1));
(1"S0\", (0,,)) SW);
label (ot (90) +\ "ddistance""' (0, 3), W);
" (4, 0), S);
dot 4.5));
label label ( (1.25, 4.8), N)
(2.,5 2.2));
Label(sccalee..B)))>\iiaaaa (2.5, 2.2), 8);
(ot((225,55.2);
label((sclee..B)))\\ara...
(4.25, 5.2), SE); ;
dot (5.66 2.8));
label (s))
(5.6, 2.8), N);
dot (6.8, 1.4)); (sbale "Angela\", (6.8, 1.4), E); [/asy]
Model Generation
To determine which student has the greatest average speed, we need to look at the height of each column the graph.
The height of each column represents the distance covered time. The student with the greatest average speed will have the highest column. The graph shows that the highest for Carla is 5.2 units, which is the highest among all the students. Therefore, Carla has the greatest average speed.
by the student in a certain amount
</easoning>>
<answer>
\bboxe/ Ntext [Carla))
Ground Truth
\ttet ( [Evelyn)
35

---

Published as a conference paper at ICLR 2026
Sample Generations
Question
What is the distance, in units, between the points (2, -6)
+ and (-4, $) Express your answer in simplest radical form.
Model Generation
The distance between two points (xx],, y_1)s and (x_2, y_2)$ is given by WVssrt (x_2 x_ (y_2 -2s)
Flug, in &ilsqrt (6 (-4))
(-6))
(6'g 9-2) \3sgrt 81) (1sq)t
</reasoning>
<answer> \boxee>>\ooxxd//33 l3lgt
(eotsirx/oot_ii]><<>( |>"">"
Ground Truth
3\ (Ngggt((13)
STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
This work made use of large language models to assist with proofreading and improving the clarity of the writing. All research ideas, theoretical development, and experiments were carried out solely by the authors. When used for coding, was solely used for plotting purposes.
36


Reviews:
[{"review_id": "llbm98Tm5A", "reviewer": "Reviewer_TVg1", "summary": "The paper deal with RLHF with diffusion language models (dLLM). RL methods for fine-tuning LLMs need likelihoods, however DLMs do not provide generation likelihoods. Thus current RL methods either (a) use cheap approximate likelihoods or (b) use the expensive to compute ELBO (which also is approximate but consistent). This work proposes Group Diffusion Policy Optimization (GDPO), which uses ELBO for the likelihood but instead of the. standard Monte-Carlo ELBO, uses low-order quadrature integration to produce a lower variance estimate. Their results show improved performance over both the LLADA and the more recent d1/diffu-GRPO model on a variety of tasks.", "questions": "Some questions have been mentioned in the previous section.Additional to those:\n\nI would like to see some more ablations with varying N (quadrature), not just the accuracy.\n\n Since the experiments are only with small N (upto 4), what happens if we use a larger N. Since the estimate should get better with higher N, I am also confused why you have small optimal N, instead of as high an N as computationally feasible.\n\nSince there is non-trivial amount of additional computations to compute their approximation to the ELBO, a comparison with baseline models under same computational cost (FLOPS, wall-clock or any other) is important.\n\nIs there some easy metric or other diagnostics to determine if the used N is too small.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem of inefficient likelihoods for dLLMs is an interesting problem. The semi-deterministic Monte Carlo (SDMC) using quadrature integration is a neat, targeted variance-reduction idea and new to the best of my knowledge.\nExperiments are not just limited to more standard tasks like Countdown and GSM used in dLLM literature, but also code related ones.\nThe paper also provides error analysis for the SDMC estimator, providing insight for determining when their method might give better results.", "weaknesses": "Theoretically, the claims rely on smoothness and convex/monotone properties of the integrand tied to KL forms and forward-process assumptions. It’s not fully clear these conditions generally hold.\n\nThere are other methods which have been proposed to address the likelihood problem for dLLM tuning. Some of these even focus specifically on variance reduction of the ELBO in dLLMs (for eg diffuCoder). These have not been compared against or even discussed.  \n\nThe improvements on GSM are minor (and are in fact worse on MATH). They are also worse than wd1, which is a missing baseline on multiple tasks. Similarly coding results are not compared against diffuCoder.\n\n[1] wd1 Weighted Policy Optimization for Reasoning in Diffusion Language Models\n[2]  PADRE: Pseudo-Likelihood based Alignment of Diffusion Language Models\n[3] Test-Time Alignment of Discrete Diffusion Models with Sequential Monte Carlo \n[4] DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N2ntgdPmD0", "reviewer": "Reviewer_2CTL", "summary": "This paper introduces Group Diffusion Policy Optimization (GDPO), a novel reinforcement learning algorithm tailored to enhance the reasoning capabilities of Diffusion Language Models (DLMs). At its heart, the work tackles a central challenge in applying RL to DLMs: the difficulty of performing tractable likelihood estimation, which has long obstructed conventional policy optimization. The authors pinpoint the high variance and computational burden of existing sequence-level ELBO estimators as the key bottleneck. To overcome this, GDPO pioneers a new, low-variance ELBO estimator built on a Semi-deterministic Monte Carlo scheme—motivated by a fine-grained breakdown of the underlying variance sources. Empirically, GDPO consistently outperforms the strong baseline diffu-GRPO across a variety of reasoning, mathematics, and coding benchmarks, demonstrating its practical strength in fine-tuning DLMs.", "questions": "1. How was the KL term in GDPO estimated? A more detailed explanation in the paper would be helpful, as this aspect currently seems underexplored.\n2. Have the authors experimented with techniques similar to those in GSPO, such as normalizing the reward by the length of the generated sequence?\n3. Is the variance reduction technique introduced in this method orthogonal to the approach mentioned in LLaDA 1.5? If so, could they be combined to further enhance performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel idea and introduces a more principled approach for applying GRPO to diffusion large language models. This perspective is insightful and offers valuable inspiration for the research community.\n2. GDPO demonstrates substantially better performance than the baseline diffu-GRPO on planning-intensive tasks such as Countdown and Sudoku, highlighting its advantage in structured reasoning and planning.\n3. The analysis of variance sources in ELBO approximation is well-motivated, and the proposed semi-deterministic Monte Carlo method appears effective in significantly reducing variance.", "weaknesses": "It remains unclear why GDPO underperforms diffu-GRPO on certain mathematical reasoning benchmarks (e.g., GSM8K and MATH 500). The authors are encouraged to provide deeper insight into the potential reasons behind this performance gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6RjZ84HdyD", "reviewer": "Reviewer_xyfr", "summary": "The paper studies the problem reinforcement learning with diffusion language models. A key challenge in RL with DLMs is that likelihoods are hard to estimate. Prior methods either used biased token-level proxies or high-variance sequence-level estimates. As an alternative the authors propose Group Diffusion Policy Optimization (GDPO), which keeps a GRPO-style group advantage but replaces token-level terms with the sequence-level ELBO estimated by a semi-deterministic Monte-Carlo, This consists of fixing a few diffusion timesteps via Gaussian quadrature (2–3 points) and using a single inner MC sample, yielding a provably lower-variance estimator than standard double-MC under tight evaluation budgets. The paper analyzes ELBO variance, showing timestep sampling dominates and  provides asymptotic error bounds/bias discussion for SDMC. Empirically, using LLaDA-8B as the base model, GDPO improves over the base models and over diffu-GRPO across math (GSM8K, MATH500), planning (Countdown, Sudoku), and coding (HumanEval, MBPP), including better length generalization to 512 tokens.", "questions": "* Are the results in Table 2 and 3 single seed for training and inference? If so I think it would be better to report mean@k. \n* The authors mention that their approach doesn't require too many compute resources, but there is not much detail about how much time training takes on the said resources and how it compares to say diffu-GRPO. Could you please clarify that?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper overall is quite well written. The problem is defined quite clearly and approach explained clearly. \n* The variance analysis in Section 3.2 is quite useful and clear motivation for the problem.\n* The variance reduction approach is explained clearly and coupled with useful theoretical results establishing lower variance. The authors also validate this empirically. \n* The resulting algorithm is fairly efficient and simple of implement.\n* The empirical results show consistent improvements over some baselines across a variety of different tasks (math, coding, planning).", "weaknesses": "* The paper misses some closely related prior work on RL fine-tuning of diffusion language models [1, 2, 3]. In particular, I would like to see a comparison on how the SDMC compares with the variance reduction scheme from [2]. I believe a comparison to these baselines would be critical. \n* I also think the empirical results would benefit from having some simple test-time baselines like majority voting. \n* The experiments are limited to a single base model. I think it would benefit the paper to have some experiments with another base model (e.g. https://github.com/DreamLM/Dream)\n\n\n[1] Venkatraman et al., 2024. Amortizing intractable inference in diffusion models for vision, language, and control.\n\n[2] Zhu et al. 2025. LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models.\n\n[3] Zekri and Boullé, 2025. Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NB4g8xlQLI", "reviewer": "Reviewer_fTLx", "summary": "This paper suggests that the direct multiplying token-wise probabilities (used in diffu-GRPO) is not a good estimation of the probability of the whole sequence. Considering the expensive cost strictly following the ELBO, the authors analyze the main cause of the estimation variance, and find that the randomness of sampling $t$ is the main cause. Thus, GDPO uses deterministic time to consider the estimation as a time integral to reduce variance. Experiments indicate that it outperforms the original LLaDA and diffu-GRPO on multiple benchmarks.", "questions": "I think the main weakness is the lack of experiments, see weaknesses for details.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- This paper provides a detailed analysis on the origin of the variance in diffu-GRPO, taking it as the motivation for deterministic $t$ decisions. This analysis is valuable for DLM RL algorithm designs.\n- The comparison between GDPO and diffu-GRPO is comprehensive, including multiple domains (math, planning, coding).\n- The paper is well written and easy to follow.", "weaknesses": "- I think the most severe problem is that the authors perform too few comparisons, only with LLaDA and diffu-GRPO. Actually, there have been much more related works on DLM RL, such as [1,2]. It is necessary to compare them directly.\n- I do not consider diffu-GRPO as a token-level RL method. Actually, it also follows the ELBO in L 195 to provide an approximation for the probability to generate the certain sequence, but with an extremely small number of sample times (N=1). Increasing this could also make the algorithm more stable. We agree that use deterministic $t$ may further reduce the variance. However, other than comparing with diffu-GRPO, the paper does not provide a comparison between this two methods under the same number of MC sample times.\n- There also lacks ablation studies to validate the effectiveness of the quadrature rule.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)