Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models
Authors: Tianren Ma
Mu Zhang
Yibing Wang
Qixiang Ye
Venue: ICLR 2026 Poster (2026)
Keywords: discrete diffusion
masked diffusion
math reasoning
image generation
reinforcement learning
GRPO
Primary area: generative models
TL;DR: We introduce **MaskGRPO**, the first viable approach to enable scalable multimodal reinforcement learning in discrete diffusion with effective importance sampling and modality-specific adaptations.

Abstract: Optimizing discrete diffusion model (DDM) with rewards remains a challenge—the non-autoregressive paradigm makes importance sampling intractable and rollout complex, puzzling reinforcement learning methods such as Group Relative Policy
Optimization (GRPO). In this study, we introduce **MaskGRPO**, the first viable approach to enable scalable multimodal reinforcement learning in discrete diffusion with effective importance sampling and modality-specific adaptations. To this end,
we first clarify the theoretical foundation for DDMs, which facilitates building an importance estimator that captures valuable token fluctuation for gradient updates. We then delicately tailored the rollout method for visual sequences, which yields diverse completions and reliable optimization gradients. Across math reasoning, coding, and visual generation benchmarks, MaskGRPO brings more stable and efficient updates, **doubling** reinforcement learning gains while speeding up training by up to **30%**. This study establishes MaskGRPO as a systematic policy optimization approach and the first practical way for discretized visual diffusion. The code is available at https://github.com/martian422/MaskGRPO.

Published as a conference paper at ICLR 2026
CONSOLIDATING REINFORCEMENT LEARNING FOR MULTIMODAL DISCRETE DIFFUSION MODELS
Tianren Ma, Mu Zhang, Yibing Wang, Qixiang Ye* University of Chinese Academy of Sciences mat annen 8@mai 1s v.cas.ac.om gxyelucas.ac.oo
ABSTRACT
Optimizing discrete diffusion model (DDM) with rewards remains challenge-the con-autoregressive paradigm makes importance sampling intractable and rollout complex, puzzling reinforcement learning methods such as Group Relative Policy Optimization (GRPO). In this study, we introduce MaskGRPO, the first viable approach enable scalable multimodal reinforcement learning in discrete diffusion with effective importance sampling and modality adaptations. To this end, we first clarify the theoretical foundation for DDMs, which facilitates building an importance estimator that captures valuable token fluctuation for gradient updates. We then delicately tailored the rollout method for visual sequences, which yields diverse completions and reliable optimization gradients. Across math reasoning, coding, and visual generation benchmarks, MaskGRPO brings more stable and effi cient updates, doubling reinforcement learning gains while speeding up training by up to 30 %. This study establishes MaskGRPO as systematic policy optimization approach and the first practical way for discretized visual diffusion. The code is available at https:: httgttttuub....m com ma rt an 422
GSMBK
MBPP
GonEval
HPSv3
Semi-AR sampler
reversed tokenss
45.4
0.90
Af-iike
re masked
$1.9
Importance estimation
8.81
Tg # exp (f, 9) - (o).
070
Daid
78.2
850
Bandom
10.0
0:5
masked
8:0
LLaDA
wi diffu-GRPO
MMaDA
w/ MaskGRPO (ours)
Emerge sampler reversed Image tokeens
Figure I: Left: MaskGRPO consistently improves the base model with significant RL income across text and image generation tasks. Right: an intuitive demonstration of our method, integrated with modality-speciff innovations on importance estimation and sampling methods.
INTRODUCTION
Recent progress of post-training generative models has been driven by advances in optimization algorithms, architectural design, and large-scale reward-based learning (Rafailov et al. 2023 Liu
2025 Deng et al. 2025). Among these, Group Relative Policy Optimization (GRPO) Shao al.
et al. 2024) has emerged as powerful and scalable paradigm, improving reasoning performance of large language models and enhancing preference alignment of visual generative models. However, extending such policy optimization to discrete diffusion models (DDMs) remains challenge.
Unlike autoregressive models that decode sequentially, discrete diffusion generates tokens in parallel at arbitrary positions. This parallelism complicates both rollout generation, where stochastic yet coherent samples are required (Liu et al.] 2025) for exploration, and importance estimation, which crucial for optimization (Schulman et al. 2017). Existing approaches offer only partial solutions: semi-aue aumi-autoregresssvve samplers (Ariila et al. 2025 Nie et al. (2225) mitigate inference issues for text,
"Corresponding author.

---

Published as a conference paper at ICLR 2026
while low-confidence re-masking for images (Chang et al. 2022) lack stochastie flexibility for robust group comparisons. Similarly, early attempts Zhao et al. (2025); Tang et al. (2025) at importance
estimation relied masking heuristics that violated conditioning assumptions. Monte Carlo-basel
estimators Zhuu et al. (2025) improve faithfulness but remain computationally expensive.
In this study, we propose MaskGRPO, consolidated extension of GRPO to multimodal discrete diffusion (shown in Fig. D built upon modality -ppecifc innovations in both sampling and importance estimation. MaskGRPO guided by the distinct structural properties of language and vision:
Language. While training native discrete diffusion models depart from the autoregressivg 7 paradigm, their prediction natural language still exhibits degree of "ARness" (Gong al. 2025); ): tokens
closer to observed context are predicted with higher certainty, and rollouts diverse as length extends. Leveraging this property, we introduce fading-out masking estimator, which progressively increases the masking rate toward later tokens with well-controlled randomness. This concentrates estimation on high-uncertainty regions, towards a more efficient and empirically reliable objective.
Vision. Images lack a sequential structure and exhibit strong global token correlations (Chan et al. 2024). We argue that effective likelihood estimation requires highly truncated mask rates to capture informative variation. Furthermore, we adapt sampler that relaxes rigid scheduling constraints in existing methods via probabilistic decoding. By encouraging diverse yet high-quality rollouts, the training process better aligns with the GRPO principle of exploiting group-relative advantages.
Beyond empirical results in mathematical reasoning and coding that almost double the income from RL, our method also demonstrates significant improvement text-image alignment, and visual fidelity. Building upon clarified foundation for DDMs, our analysis highlights that policy optimization in discrete diffusion is only effective when samplers and estimators designed in modality-aware fashion. Through these contributions, we build the first systematic GRPO approach for multimodal discrete diffusion. This establishes DDMs and points toward a more general theory of preference-driven optimization across modalities.
foundation for reward-based learning in
2 PRELIMINARIES
2.1 DISCRETE DIFFUSION MODEL
DDM defines a forward process on discrete variables by gradually corrupting tokens to absorbing state m through a continuous-ting Markov process. We denote clean data as zı=0 (Zg for short), and noise it gradually / + 1. Let Or be the noise scheduler (a monotonically decreasing survival function that satisfies ag = 1, 01 = 0 the corrupted data distribution at time determined as
Ft g Fr Fo, t). (x (ro, t) Cat( og + (1 - (1-a4)mm m))
Let (( m) be the indicator function that is activated only if the i-th position of x, is m. For linear scheduler, the objective is derived as the evidence lower bound (ELBO) of log To (x)l)x))
CDDM = -E,, Do.
8( m) log To (e( = -E,, æo. a. In xo)]-
(2)
We denote the loss term by Ext (0)) for later usage. For conditional generation where a prompt c given, we write Exp zolc) for simplicity. Following MDLM's deduction Sahoo al. (2024).
assume that the network can reconstruct 4o perfectly, we use (I( to approximate this denoising process, and get the sampling rule as
if v. St, 3t + m,
I-as I-ar
if m,
c T-o: n
(xt),
.r + m, Jt m, otherwise.
0.
In this paper, we autoregressive in its conventional sense of causally ordered next-token prediction.
2

---

Published as a conference paper at ICLR 2026
2.2 GRPO FOR AUTOREGRESSIVE MODEL
Formally, for each question с ~ D, GRPO samples a group of G responses (also addressed as rollouts) {or, 02,...,0 from the old policy Wilyy For rollout os the reward system gives action value r3 and the relative advantage of it is normalized as
T2 mean (ry) std( }/1))
A
(4)
For position k < loi1, the token-level importance is calculated as
Tolo"lc.055 = exp llog no no(ollc, x - log Tem (o c, o²²
(5)
Taw (o c, k²²)
With controlling the clip range, the reward component is defined
lo:1
R(0,c) c)
1 Σ 0
1- min (ptA,, 1- (1
(6)
Finally, the GRPO objective expressed as a reward-penalty tradcoff, as
max BewDDen.oaaef [((0, c) 3I.. (To(-)c) mer((
where 3 regulates the strength the KL regularization.
2.3 ACCOMMODATING GRPO FOR DISCRETE DIFFUSION MODEL
We accommodate GRPO's objective for DDM in this section. To avoid confusion and align with the settings in Eq. 7, we move the timestep notation to the top-right corner of the variable. Accordingly, let of q L) denote the corrupted (reversed) response with strength I. Rolling back sequentially on AR model's response can be regarded as reversing on the timeline of o, i.e., the first k tokens ṭ* of AR model's 's' response be regarded re- re-masked with t jo, -k Hence, for each lo,
completion, we can approximate the sub-sequence level importance p' by gradually reversing it.
Recalling TTp from Eq. 2 for small interval öt, let o' o' - 0 denote the tokens that unmasked from timestep + öt to We propose that the differentiation on DDM's intractable log-likelihood be approximatedPusing
log (o'lc,
log T2(ō"|c, o'+1) Lo, - (o olc)
(8)
The above expression indicates that, to evaluate the fluctuation of likelihood for newly unmasked tokens o', we can utilize the difference model's prediction for the full sequence at time t. Hence, we derive the calculable importance estimation and KL divergence
p = exp (Coo (of, 0,|c) breLY (o' ozl)).
Dit KL exp (brege (o), o.lc) L. (0) or cc) - (Lroae a (0% os|c) Ln (0), oilc) - (10)
We temporarily skip the clip operation for simplicity and accommodate Eq. aas
A
max Egw Eo-DooLa
(
t, t= j/P
(11)
O
Upon this foundation, we revisit the prior endeavors on DDM optimization: diffu-GRPO Zhao et al. (2025) applies masks prompts and extract likelihood on the entirely masked completions o" == Following L.LDD- -.5 Zhu et al. 2025), UniGRPO Yang et al. (2025) iteratively masks varying ratio of completions. While these strategies provide gradient signals, they either disrupt the conditional dependency pose high budget for Monte Carlo style estimations. In summary, current inefficiency of likelihood estimation ties DDM to limited settings and obscures its potential broader contexts, especially in reasoning or visual generation that may involve thousands of tokens per sample.
*While previous works omit correlating discussion, this approximation is critical for deduction and code implementation. Please refer the Appendix C for details.
3

---

Published as a conference paper at ICLR 2026
AR-like Reversing
Random Reversing
-20
122
C6 Position
160
Tolea Posttion
Prompt (nover maskedy:
Completion (not masked];
Masked completion token:
Figure 2: A demonstration of different reversing (re-mask) methods with ratio 0.6. Random reversing (right) applies masks to all the tokens with equal probability, while AR-like reversing (left) adapts a fading-out strategy. See Appendix for complete showcases.
MASKGRPO
3.1 IMPORTANCE ESTIMATION
Importance sampling is central to the GRPO objective, where it functions an effective "reweighting" of rewards to align with the current policy's distribution with reduced bias. In practice, the importance calculated by the differentiate of predictions as Eq. Inspired by model rollouts' behavior, we modify the estimator to capture valuable fluctuation instead of highly confident predictions from low-mask sequences.
First, to allocate the timestep budget effectively, we clamp the sampling range from (0. to (% 1). where y serves as cut-off of mask ratios. Second, rather than relying on randomly masking, we design estimators that utilize the autoregressive biases in language and docality-driverr correlations in vision. We implement the reverse process by managing independent random seeds on each device, which is crucial for stable importance and KI. computation. With the designed operator, we obtain stable and low-variance estimates (see 4.3 for details) driven by the stochasticity of ô, Rev(o, t).
Let language tokens fade out. DDMs exhibit causal bias for language (Gong et al.) 2025), particularly logically related tasks such as math and code. This property, referred to as AR-ness, has been identified to have strong correlation with model's overall performance. Besides, the semi-autte aumi--utogressive sampler (Alg. is utilized, the rollouts also exhibit higher divergence as block extends, i.e., at the start of response, the model's 's reasoning divergence emerges as the reasoning proceeds (see (see D.I for evidence). This observation motivates exploit the importance estimation through AR-like reversing procedure, and assign higher attention to the later tokens. Alg. maintains delicate balance between randommess and fading-our property with no additional calculation, and serves plug-and-play module, which can bring significant improvement by simply replacing the original reverse method with our AR-like version.
rather simple setups, while real
Algorithm 1 AR-like Reversing (for text, ours)
Algorithm 2 Random Reversing (for image)
Require:
Token batch X E [R prompt indi- Require: Token batch E TR BxL prompt indi- cator C. mask token m. seed ratio
cator C, mask token m, seed ratio
Set random seed with
randomness is managed Set random seed with
#
2: Prompt (padded) length L. + C
3: Non-prompt length T.o + I Tue
# similar as in q( ()
4: Linear decay d 4 linspace(1,0. I.o)
2: Constant curve p p.
5: Normalize P.,
d-((1-))L.)
3: Expand to batch size P <- repeat(p. B)
Σ
4: Sample random matrix R U(0, B)L
6: p + Ohe θ Pp.P P +- repeat(p, B)
7: ~ (O, BxI. M + (-C) (R P)
5: Determine mask M + (-C) (R RRR< P)
8: Apply masking X +- where M, m, m,
6: Apply masking X + where(M, M,M.M.M.... X)
9: return X, M
7: return M
4

---

Published as a conference paper at ICLR 2026
3.2
ROLLOUT ADAPTION
MaskGil Sampler
Lioeighg
Emerge Sampler (ours)
Figure 3: A comparison of MaskGIT-style sampler and emerge sampler. With identical sampling parameters on MMaDA (equipped with a 8192-vocab visual tokenizer Xie et al. (024)), images sampled by the emerge method demonstrate better texture and expressiveness.
Let visual tokens emerge from masks. Complementing the estimators, align the rollout sampler with structural patterns, enabling efficient and stable training with GRPO. For text generation, we employ the widely adopted semi-autoregressive sampler (Alg. 3 which uses low-confidence re-mask strategy with block-wise decoding, reflecting the inherently sequential structure of language.
While expressed discrete vocabularies, visual tokens differ from language significantly in terms of entropy, bits information and so on (Chan et al. 2024). The confidence-basee MaskGIT sampler, which was proposed with 1024-vocab tokenizer, while viable, does not perform expected for high-fidelity tokenizers. This curse has been partly discussed as sampling inaccuracy (Zheng et al.] 2025; Ma al., (025a)), yet noticed that it becomes severe on large-vocabulary visual tokenizers and cannot be addressed by simply operating at higher precision. To overcome this problem which hinders model's potential on visual generation,
refer [Shhoo al. (2024): ; Shi al. (2024) and adapt their sample intuitiveness to discrete image generation. As shown in Fig and Alg. 4 the emerge sampler does not enforce decoding quantity per prediction, but let the visual tokens emerge from masks naturally with probabilistic control. While faithful to the principled DDM theory (Eq. the emerge method shows significantly better expressiveness for vision. Cat(-, T) is implemented with torch. mult t inomial.
As for reversing, unlike continuous diffusion RL methods Liu et al. (0255) where the importance is calculated across almost all traversed timesteps, we find that discrete visual diffusion requires large truncation on reverse range. The tokenized patches show strong global correlations, making the prediction largely insensitive to small mask ratios, and small truncation may even lead to exploded variance. Therefore, we keep the reversing random as Alg. 2 while the reverse strength held at high level, e.g., setting 7 0.8 obtain meaningful importance estimates.
Algorithm 3 Semi- (for text)
Algorithm Token Emerge (for image, ours)
1: Inputs: prompt c, completion length L
Inputs: prompt c, completion length
2: Settings: block length Lblock, token per step 8
2: Settings: total steps K, scheduler az
3: Initialize:
m
3: Initialize: I1 +- m².
4: for = 1 to L/ Lotock do
4: for k 1 to K do
5: range [[b L) bLotock]
5: t K-k+1 -
K-k
6: Tb + 2 zranne
K 6: logits, folxt, c)
K
7: for
Lptock/ do Peonf + fz, c) rrange]
7: logits, + fo [2,, Ø)
8:
8:
logits t- logits + (logits - logits,)
9:
Th +- where(M; m, argmax((eent))) m.))
9:
π + boftmax((ogitss
10:
mie argsort()sscrr){ T-block k:s)
11:
Zh + where(me. m, 2x))
10:
9s + - T 6m
T-ar
12:
end for
11:
Tprod + i ~ Cat(f; qo)
13:
x: rr[rage]] <- 2+b
12:
Ig where(x; m, Tpredd x+))
14:
end for
13: end for
15: Return: fully unmasked sequence xr
14: Return: fully unmasked sequence To
5

---

Published as a conference paper at ICLR 2026
3.3 ALGORITHM FRAMEWORK
To summarize, we implement GRPO for discrete diffusion models with integrated efficient modality- specific importance estimators, and modification on visual sequence sampling. The overall framework consists of: (i) Sample a set of full completions for each prompt with method 3 4 (ii) for each completion o, generate multiple masked óz ~ Rev(o, 1), where Rev(,, 1) is our designed reverse
function with controllable randomness, as shown in Alg. - (iīi) estimating per-completion
advantages A based on reward r, and (iv) updating the policy using importance jzs and divergence
DiL An algorithmic demonstration of MaskGRPO provided in Alg. 5
Algorithm 5 MaskGRPO Policy Gradient Optimization (ours)
Require: Reference model Ted, prompt distribution D, number of completions per prompt G, number of inner updates μ4, random seed set S S==lmw
1: Initialize policy <- Tref
2: while not converged do
3: Told < TT
4: Sample a prompt D
5: Sample G' completions o T- (. c), i E [G]
6: For each o,, compute reward r; and advantage A, using Eq.
7:
for gradient update iterations j = 1,..../ do
8:
Get timestep: ty + + - 7
9:
Construct masked completion Oi,t, t,)
For Ne, Told, use Eq. 9 10 to estimate importance P and Di with o,,t,
10:
Compute MaskGRPO objective Eq. and update mo via gradient descent
12:
end for
13:
end while
14:
return m
EXPERIMENT
4.1
SETUP
We conduct experiments across multimodal scenarios and evaluate MaskGRPO extensively math reasoning, coding, and eot-tto-mmaee generation benchmarks. In this section, we provide brief demonstration with training and evaluation details in Appendix.
Models We select
Nie et al. 2025 open-sourced native DDM and its
multimodal adaption MMaDA- -B-aae-B-BBase Yang et al. (2025), which unlocks the ability to perform discrete diffusion on image sequences, the start point of optimization. Both models are initialized from publicly available pre-trained checkpoints.
Reward function We follow common practices and use reward system for reinforcement learning. For language tasks, we utilize simple composed reward function of formatting and correctness. For image generation, following the practice recent RL work Geng et al. (2025), the reward is composed with UnifiedReward Wang al. (2025a), for scoring text-image alignment, HPSv3 Ma cet al. (2025b), for assessing the aesthetic quality the image alongside its alignment, and the classic CLIP Score Hessel et al. (2022).
Metrics We evaluate the proposed MaskGRPO framework on text and image generation tasks, using a suite of standard benchmarks to assess its performance gain. (1) Text Generation. To evaluate model's mathematical reasoning and coding capabilities, we use GSM8K Cobbe al. (202T), MATH500 Lightman et al. (2023) and MBPP Austin et al. (202T) benchmarks. We also compare with the recent RL baselines including difPu Zhao et al. (2025), wall (Tang al.) (2025) and UniGRPO tre-implemented due to unavailable codebase) Yang et al. (2025) on these tasks. (2) Image Generation. To evaluate model's text-image alignment, first utilize the widely adopted

---

Published as a conference paper at ICLR 2026
Table Evaluation on math reasoning and coding benchmarks. For fair comparison, we choose MAADA-8B-Instrue as the initial point. All results are reported with Pass metric. refers to our re-implementation.
GSM8K
MATH500
MBPP
RL Method Seq Len
512
256
LLaDA-8BB-nsstruc
76.7
78.2
32.4
36.2
39.0
diffi-GRPOO /han
han et al. 2025
79.8 (+3.1)
81.9 (+3.7)
34.4 (+2.0)
39.0 (+2.8)
42.1 (+3.1)
w/ UniGRPO Yang al.
2025
81.1 (+4.4)
(+3.8)
35.0 (+2.6)
38.8 (+2.6)
43.1 (+4.1)
w/ wdll Tang al. 2025
80.8 (+4.1)
82.3 (+4.1)
34.4 (+2.0)
39.0 (+2.8)
TraceRI, Wang
2025b
82.1 (+5.4)
83.3 (+5.1)
(+3.5)
(+3.3)
43.9(+4.9)
(askGRP((uurs)
84.7 (+8.0)
85.6 (+7.4)
37.6 (+5.2)
41.5 (+5.3)
45.4 (+6.4)
Table 2:
Evaluation on GenEval. SFT indicates that we SFT the base model with BLIP3-o dataset Chen et al. (2025a) for clean instruction-tuning data distilled from GPT-4o.
GenEval
Model
Single.
Two.
Count.
Color.
Pos.
Attr.
Overall
Continuous Generation
SDXL Podell et al. 2023
0.98
0.74
0.39
0.85
0.15
0.23
0.55
DALL
E Betker et al. 2023
0.96
0.87
0.47
0.83
0,43
0,45
0.67
SD3.5-L Esser
2024
0.98
0.89
0.73
0.83
0.34
0.47
0.71
FLUX.
Labs
0.98
0.93
0.75
0.93
0.68
0.65
0.82
Discrete Generation
Show-o Xie et al. (2024
0.95
0.52
0.49
0.82
0.11
0.28
0.53
Janus-Pro Chen
2225bb
0.99
0.89
0.59
0.90
0.79
0.66
0.80
MMaDA Yang et al.
2025
0.96
0.60
0.45
0,81
0.14
0.25
0.56
w/ UniGRPO Yang et al. 2025
0.99
0.76
0.61
0.84
0.20
0.37
0.63
w/ MaskGRPO (ours)
0.99
0.90
0.75
0.88
0.65
0.69
0.81
w/ SFT+MaskGRPO (ours)
0.99
0.92
0.85
0.91
0.91
0.83
0.90
GenEval Ghosh et al. (2023) and DPG-Bench et al. (2024) (see Appendix as the metrics. Then, we evaluate the generated samples' aesthetic quality using human preference scorers like DeQA You et al. (2025), ImageReward Xu et al. (2023). and HPSv3 Ma et al. (0255).. For references rather than detinitive comparisons, we include both specialized diffusion models, such SDXL, and leading discrete generation models like Show-o Xie et al. (2024) and Janus-Pro Chen et al. (002b)).
4.2 PERFORMANCE AND COMPARISON RESULTS
Language Tasks MaskGRPO substantially enhances the mathematical reasoning and coding capabilities LLaDA. As shown in Table [] our method achieves over 5% absolute improvement in accuracy on GSM8K. MATH500, and MBPP. nearly doubling the RL gains compared to prior methods with less steps (6000 vs 7000+), On GSM8K, MaskGRPO allows the model to surpass previous approaches while requiring only half the completion length (256 vs 512), demonstrating its effective improvement in reasoning ability. Representative examples are included in Appendix F3
Visual Generation Tasks To our knowledge, MaskGRPO is the first method achieve effective GRPO optimization of aesthetic quality and text-image alignment in discrete diffusion models. 3 shows consistent improvements alignment with human preferences, which are not reported in previous DDM works. Moreover, results in GenEval (Table 2 further confirm the effectiveness of our framework: with a well-designed RL setup. discrete generation models can approach the performance of leading commercial systems. Qualitative samples provided in Fig. Notably, while UniGRPO claims its mastery of visual RL, neither training configuration nor performance comparison is reported. Therefore, we index its final performance RL. results.
7

---

Published as a conference paper at ICLR 2026
de mog wnly sieting that vecuec airel an occecft
diesed sicee thin
wed tnglesdoon den compata for spece sertudung compledty and taes nandan tivciie overatal quqparo.
aberad said capkcs and
abae capicten catyyaal abee
whine offied collocaoe deep mn woelt cunduu belt anctiiii rakk war tane, cooiing no cotro ahott
han the caido uthoo cuts shido Lewwn grund.
ghe thouugh criett Batten dylls the longtennd pigs Bul us dayyccaa hoved ss sctes aoee auttee seael words in
dentr beoinin, oo cocertuaaa cuntnst lack
per de bo deiing resuaa grerr ago chin aroiue unte
foeee ba actl bi the of a landirg
Figure 4: Qualitative comparison. Results generated with identical sampling parameters and shown (original, w/ RL) pairs. MaskGRPO demonstrates substantial improvement on the aesthetic quality of generated images, in terms of artistic style, photographic details and overall atmosphere. We strongly recommend that the readers view more portrait samples at Fig. 10
Table Evaluation on compositional generation and human preference metrics. We calculate the Preference Scores samples generated by DPG-Bench prompts.
Model
Compositional Generation
Preference Scores
GenEval
DPG-Bench
DeQA
ImageReward
HPSv3
MMaDA
0.56
0.71
3.99
0.93
8.81
w/ MaskGRPO
0.81
0.75
4.10
1.18
9,40
SFT+MaskGRPO
0.90
0.82
4.18
1.30
9.63
4.3 DISCUSSION
Truncation hyper-paramend parameters We perform ablation studies on GSM8K with timestep truncation ratios 0.2, 0.4, 0.6, 0.8 for 4000 steps. As shown in Fig. (a). both the absence of truncation and overly aggressive truncation degrade training stability. To promote stable learning rather than premature convergence, adopt 0.6 the default setting.
Reverse method Concurrent work TraceRL Wang et al. (2025b) proposes track the generation trace and reverses strictly along these recorded traces. This mechanism relies on predefined paths and has only been demonstrated on block-attention architecture (SDAR JetAstra-MIL (L(2a)) (2025)). Moreover, TraceRL requires maintaining trace maps throughout training, and its deterministic reversal leads to limited flexibility in estimating prior tokens. We re-implement TraceRL on the full-attention language model, namely LLaDA-8BB ---A----Insttcct. and report results in Tab, In addition, fix 7 0.6 and ablate the reverse strategies Fig. (b). Our proposed AR-like reversing method consistently outperforms TraceRL in reinforcement learning. We attribute this performance gap to TraceRL's path-dependent formulation, which constrains exploration and induces biased estimation of sequence- level importance. For further qualitative evidence, we provide visualized comparison of reversing strategies under varying ratios in Appendix F.I
KL divergence comparison We also present the evolution of KL divergence during RL training across different reverse strategies in Fig. bottoom. A gradual increase divergence is typical in GRPO-style optimization, reflecting the model's shift away from the reference distribution. For
8

---

Published as a conference paper at ICLR 2026
GSMBK
GSMBK y 0.6
0.03
0.03
Val ddat on Conmero ass
Reward
2.50
0.00
Elan
AR-like reversing (ours)
0083
Trace reversing
0.76
Random reversing
0.62
1.00
2500
1903
1.0
MaskGRPO (ours)
UniGRPO
0.8
diffu-GRPO
Divergence
0.4
0.0
1000 2000
3000
6000
7000
8000
Figure 5: Ablative results. a: ablation on timestep truncation in language tasks. b: ablation on reverse methods in language tasks. Bottom: KL divergence during RL training under different reverse strategies. See text for detailed explanation.
clarity of visualization, discard divergence values above 1.5 prior before smoothing and plot these filtered points as markers above the x-axis to indicate their frequency.
In math-reasoning tasks, occasional large divergence values expected: they signal that the policy is exploring predictions that differ substantially from the reference model. However, frequent spikes can introduce instability and slow convergence. UniGRPO exhibits high divergence from the outset and continues to generate sharp fluctuations throughout training, indicating unstable updates. By contrast, difPu owing
deterministic masking of all response regions during reversal, largely eliminates such spikes but also restricts exploration. MaskGRPO achieves more favorable balance: it maintains stable training dynamics comparable to diffu-GRPO while preserving healthy level of divergence that supports effective exploration. Please note that, while KL divergence provides useful diagnostic insight, should not be treated as a standalone measure of training quality. A complete assessment requires considering performance trends alongside final task outcomes.
Rollout comparison We investigate how the proposed emerge sampler improves generation quality under reinforcement learning. As an initial step, substitute the vanilla MaskGIT style sampler with our method and evaluate the performance on MMaDA using GenEval. Before RL, our method produces samples with better textures, but the GenEval score worse than that of the vanilla method (0.51 vs. 0.56). This discrepancy arises because some of our outputs exhibit unstable or deformed object boundaries, which negatively affect detector-based metrics.
After RL training, however, these instabilities are largely eliminated. The emerge sampler not only facilitates broader exploration during policy optimization, but also guides the model toward higher- quality local optima that would otherwise be inaccessible to MaskGIT sampling. Consequently, our method achieves a higher GenEval score (0.81 vs. 0.77), while also producing more stable and expressive generations. See (6) This progression highlights key advantage: although the emerge sampler may underperform at the pre-RL stage, its enhanced exploration dynamics ultimately lead to stronger convergence and superior sample quality compared to the vanilla baseline.

---

Published as a conference paper at ICLR 2026
UNIGRPO
MaskGRPO
Figure 6: Visualization on GenEval. Prompt: photo of suitcase lest of a banana.
Ablation on design choices. We evaluate the impact of our reverse and sampling choices within MaskGRPO.
Table 4: Ablation on math reasoning.
The results, summarized in Tables and demonstrate the effectiveness of each component.
Design
GSMSK
Baseline (difft-GRPO)
79.8
+3.1
For GSM8K (256 completion length), the transition to AR-like reversing provides the largest single jump in per- formance. This confirms that utilizing the autoregressive bias critical for importance estimation, aligning with the divergence analysis.
+ Managed Randomness
80.4
+3.7
AR-like Rev.
83.5
+6.8
Truncation, etc.
84.7
+8.0
Table 5: Ablation on image generation.
In the visual domain, the emerge sampler amplifies the gains more than 20 points. Finally, the algorithmic modifications including ratio truncation act as a universal stabilizer, consistently improving performance across both modalities, confirming that MaskGRPO effectively consolidates these RL improvements for DDMs.
Design
GenEval
Baseline (UniGRPO)
0.63
+0.07
+ Truncation, etc.
0.75
+0.19
Emerge Sampler
0.81
+0.25
CONCLUSION
In this work, we introduced MaskGRPO, modality-aware extension of Group Relative Policy Optimization for discrete diffusion models. Recalling rollout sampling and likelihood estimation, we developed tailored strategies for language and vision generation: fading-out masking for text and probabilistic decoding for images. Our experiments demonstrate that these design choices substan- tially improve reasoning accuracy, text-image alignment, and sample diversity. These results highlight the importance of modality-specific samplers and estimators for effective policy optimization, and pave the way for unified reinforcement learning approaches across multimodal discrete diffusion.
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China under Grant 62225208. 62521007 and CAS Project for Young Scientists in Basic Research under Grant No. YSBR-SBRIII
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block Diffusion: Interpolating Between Autore- gressive and Diffusion Language Models. arXiv preprint arXiv 2503.09573, May 2025. doi: 10.48550/arxiv.2553.0995
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carric Cai, Michael Terry, Quae Lc, and Charles Sutton. Program Synthesis with Large Language Models. arXiv preprint arXiv 2108.07732, August 2021. doi: 10.48550/arXiv. 2108.07732.
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and
10

---

Published as a conference paper at ICLR 2026
Aditya Ramesh. Improving Image Generation with Better Captions, 2023. URL cdn openai wcm/papers/dall-a p.
David M. Chan, Rodolfo Corona, Joonyong Park, Cheol Jun Cho, Yutong Bai, and Trevor Darrell. Analyzing The Language Visual Tokens. arXiv preprint arXiv 2411.05001, November 2024.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked Genera- tive Image Transformer. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 11305-11315. IEEE, 2022. doi: N0.1109/CVPS52688... 2022.01182222110000
Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset. arXiv preprint arXiv 2505.09568, May 2025a. doi: 104485500a2XX..2555500566
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. arXiv preprint arXiv 2501. 1001178111
January 2025b. doi:
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers Solve Math Word Problems. arXiv preprint arXiv 2110.14168, November 2021.
doi: 10.48550/arxiv.211 10. 10.14168.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging Properties in Unified Multimodal Pretraining. arXiv:2505.1480 14683, July 2025. doi: (0.485550/rxxv.255555 14683.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling Rectified Flow Trans- formers for figh-Resolution Image Synthesis. arXiv preprint arXiv 2403.03206. March 2024. doi: 11485500arXxv.2443033322
Zigang Geng, Yibing Wang, Yeyao Ma, Chen Li, Yongming Rao, Shuyang Gu, Zhao Zhong, Qinglin Lu, Han Hu, Xiaosong Zhang, Linus, Di Wang, and Jie Jiang, X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again. arXiv preprint arXiv 2507.22058, July 2025. doi: 10448500arXi..250.22088.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Object-Focused Framework for Evaluating Text-to-Image Alignment. Annual Conference on
GENEVAL:
An
Advances in Neural Information Processing Systems 37:
Neural Information Processing Systems 2023, NeurlPS 2023, 2023.
Shansan Gong, Ruixiang Zhang. Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. arXiv preprint arXiv 20639, June 2025. doi: 10.48550WarXi..2556.2000
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. arXiv preprint arXiv 2104.08718, March 2022. doi: 211.5500/
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. arXiv preprint arXiv 2403.05135, March 2024. doi: 10.48550/uxxiv.2403.555//
JetAstra-MI. JetAstra/SDARR 2025. URL gi wwi.hub.cccm com/ wtt.oo///JJtttttrtoo ctt.thcccm///ttttttraaaaaaaaaa/a
Xuan Ju. Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu. Pnp inversion: Boosting iestusion-based editing with lines of code. The Twelfth International Conference on Learning Representations, 2024.
11

---

Published as a conference paper at ICLR 2026
Black Forest Labs. Flux-dev.1, 2025. UBL.gi.habb.ccm/blaacc-freeeee
flux
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long. Xinchi Deng, Yingfang Zhang. Xingchao Liu, Minbin Huang. Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang. Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng Fang, Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin. Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu. Xiao Xiao, Yan Chen, Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, and Qinglin Lu. Hunyuan-DiT: A Powerful
Diffusion Transformer with Frainer Chinese Understanding. arXiv preprint arXiv 2405.08748, May 2024. doi: 10. 144555//aXx..2445.888788
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's Verify Step by Step. arXiv preprint arXiv 2305.20050, May 2023. doi: 10.48550/arxiv22005200555
Jie Liu, Gongye Liu, Jiajun Liang. Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-GRPO: Training Flow Matching Models via Online RL. arXiv preprint arXiv 2505.05470, July 2025. doi: 10448550/aXx..5555.55770
Tianren Ma. Xiaosong Zhang, Boyu Yang, Junlan Feng, and Qixiang Ye. ReDDiT: Rehashing Noise for Discrete Visual Generation. arXiv preprint arXiv 2505.196566 2025a. doi: 10.48550/arXiv. 2505.19655
Yuhang Ma. Yunhao Shui, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. HPSv3: Towards Wide- Spectrum Human Preference Score. arXiv preprint arXiv 2508 03789, August 2025b. doi: 10.
48550/arXiv.2500 250..x00....008088.8.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. arXiv preprint arXiv 2502.09992, February 2025. doi: 10.48550/axXi..2522.00m
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolutton Image Synthesis. arXiv preprint Xiv 2307.01952, July 2023. doi: 1185500arxix223070019222
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly Reward Model. arXiv preprint arXiv 2305.18299 2023.
Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusson Language Mod- els. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC. Canada, December 10 15. 2024. arviv, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint arXiv 1707.06347, August 2017. doi: 10.48550/arXiv. 1707.06347.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv 2402.03300, April 2024. doi: 10444550AarXxx.2402200020
Jiaxin Shi, Kehang Han, Zhe Wang, Arnand Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. NeurIPS, 37: 17:10131--10311 2024.
Xiaohang Tang, Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. Wdl: Weighted Policy Optimization for Reasoning in Diffusion Language Models. arXiv preprint arXiy 008888888. July 2025. doi: 00.8550/arXiv.2557.0o0 008888888
12

---

Published as a conference paper at ICLR 2026
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang. Yueze Wang. Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang. Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next- Token Prediction is All You Need. arXiv preprint arviv 2409. 18869, September 2024.
Yihin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified Reward Model for Multimodal Understanding and Generation. arXiv preprint arXiv 2503.05236, March 2025a. doi: 10.48550/
arXiv.2503.05236.
Yinjie Wang, Ling Yang, Bowen Li, Ye Tian, Ke Shen, and Mengdi Wang. Revolutionizing Re- inforcement Learning Framework for Diffusion Large Language Models. arXiv preprint arXiv 2509.06949, September 2025b. doi: 10.48550/arXi..25000066 06994..
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One Single Transformer to Unify Multimodal Understanding and Generation. arXiv preprint arXiv:2408. 10..:228881288, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and Evaluating Human Preferences for Text-to-Imag Gener- ation. Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, 2023.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Shen, Yunhai Tong, and Mengdi Wang. MMaDA: Multimodal Large Diffusion Language Models. arXiv preprint arXiv 2505. 10.485500arXxi22555..555
2505.15809. 15809, May 2025. doi:
Zhiyuan You, Xin Cai, Jinjin Gu, Tianfan Xue, and Chao Dong. Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution. arXiv preprint arXiv 2501.11561.
March 2025. doi: 10.48550/a2Xi.2501.1115
Siyan Zhao, Devaansh Gupta, Qinging Zheng, and Aditya Grover. DI: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning. arXiv preprint arXiv 2504. 12211, June 2025. doi: 10.485500arxxv.25044, 12216.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.net, 2025.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, OpenReview.ned 2025.
Fengqi Zhu, Rongzhen Wang. Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. LLaDA 1.5: Seriance-Reducee Preference Optimization for Large Language Diffusion Models. arXiv preprint arXiv 1922., May 2025. doi: 21855500rxx..22551122223.
A ETHICAL STATEMENTS
We have provided key algorithms in the main text. Further implementation details are available in the source code. In the process drafting this paper, Large Language Models (LLMs) were solely utilized for grammatical checking. No LLM was involved in core academic work such as conceptualization, literature review, data analysis, or argument construction of this study.
As the human authors of this paper, bear full and sole responsibility for the paper's content, including the accuracy of research data, validity of academic arguments, integrity of research methods, and compliance with academic ethics.
13

---

Published as a conference paper at ICLR 2026
B RELATED WORK
Discrete Visual Diffusion Models Discrete diffusion, its core idea of predicting multiple discrete targets, has been applied to visual generation with prior endeavors like MaskGiT/Chang et al. (2022), where low-confidence re-mask sampler with Gumbel noise is used for inference to enhance sample quality. However, this can limit output diversity and makes reliable likelihood estimation challenging (Zheng al. 2025 Ma al. 2025a), creating bottleneck for online RI.
Text Diffusion Models Discrete diffusion models have emerged as powerful altemative (Sahoo it al. 2024, Nie et al. (2025) to autoregressive paradigms for language generation. Recent works Zhu al. (2025); Gong et al. (2025) have demonstrated competitive performance reasoning tasks. To obtain optimal results, while training in fully random noised corpses, current srate-of-the-and performances are usually obtained via aunneesvtreeressivee decoding (Arriola et al., [025] Nie et al.
2025). While this inconsistency leaves space for more sophisticated training design, it also partially demonstrates the causal nature of language modality.
Group Relative Policy Optimization GRPO and related reward-based optimization techniques have shown success in reinforcement learning for text generation and reasoning autoregressive LLMs (Shao et al. 2024). As continuous diffusion has been integrated into the unified model's framework (Deng al.) 2025 Zhou al. 2025). Liu et (2025) also adapt this RL paradigm in recent works via designed SDE solver. However, GRPO's s application is fundamentally limited in discrete diffusion models, as renders intractability on computing importance sampling weight, due to the lack of a factorized likelihood (Zhu et al. 2025).
C DISCUSSING THE APPROXIMATION
C.I ELBO ESTIMATION
The intractable log-lik likelihood log no (y(2)) DDMs is often approximated from its ELBO:
(ylz)) A By (//,t y/æ) s log (yla).
(12)
And LLaDA-1.5 provided an estimation with proved low variance as
M 5 m) log nolu v/z) z log (y)x),
(x(+)) =
(13)
where 1, = j/v is a discretization of i, and n is sampled as yy ~ <ly 2 y, 42) In practice the time- weighted term is absorbed in l an average on masked tokens, with the simplified form written as:
1 Σ (x (x(x))=
(14)
j=1
C.2 IMPORTANCE ESTIMATION
We discuss the importance estimation in Eq. 8 Note that do not intend to establish striet proof for this, but kindly discuss the viable implementation which is shared among current RI. works. First, let o's subscript index & temporarily represent its k-th element, we recall the loss expression as
|o]
(x olc)
8 m) log (ouloo'.c).
(15)
Given partially masked completion o², the above term describes deviation between model's prediction p
ा on masked positions. We also use of - ++t to denote the tokens that are unmasked at timestep t, with öt
d'1 For clarity, let (c, t8t) y o¹. Using the low-variance estimation To]
14

---

Published as a conference paper at ICLR 2026
in Eq. 14
discuss Eq. 8 follows:
log n (y]æz) log (2(()x))
(v/z)) -Ba(
Σ
en (1 Ex y/a)
(16)
Considering the small incremental of y comparing to æ, and the computational resource allocated to the inner-loop y is limited, we can make further approximation by calculating step łj 1, where all tokens in of are pending:
cx ,0/z))))))))))))))))( Lo (v',, ,
ex, (ml y/(²) - ETD (ml (ml),)
20
1=1
lxy (ò', olc, otc,) l ,0c, of
lx l (6' + tôt ./c) lā + o olc)
(,,(0', - ( olc)
(17)
D ADDITIONAL MATERIAL
D.1 DIVERSITY IN GENERATION TRAJECTORIES
T=0.3
Average Edit Distance
T=0.5
T=0.9
389
Token Position
Figure 7: A visualization of diversity in trajectories.
We provide an empirical support for the observation that text tokens exhibit increasing trajectory diversity generation progresses. For fixed prompt, generate N 16 independent continu- ations across different sampling temperatures (T)) which directly control the output entropy. The fixed at gen len/step /tll//lo__k_ll llnc Ien in Figure 7 plot the Average Pairwise Levenshtein Edit Distance among partial generations up to token positions. The consistently positive slope of all curves confirms that the divergence between generation trajectories grows significantly as token position extends. Furthermore, the sensitivity to T shows that higher 7' accelerates this divergence, resulting in a steeper curve, while lower temperatures (e.g., T = 0.1) maintain higher stability and determinism in the early stages of generation, validating the premise for focusing on later tokens LLaDA's rollouts. During GRPO training, the temperature is set to T 0.9 for language tasks (the default value of GRPO trainer, same as other methods) throughout the training process.
generation parameters
5. The curves
D.2 IMAGE EDITING EXAMPLES
We conducted preliminary study image editing, which is a key capability for modern generative models and remains largely underexplored for discrete approaches. We chose the PIE- WIE-Benchpp Ju et al. (2024) test suite for evaluation. Representative examples of the results included in Fig, 8
15

---

Published as a conference paper at ICLR 2026
Target: [fox] is shown (smiling] in [high polygonal style]
Target: [white] plate with grilled [salmon] [and mushroom] on
Target: [parrot] on [flower] in the garden
Target: [realism style] [painting standing branch
Original Image
Emerge method (ours)
MaskGIT method
Figure 8: A visualization of image editing examples.
16

---

Published as a conference paper at ICLR 2026
The image editing process relies on reverse masking strategy to selectively keep remove parts of the image tokens o before regeneration. We investigated two distinct reverse masking methods: (1) Baseline (Random Masking): This method involves re-masking 70% of the total image tokens, corresponding to 0%.5 under the cosine scheduler. (2) Score-based Masking: To preserve the image structure relevant to the original content, we input the original image and the modified prompt into the model. We extract the predicted logits of the original image's corresponding tokens as "scores," which estimate the model's confidence in preserving the original content. We then mask the 70% of tokens associated with the least scores, thereby attempting to keep the most confident structural elements unmasked.
The re-meree e-ssaee image sequence is subsequently fed into the model for the completion steps. We use 9 additional steps to generate the edited image. We compared the standard MaskGIT method with the Emerge sampler under this editing setting. The quantitative results are summarized in Table
Table Quantitative results for Image Editing on PIE-Bench.
Masking Method
Sampler
Structure Distance +
LPIPS
SSIM t
Random
MaskGIT
141.90
410.13
50.78
Random
Emerge
103.68
270.07
60.52
Score-based
MaskGIT
23.55
90.91
77.44
Score-based
Emerge
21.90
87.40
78.07
The results demonstrate that the Emerge sampler consistently produces more coherent and higher- fidelity edits, as evidenced by superior metrics (lower Structure Distance, lower LPIPS, and higher SSIM) across both masking strategies. Notably, the combination of score-basee based masking with the Emerge sampler yields the best performance.
That said, we note that strong image editing performance typically requires large-scale supervised training, which MMaDA was not trained on. Consequently, these results serve as early evidence of the framework's potential in image manipulation.
E IMPLEMENTATION DETAILS
E. TRAINING
For language tasks, Following the practice of diffu-GRPO, we conduct with a similar learning rate of
3e -6,6, rollouts per prompt G 6, iteration μ = 12, and a global batch size of 96 (bs = on 8x A 100 GPU, with gradient accumulation tokens per step. Let global steps be the update steps divided by iterations, we train max = 500 global steps all tasks, which takes up to 25% fewer training steps than that of diffu-GRPO.
2). The rollout is sampled with a block length of 16, and 2
For GenEval task, we use rollouts per prompt G = for exploration, iteration = 8, accumulation 4, and global batch size of 256. Each rollout is sampled with our emerge sampler, using 12 steps at CFG= 3.5 with a cosine scheduler. The RI. training takes 1500 global steps.
E.2 EVALUATION
For language tasks, evaluate all tasks with (-shot prompting. We block length of 16 and decodes 2 tokens per step for math tasks, and the MBPP protocol is specified in the following paragraph. All performances are reported using the pass@1 metric. For image generation, the sampler decodes an visual sequence of 1024 tokens (which represents an image with resolution 512 512) with steps, and is equipped with classifier-fi ffre guidance at 3.5, consistent with the original MMaDA configuration.
MBPP Evaluation Protocol. We specify the standardized protocol used for evaluating models on the Mostly Basic Python Problems (MBPP) benchmark. We clarify this protocol to address the significant variance in results reported in the literature, which stems from inconsistent settings for
17

---

Published as a conference paper at ICLR 2026
generation parameters Lree__een/stepbblooe len). different prompt designs, and distinet data subsets (e.g., sanitized json). Such variations impede direct model comparisons.
Following the evaluation setup of LLaDA-SB-Instruce test set consists of the first 500 samples (1-500) from the mbpp. jespp.jsce jaonl file in the official dataset. The evaluation is conducted in zero-shot setting, using the same prompt format the 1m-eval library. The generation parameters are fixed at gen_len/sttteeeeeeee Performance reported using the pass@) I metric, which measures the percentage of test cases passed on the first attempt.
specify our standard follows: The
htlp//bock___pppppppppppppplp/pp block_len = 256 / 255 32.
E.3 DATA USAGE
Language tasks. We use the standard training sets for GSM8K and MATH500. For MBPP, we follow DiffuCoder and use Acecode-87K, an open-source code dataset. Notably, diffi-GRPO reported using KodCoodeLight-RL-IO0
which refers to multiple traverse over dataset, given its reported 7500 steps. Besides, as the corresponding implementation is missing from its codebase, we re- implement and discovered limited effectiveness. We report its best performance on MBPP with our re-implementation Acecode-87K.
Image generation tasks. For the results reported in 2 we use filtered 15K GenEval-style prompts (provided in the repository). For general prompt following ability, followed X-Omni by randomly sampling 90K prompts from mi dijerneey-prrrr prompts, dataset of real user instructions, and augment it with 60K compositional GenEval-style prompts from Blip3-o (guaranteed that there is no overlap with the benchmarks).
Additionally, utilize instruction tuning data from Blip3-o for SFT. This procedure is optional, and we have denoted the corresponding results with explicit SFT mark in the tables. As observed, MMaDA's prompt following ability can be improved with such extremely clean supervision signals, and the generated images have more accurate demonstration of spatial relationship, and clearer boundary among objects, compared soley results. We train on this dataset for 1000 steps with global batch size 128, and a learning rate of 36-6.
E.4
REWARD DESIGN
Language tasks. For GSM8K and MATH500, the reward consists of two components:
Correctness reward: returns for an extracted and correct final answer, and otherwise.
Format reward: returns 0.5 if the reasoning process is properly enclosed in and 0 otherwise.
For MBPP, adopt DiffuCoder's scheme, combining correctness and format rewards. The format reward ensures completions are wrapped in // 117 while the correctness reward tests generated code against predefined test cases.
Image generation tasks. For image generation tasks, the reward can be calculated with:
UnifiedReward, evaluates image-prompt alignment, divided by [0, 1].
HPSv3, assess visual quality and text-image alignment, the score is divided by 5 to an approximate range [0, 2].
CLIP Score, measures similarity between encoded text and image features, ranging from [0. 1] (typically 0.2-0.4). We retain this metric to mitigate reward hacking.
We recommend using the sum of these three components. In the updated version, we also support
using single GenEval score the default reward, the results are shown in 2
E.5 CLARIFICATION
We select LLaDA-8B AAA--B-Insttuet for language tasks and MMaDA-2 -B-AA--B-RR SB-Base for image generation tasks. Both models share similar architecture and initialized from LLaDA-8B-Bal While this choice
18

---

Published as a conference paper at ICLR 2026
does not affect our claims multimodal reinforcement learning, clarify our rationale: although MMaDA released a MixCoT checkpoint, its performance on math and coding tasks is severely limited. With reasoning enabled, it achieves only 48% accuracy on GSM8K, about 30% lower than LLaDA-8B-Instrue with same sampling parameters. We attribute this to potentially insufficient training suboptimal recipe leading catastrophic forgetring. Although MaskGRPO applied to MAADA-SB-MixCos yields an improvement of over 6%, the results are not comparable since prior works consistently use LLaDA the baseline.
F
MORE RESULTS
F.1 VISUALIZATION REVERSE METHODS
We provide visualization of trace reversing Wang et al. (2025b) and AR-like reversing methods in Fig- with block length 16. AR-like method balances the autoregressive bias and randomness, while trace method reverses the decoding path with rigid paths. The strict rracebbaedd method may also suffer from coarse approximations when applied to highly parallel decodings, e.g., visual generation scenarios, where hundreds of tokens will be updated within only few steps.
Figure A comparison of trace reversing (above) and AR-like reversing (below, ours).
F.2 IMAGE GENERATION
We report the complete evaluation results on DPG-Bench Tab. 7 We also provide more generated portrait samples in Fig. TO
19

---

Published as a conference paper at ICLR 2026
Table 7: Evaluation DPG-Bench. SFT indicates that tune the base model with BLIP3-o
dataset (distilled from GPT-4o) for clean supervision data before RL training.
DPG-Bench
Model
Global
Entity
Attribute
Relation
Other
Overall
Podel et al. 2023
83.27
82.43
80.91
86.76
80.41
74.65
Hunyuan-Dij Li ct al.
(2024)
84.59
80.59
88.01
74.36
86.41
78.87
Emu3-Gen Wang al. 2224
85.21
86.68
86.84
90.22
83.15
80.60
SD3-Medium Esser et a. 2024
87.90
91.01
88.83
80.70
88.68
84.08
MMaDA Yang al. (2025)
77.52
77.52
77.67
81.56
63.13
70.51
w/ MaskGRPO (ours)
80.92
79.52
85.41
83.05
69.12
75.81
w/ SFT+Mas (ours)
85.96
84.05
89.76
83.43
79.88
81.76
len
Figure 10: More generated portrait samples with identical prompt and sampling parameters (CFG=3.5, 64 step). Above: sampled by default MaskGGT-style sampler before RI. training. Below: sampled by emerge sampler after RI. training. Prompts available in the supplementary material.
20

---

Published as a conference paper at ICLR 2026
F.3 MATH REASONING
Generation Sequence Length: 256
Question: There are 29 pupils in class. The teacher has 9 coupons; each coupon can be redeemed for 100 bottles of apple juice. The teacher gives each student bottles of apple juice drink for lunch. After redeeming all her coupons and giving each student their apple juice lunch how many bottles of apple juice does the teacher have for herself?
<reasoning'> First, we need to determine the total number of bottles of apple juice the teacher redeem with her coupons. Each coupon can be redeemed for 100 bottles, and the teacher has coupons. Therefore, the total number of bottles the teacher can is:
100
900
Next, need calculate the total number of bottles of apple juice given the students. There are 29 pupils in the class, and each pupil receives 2 bottles. Therefore, the total number of bottles given the students is:
29 2 58
Now, we subtract the number of bottles given to the students from the total number of bottles the teacher redeem to find out how many bottles the teacher has for herself:
900
58 842
Thus, the teacher has 842 bottles of apple juice for herself.
/reasoning 842
Generation Sequence Length: 512
Question: watermelon farm produced 120 baby watermelons. Ashlyn went the farm after two months and found out that 30% of the watermelons were ready for harvest so she took them home. When she came back two weeks later 3/4 of the remaining melons were ready so she harvested them. How many melons were not ready to be harvested after the two weeks?
<reasoning>> To determine how many watermelons were not ready to be harvested after two
weeks, we need follow these steps:
1. Calculate the number of watermelons ready for harvest after two months.
Subtract the number of watermelons harvested after two months from the total number of watermelons to find the remaining watermelons.
3. Calculate the number watermelons ready for harvest two weeks later.
4. Subtract the number of watermelons harvested two weeks later from the remaining
watermelons to find the number watermelons not ready to harvested.
First, let's calculate the number of watermelons ready for harvest after two months:
30% of 120 0.30 120 = 36
So, 36 watermelons were ready for harvest after two months. Next, we subtract the number of watermelons harvested after two months from the total number of watermelons find the remaining watermelons:
120 36
So. there are 84 watermelons remaining after two months. Now, let's determine how many watermelons were ready for harvest two weeks later:
3 of 84
3
84 63
4
So, 63 watermelons ready for harvest weeks later. Finally, subtract the number of watermelons harvested two weeks later from the remaining watermelons to find the number of watermelons ready be harvested:
84 63 21
So, 21 watermelons were not ready to be harvested after two weeks,
/reasoning"> 21
21

---

Published as a conference paper at ICLR 2026
F.4 CODE GENERATION
Generation Sequence Length: 256 Question: You are an expert Python programmer and here is your task: Write a function to find the vertex of a parabola.
Response: To find the vertex of a parabola given by the equation
ax be +
you can use the following formulas:
The x-coordinate of the vertex given by
2a
2. The y-coordinate of the vertex is given by
b²
4a
Here's Python function that implements these formulas:
def parabola_verte wertex (a, b, c)
x_vertex -b / (2 * a)
y_vertex
(b**2) / (4 a) return (x_vertex, y_vertex)
Test cases:
"assert puraabll_vvrrexx(333]] 1.55),, True
"assert
222222222222222
True
"assert p1 4.0)", True
22


Reviews:
[{"review_id": "uOIh4AHpry", "reviewer": "Reviewer_mNG6", "summary": "This paper tackles reinforcement learning (policy-based methods such as GRPO, PPO) for discrete diffusion models (dLLMs for text generation, bidirectional discrete diffusion models for visual generation). The authors propose to improve GRPO with effective importance sampling and modality-specific adaptations to the text and visual generation domains. For text generation, it proposes a semi-autoregressive reverse masking strategy to align better with the inherent causality in reasoning tasks. For visual generation, it adopts a probability-based sampler for better visual textures. The method was shown to improve LLaDA and MMaDA on math, coding, and compositional visual generation.", "questions": "See \"weaknesses\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The AR-ness introduced back to the masking scheme to improve text-based reasoning tasks looks convincing.\n+ Evaluation on several benchmarks shows promising results.", "weaknesses": "+ The overall novelty is a bit limited. The AR-ness that could help dLLMs with reasoning tasks has been noted in previous work. While there are improvements over MaskGIT sampler for visual decoding and sampling, the latter is a relatively old method. Visual generation is evaluated only on text-to-image generation, which is not the biggest advantage of discrete diffusion models. There could be an evaluation of image editing to make the results more significant (however, recent approaches like Transfusion [1] and BAGEL [2] do explore continuous diffusion heads instead of discrete diffusion models for the visual generation part).\n+ Minor point: “3.2 ROLLOUT ADAPTION” was placed in the wrong place. Shouldn’t it be right before “Let visual tokens emerge from masks”?\n\n[1] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model\n\n[2] Emerging Properties in Unified Multimodal Pretraining", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5KpKWeT4o", "reviewer": "Reviewer_q71V", "summary": "The work proposes a method adapting GRPO for multimodal discrete diffusion models, by using modality specific sampling algorithms for rollouts (semi-autoregressive for text, and random probabilistic unmasking for images), as well as modality specific remasking strategies to use for updates in the GRPO objective (autoregressive-like remasking for text and standard random remasking for images). This is done along with limiting GRPO updates to trajectories in the time interval $(\\gamma, 1)$. The modified GRPO method is used to fine-tune LLaDA for text reasoning tasks as well as MMaDA (a multimodal discrete diffusion model) for text-image alignment and aesthetic quality, where it is shown to outperform alternate RL methods.", "questions": "1. For text, if tokens later on in generation (more towards the end of generation) are more useful for training, in what sense does the AR-like process assign them “higher attention”. \n    - This appears to conflict with the truncation ratio being set at $\\gamma > 0$  since it will exclude a group of tokens near the end of the (block-autoregressive) generation.\n    \n2. Is the masking rate multiplier ($\\frac1t$ for linear) used in the likelihood computation at the token-level (for instance Equation 8)? The notation in Equation (2) implies it is.\n    - If it is, is the masking rate adjusted for the alternate remasking strategies considered (namely autoregressive-like remasking)?\n\n3. The mechanism behind why smaller truncation ratios lead to training failure (for image fine-tuning) is unclear. An explanation is given in terms of samples having stronger correlations between image patches, but its unclear to me why this results in collapse.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. A comprehensive set of baselines are evaluated for experiments.\n2. The generated image samples in Figures 4, 7 are convincing in terms of improvement.\n3. The ablations with truncation ratio $\\gamma$ are helpful in establishing its impact (Figure 5).", "weaknesses": "1. The writing needs polishing and proofreading\n    - There are numerous grammatical mistakes that interfere with the clarity of presentation\n    - The citation format is incorrect (using `\\citet` rather than `\\citep` )\n    - The Emerge sampler is described unclearly in text (lines 238-247)\n\n2. The Emerge sampler (algorithm 4) doesn’t seem novel, and appears to be the same as the usual random unmasking from masked diffusion models (eg. without confidence based heuristics for unmasking). This is discussed in the original MDLM paper (Sahoo et al., 2024) as well as other early work on masked diffusion (eg. (Shi et al., 2024)). Despite this, the text appears to frame the sampler as a novel contribution (eg. line 244 ‘we refer to MDLM … and propose the … sampling strategy’ and line 247, ‘our sampler’). This should be clarified.\n\n3. A number of claims are made which require more detail or justification:\n    - The motivation for the autoregressive-like remasking invokes an observation that tokens with high entropy provide more informative signal for training, and that later tokens (in AR generation) tend to diverge more. This should be supported by some evidence from rollouts\n    - Remasking with the $\\mathrm{Rev}(\\cdot, t)$ operator is asserted to be more stable and have low-variance (line 192, and also on line 189, the estimator is asserted to be “low-discrepancy”)  (I am assuming, compared to random remasking) - but this statement should be verified explicitly.\n    - TraceRL is claimed to induce “biased estimation of sequence-level importance” - why is the method biased, compared to remasking with $\\mathrm{Rev}(\\cdot,t)$? This seems important since the introduction of a new remasking strategy is a core aspect of the method, and the straightforward thing appears to be reusing the partially masked completion obtained during rollouts.\n\n4. It would be helpful to list the time taken to achieve the reward improvement (or some proxy), since the importance weight computation in this method appears to be more computationally intensive than the mean-field approximation for token level likelihood used in diffu-GRPO \n\nI recommend for a reject, mainly due to point 2 above, which I view as critically important.\n\nJiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and\ngeneralized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUUOpOsi6L", "reviewer": "Reviewer_Extk", "summary": "The paper proposes MaskGRPO, an RL framework extending GRPO to multimodal DDMs. It introduces modality specific rollout strategies with AR-like reversing for text and probabilistic emerging for vision, along with a low-variance importance estimator for stable and scalable optimization across discrete diffusion. Experiments show consistent gains in reasoning, coding, and visual generation.", "questions": "1. How is the mask ratio in the Rev(·, t) operator scheduled? Is it fixed or dynamically adjusted during RL training?\n\n2. How does the training cost of MaskGRPO compare with other diffusion-RL methods such as Diffu-GRPO?\n\n3. The paper introduces modality-specific rollout strategies, but it is unclear how sensitive the performance gains are to these design choices. Could the authors provide ablation or analysis showing how each component (AR-like reversing, emerging sampler) contributes to the overall gains?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem is clearly defined — the paper directly targets the limitation of static embeddings in collaborative code completion by proposing a dynamic, incremental embedding approach.\n\n2. The overall framework is well-structured and conceptually coherent, with a clear system design.\n\n3. The method improves both language and vision generation, demonstrating strong empirical utility.", "weaknesses": "The paper lacks discussion of training cost or comparison of resource consumption with baseline methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hWiA6Bl7n1", "reviewer": "Reviewer_pNLG", "summary": "The authors propose maskGRPO, a GRPO-style RL method for discrete diffusion models developed and shown to work on multimodal DDMs, both text and images. To achieve this, they propose a tractable importance estimator, and an approximated surrogate for importance weighting and KL in GRPO for DDMs. The authors claim more stable and efficient updates while showing improved performance on a set of both visual and language tasks.", "questions": "Can you provide either empirical or theoretical grounding in support of markGRPO being more efficient than similar methods?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is generally well written and easy to follow. The results are compelling enough and it's nice to see GRPO working on image DDMs. The formulations of the importance estimator for DDMs is also useful.", "weaknesses": "A cornerstone of the narrative of the paper seems to revolve around compute/sample efficiency, but I did not see a detailed tokens-to-gain or wall-clock comparison against the strongest off-policy GRPO / surrogate-policy approaches, or a theoretical note explaining why the method is expected to work more efficiently in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)