Inpainting-Guided Policy Optimization for Diffusion Large Language Models
Authors: Siyan Zhao
Mengchen Liu
Jing Huang
Miao Liu
Chenyu Wang
Bo Liu
Yuandong Tian
Guan Pang
Sean Bell
Aditya Grover
Feiyu Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Large Language Models
Reinforcement Learning
Inpainting
Group Relative Policy Optimization
Primary area: foundation or frontier models, including LLMs
TL;DR: IGPO, an RL method for diffusion LLMs that uses inpainting to inject partial reasoning hints when stuck with all-wrong responses, achieving SoTA results on math benchmarks for masked diffusion LLMs

Abstract: Masked diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive LLMs, offering competitive performance while supporting unique generation capabilities such as inpainting. We explore how inpainting can inform RL algorithm design for dLLMs. Aligning LLMs with reinforcement learning faces an exploration challenge: sparse reward signals and sample waste when models fail to discover correct solutions. While this inefficiency affects LLMs broadly, dLLMs offer a distinctive opportunity—their inpainting ability can guide exploration. We introduce IGPO (Inpainting Guided Policy Optimization), an RL framework that strategically inserts partial ground-truth reasoning traces during online sampling. Unlike providing full solutions, inpainting steers exploration toward promising trajectory spaces while preserving self-generated reasoning, bridging supervised fine-tuning and reinforcement learning.
We apply IGPO to group-based optimization methods such as GRPO, where exploration failures cause zero advantages and gradients. IGPO restores meaningful gradients while improving sample efficiency. We also propose supervised fine-tuning on synthetically rewritten concise traces that better align with dLLM generation patterns. With additional techniques including entropy-based filtering, our training recipe yields substantial gains across four mathematical benchmarks—GSM8K, Math500, AMC and Minerva—achieving new state-of-the-art results for full-attention masked dLLMs.

Published as a conference paper at ICLR 2026
INPAINTING- GUIDED POLICY OPTIMIZATION FOR DIF- FUSION LARGE LANGUAGE MODELS
Siyan Zhao!: 122
Mengehen Liu'
Jing Huang'
Miao Liu
Chenyu Wang' Feiyu Chen
Bo Liu'
Yuandong Tian' 1
Guan Pang'
Sean Bell'
Aditya Grover?
Meta Superintelligence Lab
UCCA
Tsinghua University
ABSTRACT
Masked diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive LLMs, offering competitive performance while supporting unique generation capabilities such as inpainting. We explore how inpainting can inform RL algorithm design for dILMss. Aligning LLMs with reinforcement learning faces an exploration challenge: sparse reward signals and sample waste when LLMs fail to discover correct solutions. While this ineffi- ciency affects LLMs broadly, dLLMs offer distinctive opportunity-theer in- painting ability can guide exploration. We introduce IGPO (Inpainting Guided Policy Optimization), an RL framework that strategically injects partial ground- truth reasoning traces during online sampling. Unlike providing full solutions, inpainting steers exploration toward promising trajectory spaces while preserv- ing self-generated reasoning, bridging supervised fine-tuning and reinforcement learning. We apply IGPO to group-based optimization methods such as GRPO, where exploration failures cause zero advantages and gradients. IGPO restores meaningful gradients while improving sample efficiency. We also propose pervised fine-tuning on synthetically rewritten concise traces that better align with dLLM generation patterns. With additional techniques including entropy- based filtering, our training recipe yields substantial gains across four mathe- matical benchmarks- GSM8K, Math500, AMC and
new state-of-the-art results for full-attention masked dLLMs. Our code is available here: https: / / g thub. com www.boooreeoeerrccctitca hwgp................o..
Masked dI.IM Inpainting Guided Sampling
Performance All-Wrong Groups Reduction
Abnetoog puatne
Profie:: "Jy can pager of beok in miissrr Hove exzry beacz "agge?"
Demoleing Step
[MASK]
[HSUS]]
B+3+24
[XASK]
[XAX]]
[MASK]
[MAaK]
[MASK]
Tainiing Stga
Your Mtops
Iterotive Demoising Steps
(c) Comparison with Prior Methods
Finul Gemeraatio::
Theze mimetes hous, so Jay reed B*2w24
hour. So.it will take bar 120/ 24 hours to
SOWeT SCL
zead all pagez. Thereforw, the snover this probleen
Fesd Eas Chanks
Igginttc Goseation
possk] Tobest
Figure 1: (a) Unlike autoregressive LLMs, diffusion LLMs can be conditioned on future reasoning hints during generation through inpainting via bidirectional attention, enabling guided exploration toward correct solutions. (b) Applying sosiinting-uuieen exploration in policy optimization outper- forms standard Group Relative Policy Optimization (GRPO) sampling and reduces all-wrong groups occurrences. (c) Our full training recipe combining Length-Aligned supervised fine-tuning on concise reasoning traces with IGPO achieves SoTA performance among full-attention masked dLLMs across four mathematical reasoning benchmarks.
Work done Meta

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent research has shown that masked diffusion large language models (dLLMs) (Austin et al., 2021; Lou al., 2024; Shi et al., 2024) such as LLaDA (Nie et al., 2025) and Dream (Ye et al., 2025) can achieve performance competitive with autoregressive LLMs of similar size. Their capabilities and performance can be further enhanced via RL post-training (Zhao et al., 2025; Gong et al., 2025b) and ability flexibly include multimodal data (Li et al., 2025; Yang et al., 2025; You et al., 2025). Unlike autoregressive LLMs, which decode in a left-to-right manner, dLLMs iteratively unmask tokens in parallel. This brings potential for faster inference as shown in closed models like Mercury (Inception Labs et al., 2025) and Gemini Diffusion (DeepMind, 2025), along with a flexible inductive bias for operations such inpainting, the ability to fill in missing content within existing text.
In this work, we explore how inpainting can be leveraged to inform post-training algorithms for dLLMs. Recent work on post-training of dLLMs has adopted training approaches similar autore- gressive LLMs, applying Reinforcement Learning with Verifiable Reward (RLVR) methods (Zhao al., 2025; Yang al., 2025; Gong et al., 2025b). However, fundamental exploration challenge persists: for challenging tasks, policies struggle to discover correct solutions and binary rewards provide minimal learning signal when most generated solutions are incorrect. This leads to substantial sample waste and poor training efficiency, exacerbating the computational costs of online RL.
The bidirectional generative structure of diffusion LLMs provides unique mechanism address this exploration challenge. Since dLLMs are trained through stochastic masking patterns, they possess inherent capability for accepting externally provided partial hints through inpuinting. We leverage this ability to introduce IGPO (Inpainting Guided Policy Optimization), novel RI framework that strategically guides exploration for dLLMs by injecting reasoning hints when answering difficult problems. Specifically, when the policy is unlikely to generate correct solutions, partial reasoning traces are injected into the generation region, and the dLLMM is tasked with completing the remaining reasoning sequence and output final answer. The final answers are verified against ground truth, and only successful completions are used for downstream policy optimization.
We demonstrate IGPO's effectiveness in group-based policy optimization methods such as GRPO (Shao et al., 2024), which are particularly vulnerable to exploration failures: when a group's responses are all incorrect, group-normalized advantage collapses to zero and resulting in zero gradi- ents. This occurs with alarming frequency in challenging domains. By reducing the prevalence of all-wrong groups, IGPO restores gradient signals and enables more effective RL. More broadly, IGPO can be viewed as a form of guided exploration that interpolates between supervised and RL paradigms. The injected tokens act as conditioning context that steers the policy's action distribution toward high-reward regions. Unlike pure SFT, which might suffer from distribution shift between data and policy rollouts (Zhang et al., 2025), IGPO maintains on-policy generation for the non-injected tokens. Finally, augment IGPO with techniques that improve learning stability and performance, including entropy-based gradient filtering for injected tokens, and conduct comprehensive experiments across math benchmarks. We evaluate each component of our approach through ablation studies. Our work makes the following key novel contributions:
We propose IGPO, the first work to utilize the unique inpainting capabilities of diffusion LLMs for reinforcement learning. By strategically injecting partial reasoning traces during exploration, IGPO alleviates the inefficiency of sparse verifiable rewards and mitigates the zero-advantage dilemmaa in group-based policy optimization methods, substantially reducing the proportion of all-wrong groups (by approximately 60% as shown in Fig 1 (b)) in training.
We propose Length-Aligned SFT for full-attention based dLLMMs using synthetically rewritten, concise reasoning traces. This design better aligns SFT data length with RL sampling and evaluation length, avoids the limitations of verbose traces, and provides stronger initialization for RL.
Our full training recipe achieves substantial improvements on mathematical benchmarks, including +5.3% on GSM8K, +8.4% on Math500, +11.4% on AMC, and +4.0 % on Minerva relative to the LLaDA-Instruct, achieving SoTA performance among full-attention based dLLMs.
We conduct a comprehensive ablation study that disentangles the mechanisms of IGPO. We show that partial inpainting consistently outperforms full ground-truth -ruth inpainting by staying closer to the policy distribution in online RL, and propose an catropy-basce gradient filtering mechanism that stabilizes training dynamics.
2

---

Published as a conference paper at ICLR 2026
We further show IGPO's robustness to imperfect reasoning traces and provide theoretical analysis of gradient recovery in the arranaae regime
PRELIMINARIES
2.1 MASKED DIFFUSION LARGE LANGUAGE MODELS
Masked diffusion LLMs (Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2024; Lou et al., 2024) employ a forward diffusion (masking) process that progressively corrupts clean sequences zo by introducing mask tokens. This process is indexed by continuous time E [0, 1]. At any timestep E (0, 1), the partially corrupted sequence T2 is obtained by independently masking tokens so that each token remains unmasked with probability decreasing in t. At t the sequence fully masked. Training specifies the forward process via a and learns a bidirectional unmasking predictor fo to recover the original tokens from 24. Each step samples E [o, 1). applies the forward masking to obtain 2; ~ (Ij (za), and optimizes masked- token objective derived from the negative evidence lower bound (NELBO), which upper-bounds the data negative log-likelihood (NLL). For masked dLLMs this NELBO reduces to a weighted NLL with weights determined by transforms of ar (Sahoo et al., 2024, Eq. (10)). For example, LLaDA (Nie et al., 2025) uses a linear schedule a = - t, leading to:
where the schedule au is strictly
[xc
(0,1,(0 mn Paass Ih o (mo)
1| maskk log fo xx)
(I)
where [xy] is the sequence length and z² the k-th token. The loss is computed only on tokens masked at time t. For prompt-conditional generation, prompt tokens are kept unmasked while continuation tokens are initialized as mask. The model then simulates a reverse process po (o( xts) over timesteps t 8, where fo provides denoising predictions for masked positions. Throughout the reverse trajectory, already unmasked tokens are preserved and carried forward unchanged.
2.2 POLICY OPTIMIZATION FOR MASKED DIFFUSION LARGE LANGUAGE MODELS
Policy-gradient post-training is widely used for LLM alignment (Ouyang al., 2022; Bai al., 2022; Li et al., 2023; Ahmadian et al., 2024). GRPO (Shao et al., 2024; Guo et al., 2025; Team et al., 2025) is value-free variant of PPO (Schulman et al., 2017) that uses group-wise, sequence-level advantages for G' responses {o, == to query
G
A r(o.)
Σ r(or).
j=1
The GRPO objective integrates ratio clipping and reverse- regularization:
(anro()) FF (11))
min A..lip -,, e. A.) BDa. (
(3)
where / To (o la,0 the probability ratio, r(.) is a reward function, 3 > 0 is the KI. regulariza- tion coefficient, >> is the clipping parameter, and Teet is the reference policy. autoregressive mod- els, the reverse-Ki is tractable via the chain rule, log (o q) = E log (o o**), but masked diffusion LLMs do not admit a left-to-right factorization because тy arises from composing reverse denoising steps of the mask predictor. To make GRPO practical for masked diffusion policies, DiffuGRPO (Zhao et al., 2025) adopts a mean-fiell approximation that yields single-pass estimators for token-level ratios and the reverse- KL; we these estimators throughout. We provide detailed background discussion in Appendix B.
mew (of F5.
METHODS
3.1 IGPO: INPAINTING GUIDED POLICY OPTIMIZATION
Zero-Advantage Dilemma. In the GRPO framework, when sampling responses (01, (n, for
given prompt q, the advantage computation relies reward variance across
3

---

Published as a conference paper at ICLR 2026
Augmented RL Sampling Group
Initial RL. Sampling Group
Inpainting-Cuided Guided Policy Optimization
Generation Area (All [Naak] Tokeus)
Response
Response 1:
Respunse 2:
Inpainting with Ground Truth Chunks
Respouse
Response 3: r
Response
Response 3 (6)) :
Gree:: Fixed chinls] Red: Engainted Geventions
X All advantages
Response 4 (0s) :
Inpainted Generations
Ground Truth Reasoning Traces
Non-zero advantages!
Groens Selected ehunks inpainting
Only succosefnl generntions (f adddedd gooonp
Figure 2: Overview of IGPO: When all sampled responses yield identical incorrect rewards (zero- advantage scenario), we perform hint-guided inpainting by generating additional responses using ground truth reasoning chunks injected hints. Ground truth traces y" segmented into variable- length chunks, and selected chunks are injected as fixed hints during generation while the model generates the remaining tokens. We then replace a fraction of the original incorrect responses with correct responses generated through inpainting, creating reward variance that enables non-zero advantages for effective policy gradient updates.
the group. However, when all responses receive identical rewards either all correct or all incorrect the advantages become zero: A, r Σラ= r(o,) O. This zero-advantage scenario makes the policy gradient component degenerate. Specifically, the clipped surrogate objective collapses to zero regardless of whether the update lies in the clipped or unclipped region, since both terms contain A; 0. The policy gradient for this prompt q therefore becomes:
lou|
A A.p. V log To(o ()) = 0 since A; = 0 Vi.
G
As result, no meaningful policy update can extracted from the reward signal, wasting compute sampling these responses. In this work, we specifically focus on mitigating the all-wrong case.
Masked dLLM Generation and Inpainting. In full-attention masked dLLM generation, the model input denoising step 0 is the concatenation [y: Rmasx], where q represents the prompt and Amassk |m sk, mask, ... ma sk denotes a fully masked completion sequence of predetermined length L. The generation process progressively unmasks these positions through iterative denoising until producing the final output.
Hint injection modifies this formulation by fixing selected positions of Zmask ground-truth tokens. During RL training, we assume access to ground-truth reasoning trace y" = lvr v2. ... Vig for every question g. For injection, we create binary mask m {0, 13 indicating which positions to inject as fixed hints, we construct the hint-injected initialization:
y mask
if m(i and S 1y²1.
hint
otherwise.
The masked dLLM then performs bidirectional denoising on [g: himl through the inpainting process, leveraging both the prompt and injected hint tokens generate coherent responses. The injected hint tokens remain fixed throughout the iterative denoising steps.
Constructing Hint Patterns for Inpainting. To construct meaningful hint patterns for the inpaint- ing process, segment the ground truth reasoning trace y² into variable-length contiguous chunks - (c1. Cz. CN }. where each chunk length Icyl is sampled from [[[smin Smax] We explicitly exclude the final answer tokens from chunking to prevent reward hacking behaviors where the model ignores reasoning and collapses. For a given hint injection ratio E [0, 1]. we randomly select [ [ chunks and set their corresponding positions in the binary mask to I for hint injection.
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 IGPO: Inpainting-Cudder Policy Optimization for Masked dLLMs
Require: Reference model Ted. prompt distribution D, ground-truth reasoning traces {y"), }, number of completions per prompt G, number of inner updates μs, hint injection ratio range [Tow, T/iggh) replacement fraction X, entropy filter threshold 7, chunk size range [Smin:: Smax
1: Initialize Te <- Tye
2: while not converged do
3: Told - Toi sample prompt q~ and responses 01.G ~ Tou (-14); compute rewards T1.G
4: if all r; = 0 (zero-advantage case) then
5:
Segment ground ground-truth truth reasoning y" into chunks fci..... cN } with lcyl ~ [e(Smin Smax 6: for i = 1....... G do
7:
Sample hint injection ratio 7J U ((Toww Thigh] and select [ chunks from
{ci..... CN] randomly
8:
Inject selected chunk tokens as fixed hints corresponding positions
9:
Generate o; via inpainting: denoise only masked positions, keep hint tokens fixed Evaluate rewards r(o;) and replace up to [ incorrect with correct ō;
10:
11:
Compute advantages A; the updated response set
12:
for n n= 1n..1.... do
Estimate log næ. log Told, log Trefi apply top-7 entropy filter on hint positions Update Ty via (f) (Eq. 5)
14:
15:
return T0
Elastic Inpainting- Triggered Sampling. With the above inpainting setup, we design IGPO (as in Algorithm I) to be elastic: hint injection is only triggered when all sampled responses in a group yield incorrect rewards (the zero-advantage case), and when activated, both the hint injection ratio and chunk sizes (l4 [mmin Smax) 1 are randomized to provide diverse training signals. Concretely, when detecting that all sampled responses {oI..... oG) for query y yield identical rewards r(o;) 0, we generate an additional set of responses {ōi..... oG} through the inpainting process. Each response o is generated via inpainting with a distinct hint injection ratio 7) Ll [[ooow Thigh] to ensure diverse hint densities. Following inpainting generation, we evaluate the correctness of {o:} and only use the correct ones for replacement. Specifically, we replace K min(Hō; : r(ā,) [AG]) of the original incorrect responses with correct responses generated through inpainting, where À E (0, 1) controls the replacement fraction.
The only modification introduced by IGPO lies in the sampling step: when the all-wrong condition detected, the original G on-policy responses are partially replaced by K srrrectness-verffiee inpainted samples. The IGPO objective therefore differs from the standard GRPO formulation only the sampling level; all other components remain unchanged. In particular, the advantages A, are computed normally according to Eq. 2. The resulting objective for an all-wrong group is identical to GRPO except for this sampling change, highlighted in blue below.
(f
min (st AB chip let -el -e) 233)) -ADaa
ADaa (ma(-lv) (ma
(5)
where CGPO-Sample(Tpg No: 9: y²) denotes the augmented sampling procedure that applies inpainting-
based augmentation when zero-advantage scenarios are detected, producing the augmented RL sampling group (o1...., KG-K ōj,... oK) containing (G - K) original responses and verified correct inpainted responses {ō:} after replacement. L, denotes the length of the i-th response (whether o; or o:). Crucially, only inpainted responses that pass correctness verification are included in the augmented group, satisfying r(ō,) 1. We built IGPO with DiffuGRPO (Zhao et al., 2025)'s log probability estimation methods, where all completion tokens are masked during estimation and we small
remove the random masking applied number of policy iterations (i.e. / = 4), this alleviates the need for random prompt masking to reduce overfitting, Inspired by Zheng et al. (2025), we compute sequence-level importance- through mean-field approximation for stability purposes.
prompt tokens as done in DiffuGRPO. Since
Entropy-based Gradient Filtering for Hint Tokens. When applying IGPO to zero-advantage scenarios, the responses generated through inpainting contain ground truth reasoning chunks that
5

---

Published as a conference paper at ICLR 2026
originate from different distribution than the current policy тe. This creates an off- off-policy learning scenario where gradient updates from ground truth tokens can confiitt with the model's current beliefs, particularly at positions where the model has high confidence (low entropy). To mitigate potential training instability from this distribution mismatch, we implement an entropy-based filtering approach that restricts learning hint token positions where the model exhibits sufficient uncertainty, as inspired by Huang et al. (2025). Specifically, for each hint token position (i.e., positions with injected ground-truth tokens) we compute the entropy. We then apply gradient updates only to the top percentile of hint token positions with highest entropy values. This selective learning strategy serves two purposes: high-entropy positions represent genuine decision boundaries where the model is naturally uncertain and thus more receptive to external guidance, and they correspond to flatter probability distributions that yield more stable gradient updates when incorporating ground truth information. This approach controls the policy shift by focusing learning on positions where the model is already open to change, rather than forcing updates against strong existing beliefs at low-entropy positions.
3.2 LENGTH- ALIGNED SFT VIA CONCISE REASONING TRACE REWRITING
To further strengthen our training recipe, we seek better RL initialization via SFT but identified generation length mismatches across SFT, RL sampling, and evaluation phases. Full-attention masked dLLMs like LLaDA lack KV cache optimization (Wu et al., 2025) by defualt, requiring full sequence attention at every denoising step, which dominates online RL. training cost. As result, we restrict RL rollouts to 256 tokens for faster convergence within a reduced exploration space, and evaluation setups in recent work (Zhao et al., 2025; Zhu et al., 2025; Nie et al., 2025) typically use 256-1024 tokens. In contrast, popular reasoning SFT corpora (c.g.. OpenRI) I) contain verbose traces often exceeding 10k tokens, creating distribution mismatch across SFT, RL, and evaluation, and include repeated reflective behaviors unsuited for limited context. To resolve this, we systematically rewrite verbose traces into concise, structured forms that preserve logical flow while respecting dLLM computational limits. Using
(Meta, 2025) with prompts detailed in Appendix I, we remove redundant reflections, condense multi-sentence elaborations into precise, mathematically rigorous statements, and retain essential reasoning. Examples of revision length distributions and before/after traces are in Appendix and I. Our Lengthh-llined SFT trains LLaDA solely on rewritten traces, improving RL initialization by avoiding implicit length compression and focusing learning on reasoning quality within fixed compute budgets. Empirical results show clear gains over training on verbose traces, and we further observe that masked dLLMs benefit from extended training (e.g.. 100 epochs) relative to AR LLMs, consistent with recent works (Ni and the team, 2025; Prabhudesai et al., 2025).
EXPERIMENTS
To investigate how the inpainting capabilities of masked dLLMs can address the exploration chal- lenges in RL and how Length-Aligned SFT improves performance, we conduct comprehensive experiments to answer the following main research questions:
(1) How effectively does our complete training approach (Length-aligned SFT with rewritten reason- ing traces followed by reinforcement learning with IGPO) improve the mathematical reasoning performance of LLaDA and reduce all-wrong groups occurrences? ($4.3)
(2) How does partial hint injection in IGPO bridge on-policy generation with ground truth guidance, and how does this improve learning compared to full supervision? ($4.4)
(3) How do key design iiooices-includee entropy filtering thresholds and reasoning trace rewrit- ing affect RL training dynamics and learning stability? ($4.4)
4.1 COMPLETE TRAINING RECIPE
Our complete learning framework consists of two-stage pipeline: Stage 1: Supervised Fine-Ture Fine-Tuning with Rewritten Traces. We begin with Length-Aligned SFT on the the OpenRI-Math-22
model using dataset's default split (94k math problems), but with all reasoning traces rewritten (See Appendix D for length distribution before and after revision). This ensures consistency between training distribution and downstream RL/evaluation phases by aligning trace lengths. Stage 2: Reinforcement Learning with IGPO. Following Length-aligned SFT, we apply IGPO to further enhance reasoning capabilities through strategic impainting-guiide policy optimization. We utilize the reasoning traces from the MetaMattQAA dataset for the elastic inpainting process, creating effective

---

Published as a conference paper at ICLR 2026
from LLaDA-Instruct
after Length-Aligned SFT
LLBS 1230
L 57. 1210
C300
4325
144
230
10R 1350 1483
153 30:0 2280
RLL Training Stops
RL Training Steps
Figure 3: RL training curves of IGPO versus normal GRPO sampling. (a) Starting from LLaDA- 8B-Instruct. (b) Starting from the length- tigtt--alinedd SFT checkpoint. IGPO exhibits superior and more stable training performance under both initialization checkpoints compared to standard GRPO sampling. Results are averaged over random seeds across four mathematical reasoning benchmarks (GSM8K, MATH500, AMC and Minerva Math), with standard errors shown as shaded regions.
guidance signals that fit within our computational constraints. Detailed training hyperparameters are provided in Appendix
4.2 EXPERIMENTAL. SETUP
We conduct experiments using LLaDA-8B-Innte as the base model with sampling temperature of 1.2 for RL online generation, where the temperature is selected based on exploration and exploitation analysis detailed in Appendix F.1. For reinforcement learning, dataset (Yu et al., 2023), specifically using the "Answer Augmentation" split and combining questions from both GSM8K and MATH500. After deduplicating identical questions, we obtain 12,794 unique training examples. For supervised fine-tuning, we utilize the OpenRI-Math rewritten reasoning traces as described in Section 3.2. We evaluate our approach on four mathematies benchmarks: GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), AMC (LI et al., 2024) and Minerva Math (Lewkowyez et al., 2022). Experiments are conducted on 8x8 80GB H100 GPUs. For UniGRPO (Yang et al., 2025) baseline, we reproduce based their Algorithm 1. We provide detailed experiment hyperparameter setups Appendix F and Appendix H.
train the MetaMathQA
dataset with
4.3 MAIN RESULTS
Table I: Performance across multiple mathematics tasks. GSM8K, MATH500 and Minerva are evaluated with @ 1 temperature of 0.0, and AMC with avg 16 temperature 0.1. Underlined scores indicate the best within each initialization group. Parenthesized deltas typeset via (o) denote absolute percentage- improvements relative the LLaDA--B-
baseline.
GSMSK
MATH500 (pass 1)
AMC (avg@ 16)
Minerva
Model
(pass @ 1)
(pass @ l)
Average
Similar autregressive L.Mss
LLaMA3-8B (Al@ 2024)
79.6
30.0
Qwen2.5-78 (Team, 2024)
85.4
49.8
Prior masked dLLM baselines
Dream-7B et al., 2025)
77.2
39.6
d1-LLaDA (Zhao al., 2025)
82.1
wdl (Tang et al., 2025)
82.3
39.0
LLaDA-I.5 (Zhu et al., 2025)
83.3
42.6
13.6
8.8
37.1
LLaDA-Instrul (Nie et al., 2025)
81.5
39.0
14.5:40
(.2
36.0 (000)
RI. from a. -nstruct
LLaDA-IIstruct Instruct + UniGRPO ( (Yang et al., 2025) DiffuGRPO
LLaDA
84.2
39.2 (00t)
15.0:40.:
11.0
36.9
al., 2025)
:404
40.2
15.5
10.3
LLDAA-Instrug IGPO (ours)
83.1
42.8 (4.8)
17.5 (4.5
12.1 (42m
38.9 38.9.429)
Length-aligned SFT on L.LDAA Instrct and RI. the SFT checkpoin
Lnnttetnttrutt Length-aligned SFT (ours)
83.6.22
45. (v.2
22.3:
10.3
10.4
J. Instruct + Lnngth SFT IGPO (ours)
86.8 (4.3))
47.4 wa.a)
25.9:.1144
13.2 (m
43.3 (9.))

---

Published as a conference paper at ICLR 2026
As shown in Table I, our training recipe demonstrates consistent improvements across all mathe- matical reasoning benchmarks. With gungh--liigedd SFT on rewritten traces, LLaDA achieves an average improvement of 4.4% compared to the base LLaDA-SB -B--trttuctt model. When applying IGPO on top of the SFT model, we observe additional improvements, resulting in a total average improvement of 7.3%. The complete two-stage pipeline yields cumulative improvements of 5.3% on GSM8K, 8.4% on MATH500, 11.4% on AMC, and 4.0% on Minerva relative to the LLaDA- baseline. Notably, on the challenging AMC benchmark, our approach achieves 25.9% (avg 16). As shown in Figure 3, IGPO exhibits superior training dynamics compared to standard GRPO sampling when initializing from before or after SFT. IGPO effectively reduces the all all-wrong group ratio by approximately 60%, as shown in Figure 1(b). Our final model (LLaDA + Length-Aligned SFT + IGPO) outperforms all baseline approaches including the recent LLaDA-1.. 1.5 model across all evaluated benchmarks. Notably, even without SFT, applying IGPO directly on L.LDD achieves better performance than the previous LLaDA-I. 1.5 and other RL methods for full-attention dLLMs, establishing new state-of-thes theftttthartttt art recipe for mathematical reasoning in masked diffusion language models.
4.4 ANALYSIS AND ABLATION STUDIES
Self-generated inpainted traces provide bet- ter learning signal than ground truth traces. The results in Figure show that partial hint in- jection achieves higher performance than full
Average Accuraty across Berchmarks
Hint Inject Ratio Comparison
0.470
Hint Inject Rado - 1.0
Ratto-
hint injection. When the hint injection ratio varies within the lower range, the model needs to generate self relforaaiin inpainting traces (with
0.465
Inpaint
0.460
an example shown in Appendix G), and only those that lead to correct solutions are added to the group for gradient updates. Through in- painting, the model attempts to coherently con- nect provided hint chunks with its own reason- ing steps. The inpainted generation produces learning signal that bridges the gap between the model's 's current capabilities and the target behavior. The self- self-generated portions reflect the model's 's current reasoning patterns and are more "on-policy" while incorporating structural guid- ance from ground truth chunks, resulting in more effective policy optimization compared to pure supervised learning, reducing the distributional mismatch. This bridging of SFT and online RL through partial self-generation leads to more effective policy optimization.
0.435
0.450
0.445
400
600
800
1000
1200
1400
RL Training Steps
Figure 4: Impact of hint injection ratio. across datasets (GSM8K, MATH500 and AMC) and seeds with standard eπrоr shown as shaded areas. We compare partial hint injection (7 (0.2, 0.6) versus full hint injection (1) 1.0). Partial hint injection consistently outperforms full hint injec- tion, demonstrating the benefits of self-generated reasoning. Both hint-guided inpainting variants outperform the baseline without any hint injection.
Entropy clipping prevents training instability from off-policy tokens. As shown in Figure 5a, we observe that learning from only the top 20% highest-entropy hint token positions (r 0.2) achieves the best performance and exhibits the most stable training dynamics. In contrast, lcarning from all hint token positions (T = 1.0) or large fraction (7 = 0.8) leads to more unstable training with performance fluctuations compared to lower values like 0.2. This empirical finding supports our motivation that restricting gradient updates to high-entropy positions prevents the destabilizing effects of large gradients on high-entropy positions. The validates the necessity of entropy- based filtering when incorporating ground truth traces from hint-guided inpainting into policy gradient training.
Effect of reasoning trace rewriting for SFT and subsequent RL training. The results in Figure 5b illustrate two key findings. First, SFT on rewritten reasoning traces produces substantially stronger checkpoints than SFT on the original traces. Our rewritten traces eliminate verbose reflection behaviors and compress reasoning into concise trajectories (up to 1024 tokens), which are better aligned with LLaDA's generation budget (256 tokens) and evaluation sequence length. This alignment improves SFT accuracy at step 0 relative models trained on the longer 4096-token traces. Second, while RL training can partially compensate for weaker SFT checkpoints the models trained on 4096-token traces recover accuracy rapidly in early RL steps-starting from stronger rewritten SFT
8

---

Published as a conference paper at ICLR 2026
Entropy basedd dradientt Filterring Hint Tokens (on GSM8K)
Length-Alignes SFT
6SMBK)
0.7
04r-
049-
082-
042
16.22 wning: parte
ba ans watoay zo thee
race 1w0 Gio
I0 1309 2000
38a 10. 120 201c
t0e 20c
360 3007 1300 2it1 1000
300 1200 1550 2he JES0
(b)
Figure 5: (a) Impact of entropy clipping threshold on hint tokens. Performance comparison across different entropy clipping thresholds applied to hint token positions in IGPO, where r 0.2
represents learning from only the top 20% highest-entropy hint token positions, while T = 1.0
indicates learning from all hint token positions without filtering. (b) SFT and RL dynamics with rewritten vs. original traces. We compare models fine-tuned on concise rewritten traces (max 1024 tokens) original OpenR1-Matt traces truncated at LLaDA's 4096 context limit. RL is then applied (GRPO or IGPO) to both models. Rewritten traces yield stronger SFT performance and superior RL outcomes. IGPO consistently outperforms GRPO with stable pass while GRPO suffers from diversity collapse. Results are run on GSM8K with temperature 0.1 and length 256.
checkpoints leads to consistently higher final performance. Importantly, across both initialization settings, IGPO outperforms standard RI. without inpainting. Additionally, IGPO preserves output diversity and stabilizes p performance throughout training, whereas standard GRPO exhibits a decline in pass@k k metrics, indicative of reduced exploration and mode collapse.
Elastic inpainting outperforms sequential SFT and GRPO We further validate the effectiveness of our elastic inpainting approach by comparing it against sequentially performing SFT on the RL dataset's 's reasoning traces followed by standard GRPO (see Appendix E for details). This ablation confirms that IGPO's ' elastic hint injection during zero-advantage scenarios is superior to uniformly applying SFT on concise reasoning traces across all prompts before applying GRPO. The uniform SFT approach can degrade initial performance due to distribution shift in reasoning patterns, whereas injecting partial hints allows dLLMs inpaint longer, more "on-policy" reasoning traces.
Robustness to noisy ground-truth reasoning traces. We further investigate how IGPO behaves when the injected ground-truth reasoning traces are imperfect. Following the setup described in Appendix J, we introduce synthetic corruption into reasoning traces by randomly perturbing tokens with probability p (including numerical mistakes, operator swaps, and logical inconsistencies). As shown in Appendix J and Figure 9, IGPO maintains strong performance even under moderate noise levels and consistently outperforms GRPO across all corruption rates. Performance gradually degrades as p increases, but the advantage of IGPO remains stable. This robustness arises from two design choices: only partial reasoning chunks are injected during inpainting and updates are gated by correctness verification and entropy filtering, which limits the impact of incorrect hint tokens. These results suggest that IGPO can effectively leverage imperfect reasoning traces without requiring perfectly clean supervision.
Comparison to compute-matcher GRPO. To determine whether IGPO's improvements arise from its hint-guided exploration rather than increased sampling, we compare against a compute- matched GRPO baseline that uses the same sampling budget. As detailed in Appendix L, the baseline (GRPO + resample) generates additional trajectories whenever group produces only incorrect responses, matching the total number of samples used by IGPO while remaining fully on-policy. Despite using the same number of trajectories, GRPO resample exhibits higher all-wrong ratio and lower training rewards, while IGPO consistently achieves better evaluation performance (see Figures 10 and 11). These results demonstrate that IGPO's gains do stem from increased compute or sampling volume, but from its targeted inpainting-base -bssd guidance, which provides directional exploration signals that pure resampling cannot replicate.

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
Diffusion Language Models:
Recent advances in diffusion language models have progressed from continuous approaches mapping discrete text to continuous representations (Chen et al., 2022; Li et al., 2022; Gong et al., 2023) to scaled discrete diffusion models, with masked diffusion emerging as a prominent approach (Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2024;
Nie et al., 2024). Notable developments include DiffuLLaMA (Gong et al., 2025a) and Dream (Ye et al., 2025) adapted from pretrained autoregressive LLMs, and LLaDA (Nie et al., 2025) as a masked diffusion LLM trained from scratch achieving comparable performance to autoregressive models. Commercial models like Mercury (Inception Labs et al., 2025) and Gemini Diffusion (DeepMind, 2025) have demonstrated practical viability with significantly faster inference. Reinforcement Learning for Diffusion Language Models: Applying RL to diffusion LLMs faces unique challenges due to intractable likelihood estimation, which is required for policy optimization. Recent solutions include diffu-GRPO (Zhao et al., 2025) with mean-field approximation, MMaDA (Yang et al., 2025) and coupled- GRPOO (Gong et al., 2025b) with improved masking strategies, LLaDA 1.5 (Zhu et al., 2025) addressing variance through preference optimization, wdl (Tang et al., 2025) eliminating policy ratios via weighted likelihood objectives, and SDPO (Han et al., 2025) decomposing trajectory alignment into stepwise subproblems. More detailed related works are discussed in Section C.
CONCLUSION
We introduced IGPO, a reinforcement learning algorithm that leverages the inpainting capabilities of masked diffusion language models. By injecting ground-truth reasoning hints during denoising. IGPO steers the policy toward high-reward regions and alleviates the exploration bottleneck in RL. It resolves the zero-advantage dilemma by inducing reward variance that supports effective policy gradient updates when standard sampling yields uniform outcomes. To further strengthen RL initialization, we proposed Length-Aligned SFT, which reduces the length mismatch across SFT, RL, and evaluation stages. Combined with entropy-based gradient filtering, our approach achieves new state-of-ther performance among full-attention masked dLLMs multiple mathematical reasoning benchmarks. These results highlight a new paradigm for reinforcement learning in masked diffusion language models, showing how architectural properties can be systematically exploited to address critical optimization challenges.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on algorithmic contributions to reinforcement learning and supervised fine-tuning for diffusion language models, specifically targeting mathematical reasoning tasks. Our research does not involve human subjects, does not collect or release new datasets containing personal information, and addresses computational methods for improving model performance on well- established mathematical benchmarks. The mathematical reasoning domain we target does not raise concerns about harmful applications, bias amplification, or misuse potential. Our methodology improvements are designed to enhance model accuracy and training efficiency, which we believe contributes positively the field of Al research. We have adhered standard research practices and have no conflicts of interest to declare.
REPRODUCIBILITY STATEMENT
We have made efforts ensure the reproducibility of work. Complete experimental details are provided in Table and Table the appendix, including all hyperparameters for both supervised fine-tuning and reinforcement learning phases. Our use of existing datasets (OpenR --MtRh-205t for SFT and MetaMathQA for RL training) is fully disclosed, and the revision prompt used for trace rewriting is provided in Section I. The evaluation methodology and metries for all four mathematical benchmarks (GSM8K, MATH500, AMC, and Minerva) are clearly specified. All algorithmic components of IGPO detailed in Algorithm with mathematical formulations provided throughout. We commit to releasing our implementation code upon publication to facilitate reproduction of our results.

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Ustün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.11 147440 2024.
AI@ Meta. Llama model card. 2024, https: /ttp::////thhut/t/ com wwt.ccoccmaaaae.eee... aeta- 11 lama 11ama3mammmm /bl
ob/main/ he/maann/MDDDE CARD mnd.
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. https: ptttps://kuunae //hwunlp..ppppp// github. io /blog//025/Poolat
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi. Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. The Thirteenth International Conference on Learning Representations, 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg, Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 1 798 1-19993, 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint M.Xiv:2204008662 2022.
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. ArXiv, abbs/2208.04222. 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 2021.
DeepMind. Gemini diffusion, 2025. bttps:://depee http://ttt/////////a/aooo//o//// fusion/.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq: Sequence to sequence text generation with diffusion models. In International Conference on Learning Representations, ICLR, 2023.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Yc, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference Learning Representations, 2025a.
Shansan Gong, Huangjie Zheng Ruixiang Zhang, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code 20639.
generation. 2025b. https: arxiv. org/
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Jiaqi Han, Austin Wang, Minkai Xu, Wenda Chu, Meihua Dang, Yisong Yue, and Stefano Ermon. Discrete diffusion trajectory alignment via stepwise decomposition, 2025. https: /arxiv. org/abs// 2507. 04832.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint Xi0i 100.0004433 2021.
Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S. Abdelfattah, Jae Sco, Zhiru Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided diffusion, 2025. https: : aaaiivi org/ abs /2505. 21 667.
12

---

Published as a conference paper at ICLR 2026
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M. Ponti, and Ivan Titov. Blending supervised and reinforcement fine-tuning with prefix sampling, 2025. https: //aaxxiv.org/aba....... /257. 01679.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Pairecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. Mercury: Ultra-fast language models based on diffusion. 2025, https:
2ttgg/aab//555.7729999 17298.
Daniel Israel, Guy Van den Broeck, and Aditya Grover. Accelerating diffusion Ilms via adaptive parallel decoding, 2025. https: paapp:://axiiin wwtp:////g//////aoo/o/oo//s 00413.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, pages 267-274, 2002.
Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. https: /openreview.n et et/ //foorrrmicct d= IFXT ZERXcM7.
Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath. https: / com/ com/pro///.eooooooooooo ject -nuumnna/ae wwtimmpprogress-eee- 2024.
dataaaddaatttteeeeaaaaa pdf, http:///////////t///////taaa/ea/
Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-We Kei Chang. and Aditya Grover. Lavida: large diffusion language model for multimodal understanding. ArXiv preprint, abs/2505. 16839, 2025. https:: //axxiv.. org/abs/ / 2505. 16839.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion-In improves controllable text generation. ArXiv, abs/2205. 14222205.14217,, 2022.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi thi-Quan Luo. Remax: simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint urXiv:2310. 10505, 2023.
Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang. and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprini 2006v22500.06295, 2025a.
Zichen Liu, Changyu Chen, Wenjun Li. Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint urXiv:2503.2078. 2025b.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024.
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models, 2025. https: / /tt//aivvvv///////////e/////aavv org/ abs 2505. 15781.
AI Meta. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. https://ai. meta.
2025.
Jinjie Ni and the team. Diffusion language models are super data learners. httpps//finjjie
ni not i on s weteeeeiiiieeee fus wos. Moodel 1- -w-ar----------------- -apper-Datta-eearnee s 233988f 2.39d8f03aB0 (0a.a800036600000000 96e4 992804999900111 9ac, 2025. Notion Blog.
Shen Nie, Fengqi Zhu, Chao Du Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li. Scaling up masked diffusion models text. arXiv preprint arXiv:2410.188 18 2024.
13

---

Published as a conference paper at ICLR 2026
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models, 2025, https: //axxiv. org/abs /tt/aab////// 2502. 09992.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.00 AXX::22006033336, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffusion beats autoregressive in data-constrained settings, 2025, https: //trxvv..vvv//////////eee//wav org/ ab2/250..1 5857.
Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. http: http://paaaeeee nips cc/ htspppper__iill files htg.a:/27......aoao.oaaoooaoaaa. Aabeecttdd-bssttt-oooooe
http///22//haaaaeeeeeeeeee rence. html.
Subham Sekhar Sahoo, Zhihan Yang. Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models, 2025.
attps:///arkh wtw.aaa//////ooooo.//t....t..ata abs 250./5566..0 01928.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint S.Kiv:2402.033000 2024.
Jiaxin Shi, Kehang Han. Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. http: 9aead
//papers. nips wwwpcc/pppppeeeeeeeeee waappeerrrilreeeeeeeeeeeeeeeeeee wwtpee/eeppppp//22//2222 http///hhhhha3333a3aee Sectceooeeeo---aattttt---ooeeree
html.
Xiaohang Tang, Rares Dolga, Sangwoong Yoon, and Ilija Bogunovic. wdl: Weighted policy optimization for reasoning in diffusion language models, 2025. 2507. 08838.
wrr.iiii.rr....oooooooooooo abs/
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599 2025.
Qwen Team. Qwen2.5: party of foundation models, September 2024. htttps://wwenmm hub. bulb.io/blo//eee qww.2.5/eee2/
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion 11m by enabling kv cache and parallel decoding, 2025. https: /wwwiviivvv/v./////// org/ wwt//osgg//////oo/oo..oooeeee/s/
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. ArXiv preprint, 10s/2505511809, 2025. https: / rxiv. wtwpass/rrg/a/////oeee 250.. 15809.
14

---

Published as a conference paper at ICLR 2026
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang. Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. https: /ttpa/////k/llp... github httphh/w.....o/oo//////////// dream.
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li. Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint 1.Xiv:2505.16933, 2025.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Wenhao Zhang. Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding, and Jingren Zhou. On-policy meets off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting, 2025. https: http///wxvvvv//v org/ abs 25008. 1408.
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. dI: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504. /22/6, 2025.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXlv:2507.18071 2025.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference optimization for large language diffusion models, ArXiv preprint, abs/2505 19223, 2025. https: www.xx..rr/abs///2/ 19223.
15

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS DISCLOSURE
LLMs were used only for minor editing (grammar and phrasing) and to generate speech narration for the supplementary presentation video from author-written script. All research ideas, meth- ods, experiments, analyses, and substantive writing were carried out by the authors without LLM assistance.
B PRELIMINARIES
B.I MASKED DIFFUSION LARGE LANGUAGE MODELS
Masked diffusion LLMs (Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2024; Lou et al., 2024) employ forward diffusion process that progressively corrupts token sequences zo through introduction of mask tokens. This corruption process is parameterized by time E [0. 1]. At any given timestep the resulting sequence Ia contains partial masking, where each token maintains probability og of remaining unmasked. The noise schedule o, exhibits strict monotonic decrease with respect Complete masking occurs / 1, where all tokens in J1 become masked. The training procedure for masked dI.LMs follows forward process through definition (t and a bidirectional unmasking predictor fe with learnable parameters. During each training step, we stochastically sample timestep & E [0, 1) and apply token masking according to the designated forward process. Given these corrupted sequences, the training objective seeks recover the original tokens. The standard optimization criterion employs the negative evidence lower bound (NELBO), which provides an upper bound for the negative log-like liglikelioood (NLL) of the training data. For masked dLLMs,, this NELBO reduces to weighted NLL formulation, with weighting coefficients derived from transformations of a (Sahoo et al., 2024, Equation (10)). For example, LLaDA (Nie et al., 2025) specifies the forward process through az 1 - t, yiclding the following NELBO formulation:
[1 |zd
(0,,) ma Pam-aan zerre (m)
1 mask log fo x²)
(6)
where (xye] denotes the sequence length x, and x² represents the k-th token position. The loss computation is restricted to tokens masked at timestep t.
During prompt conditional generation, the model starts with a sequence where prompt tokens remain unmasked and continuation tokens are initially masked, then progressively unmasks the continuation tokens through ancestral sampling from the reverse process pa(#,, xy)) for timesteps >> where the model fo provides the denoising predictions for masked positions. The reverse process maintains the property that unmasked tokens are carried over unchanged throughout all denoising steps.
B.2 POLICY OPTIMIZATION FOR MASKED DIFFUSION LARGE LANGUAGE MODELS
Dolicy-gradien methods have gained widespread adoption for post-training LLMs (Ouyang et al., 2022; Bai et al., 2022; Li et al., 2023; Ahmadian et al., 2024), Online RL 1---atticullrly Group Relative Policy Optimization (GRPO) has proved effective for improving language models (Shao et al., 2024; Guo et al., 2025; Team et al., 2025). GRPO (Shao et al., 2024) offers a computationally efficient alternative to PPO (Schulman et al., 2017) by using group-based statistics for advantage estimation, avoiding separate value-function training.
The GRPO objective integrates clipping for stability and reverse KL regularization:
(oRro(!)
2-D is the likelihood ratio.
min (p) Al e,11+s) (.Dx [
(7)
Ma AO (:l))
(o (9.0 Fi Teau (
where A
For a query q, GRPO samples G responses {or...., oG) from the behavior policy Te... and assigns a single sequence-leve advantage per response. Following Liu et al. (2025b), the unnormalized group-relative advantage A; r(o:) - ΣS r(o;), where the reward function. This scalar A; is shared by all tokens in o, when forming the tokenwise objective.
16

---

Published as a conference paper at ICLR 2026
Applying Policy Gradient Methods to Diffusion LLMs Applying GRPO to dLLMs is nontriv- ial. The objective in Equation (7) requires (i) token-level probabilities for importance ratios and (ii) sequence-leve probabilities for KL regularization. Autoregressive models provide per-token conditionals via sequential factorization, enabling one-pass sequence scoring by the chain rule:
log ( (o| q)) Σx= AAR(" q, 05k) Accordingly, the reverse-KL KL decomposes
log Tolo q, q,ock) Tree q. 05k)
DKL [ ④) Tror(- [) = Bo
(8)
In contrast, dLLMs do not admit a sequential factorization of T( 1). dLLM's generation invokes the unmasking predictor ʃg across M denoising steps, making тe a composition of M mappings. Exact tokenwise probabilities would require marginalization over denoising trajectories and maintaining (and differentiating through) full denoising trajectories, which is computationally prohibitive. To address this, recent work develops efficient approximations for policy optimization in masked diffusion LLMs. DiffuGRPO (Zhao et al., 2025) employs a mean-field approximation that yields single-pass estimates of both token-level and sequence-level terms, replacing explicit multi-step unrolling with a single-sample Monte Carlo estimate. While this introduces bias relative to the exact diffusion policy, it provides practical framework for GRPO-style optimization dLLMs. In our method, we adopt the mean-field estimators of Zhao et al. (2025) to compute the token-level importance ratios Pf and the reverse-Ke KL term with one forward pass per policy.
C RELATED WORK
C.I DIFFUSION LANGUAGE MODELS
Diffusion language models was first explored through continuous approaches that map discrete text to continuous representations, including learned embeddings, eequencc-to-sequcee
conditioning, and binary bit representations (Chen et al., 2022; Li et al., 2022; Gong et al., 2023). Recently, discrete diffusion language models have been scaled up significantly, with masked diffusion established as specific instance of discrete diffusion (Austin et al., 2021; Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2024; Nie et al., 2024). Notable developments include DiffuLLaMA (Gong al., 2025a) and Dream (Ye et al., 2025), both adapted from pretrained autoregressive LLMs. LLaDA (Nie et al., 2025) represents a breakthrough achieving performance comparable to similarly-sized autoregressive models. These approaches are predominantly based on masked modeling. Unlike these full-attention dLLMs, Block Diffusion (Arriola et al., 2025) introduced a hybrid approach that models sequences block-by-block while applying diffusion within each block, enabling flexible length generation and improved inference efficiency through kv-caching. Recent commercial models like Mereury (Inception Labs et al., 2025) and Gemini Diffusion (DeepMind, 2025) have demonstrated the practical viability of diffusion- code generation, achieving performance comparable to leading autoregressive models while offering significantly faster inference. More recent works have introduced caching and parallel decoding algorithms (Wu et al., 2025; Liu et al., 2025a; Ma et al., 2025; Israel et al., 2025; Sahoo et al., 2025; Hu et al., 2025) that significantly improve inference efficiency for masked diffusion language models. In this work, focus full-attention masked dL..Ms.
masked diffusion LLM trained from scratch using full-attention,
C.2 REINFORCEMENT LEARNING FOR DIFFUSION LANGUAGE MODELS
Applying reinforcement learning to diffusion language models presents unique challenges compared to autoregressive models. The primary obstacle is the intractability of likelihood functions in diffusion models, which necessitates upproximating response likelihoods for policy optimization. This requirement introduces computational overhead and potential bias, particularly when approximation errors occur in policy ratios used for importance sampling. d1 proposed diffu-GRPO (Zhao et al., 2025) which adopts an efficient approximation through mean-field approximation. MMADA (Yang et al., 2025) and diffucoder's 's coupled-GRD strategy in log probabilities estimation to achieve better learning efficiency. LLaDA 1.5 (Zhu et al., 2025) tackles the variance issues in ELBO-based likelihood estimates through preference optimization. Recently, wdl (Tang et al., 2025) addresses these challenges by reformulating policy optimization as weighted likelihood objective that eliminates the need for policy ratios. SDPO (Han et al., 2025)
(Gong et al., 2025b) further improve the masking
17

---

Published as a conference paper at ICLR 2026
decomposes the diffusion trajectory alignment problem into stepwise subproblems that align the posterior at each diffusion step. Our inpainting method can also be applicable to some of the above online RL methods.
Additionally, a closely related work in RL for AR LLMs Prefix-RF7 (Huang et al., 2025), which samples prefixes from demonstrations guide online exploration, though this is limited to left- to-right sequential generation that does not leverage the bidirectional conditioning capabilitiess of diffusion LLMs.
D LENGTH-ALIGNED SFT: SFT TRACE REVISION LENGTH DISTRIBUTION COMPARISON
As illustrated in Figure 6, the original OpenR OpenRI-Man 1-Mtth-20KK dataset exhibits substantial token length diversity, with reasoning traces extending beyond 10,000 tokens while LLaDA's maximum context length is only 4096 tokens. Naively applying SFT on this dataset would result in many truncated sequences, and even for samples within the 4096-token limit, significant distribution mismatch persists across training phases-va we use 256 tokens for RL sampling and 512 tokens for evaluation. Our rewriting using L.LMA-4-Maverio successfully constrains all traces to under 1500 tokens, creating alignment between SFT training, RI. sampling, and evaluation phases. Additionally, while reflective bchavior has been found helpful for LLaDA prior work (Zhao et al., 2025), the excessive repeated reflective patterns in the original dataset are unsuitable for its constrained generation space. The rewriting process eliminates this redundancy while preserving essential reasoning structure.
SFT traces before rewriting
SFT traces after rewriting
0.00015
0.0025
Distribution Density
Distribution Density
0.0020
0.00010
0.0015
0.0010
0.00005
0.0005
0.00000
0.0000
10000
20000
500
1000
1500
Token Count
Token Count
Figure 6: Token Length Distribution of SFT Dataset Before and After Revision. Comparison of token length distributions for the OpenR1 -Math-220K dataset (94k math problems). After revision using
token lengths are constrained to below 1500 tokens, eliminating the extreme range of the original dataset where traces could exceed 20,000 tokens. This addresses the generation length mismatch across SFT training, RL sampling (256 tokens), and evaluation (512 tokens) phases.
18

---

Published as a conference paper at ICLR 2026
E ABLATION: SET ON HINT TRACES THEN APPLY GRPO VS IGPO
In our RL. training setup, we assume access to ground-truth reasoning traces for every query in the training dataset. To investigate whether direct supervised fine-tuning on these traces provides comparable benefits to our elastic inpainting approach, we conduct an ablation study comparing two strategies: (1) applying SFT on the RL dataset's reasoning traces followed by standard GRPO sampling, versus (2) directly applying IGPO with elastic hint injection only when all generated responses are incorrect.
Specifically, we first fine-tune the LLLDD--B-Inntree model on the MetaMath dataset's reasoning traces for 20 epochs, then apply standard GRPO sampling. We compare this against our IGPO approach, which selectively injects partial reasoning hints from the same MetaMath dataset only when zero-advantage scenarios occur (i.e., when all sampled responses yield incorrect rewards).
The results in Figure 7 demonstrate that IGPO consistently outperforms the SFT-first variant. Notably, after SFT on the MetaMath dataset 20 epochs, model's initial performance drops significantly compared to the original LLaDA-8B Instruct baseline. This degradation occurs because the MetaMath dataset contains very concise reasoning traces, many shorter than our 256-token generation length limit. Consequently, the model adopts overly concise reasoning patterns that prove insufficient for the challenging problems in our evaluation benchmarks (such as AMC and Minerva).
While subsequent RL training recover performance some extent evidenced by the rapid improvement in early training steps-it ultimately fails to match the effectiveness of IGPO. This comparison highlights two key advantages of our approach: (1) the effectiveness of applying inpaint- ing guidance selectively only when the model struggles with specific queries, rather than forcing a uniform reasoning style through SFT. and (2) the critical importance of reducing all- all-wrong group occurrences, which successfully recovers gradient signals from otherwise degenerate zero-advantage scenarios.
SFT (metamath) + GRPO vs IGPO
Average Accuracy across Benchmarks
0.35
0.30
0.25
0.20
0.15
IGPO
SFT on metamath (20epochs) + GRPO
200
400
600
800
1000
1200
1400
RL Training Steps
Figure 7: Comparison of SFT-first approach versus direct IGPO application. The SFT-first strategy involves fine-tuning on MetaMath reasoning traces for 20 epochs followed by standard GRPO, while IGPO applies inpainting anpainting-guided guided exploration elastically only during zero-advantage scenarios. IGPO demonstrates superior and more stable performance, avoiding the performance degradation caused by overly concise reasoning patterns learned during SFT on short traces. Results are averaged across four mathematical reasoning benchmarks with standard errors shown as shaded regions.
19

---

Published as a conference paper at ICLR 2026
F EXPERIMENTS HYPERPARAMETERS
Table 2: Training Hyperparameters
Parameter
Value
SFT Training Parameters
Per Device Train Batch Size
4
Hardware Configuration
8x8 H100 GPUs
Gradient Accumulation Steps
8
Learning Rate
x 10
LR Schedule
Warmup-stable-decay
LR Warmup Steps
200
LR Min Value
10-6
LR Decay Period
Final 10% of steps
Number of Epochs
100
RL Sampling Parameters
RL Online Sampling Generation Length L
256
Diffusion Steps
128
Block Length
32
Sampling Temperature
1.2
Generations Per Group G
8
RL Training Parameters
Per Device Train Batch Size
8
Hardware Configuration
8x8 H100 GPUs
Gradient Accumulation Steps Effective Batch Size
1
512
KI. Beta &
0.01
Policy Gradient Inner Iterations per Generation
4
Learning Rate
x 10-7
LR Schedule
Linear decay to 0
LR Warmup Steps
50
LR Decay Period
10 epochs 1440
Training Steps
Clip Ratio Epsilon
0.2
IGPO Specific Parameters
Chunk Size lcyl ~ U[mmin Smax]
U(5 10 2/
Inpainting Ratio 7), Ul [Poo Thigh
replacement fraction À
0.5
Entropy-base based Gradient Filtering for Inpainted Tokens
0.2
20

---

Published as a conference paper at ICLR 2026
F.1 TEMPERATURE SELECTION FOR RL TRAINING
Following the methodology established by Polaris An et al. (2025) for scaling reinforcement learning on advanced reasoning models, conduct systematic analysis determine the optimal sampling temperature for our RL training process. We evaluate our model's performance across different sampling temperatures by analyzing both Pass@5 and Average @ scores on the MATH500 dataset. We also divide three temperature regions: low temperatures ( 0.8) yield high accuracy but reduced diversity generated rollouts, restricting the model's ability to explore diverse reasoning paths; high temperatures (2 1.6) preserve rollout diversity but significantly degrade accuracy due to increased noise in token generation; and the middle Controlled Exploration Zone (0.9-1.5) provides the optimal trade-off between maintaining reasonable accuracy and achieving sufficient diversity for effective RL training. Based on this analysis, we select temperature = 1.2 to balance exploration with sample quality and provide sufficient diversity for RL training.
Pass@5 Accuracy vs. Temperature
Average@3 Accuracy vs. Temperature
0.5
0.97
Diversity Quality
031
0.3
Diversity Quality
Passig Accuracy
Diversity X
Diversity V
Diversity Quality
Diversity
Quality
Quality
Quality
0.30
0.5
60
0.00
0.7
Temperature
Temperature
Figure Performance analysis across different sampling temperatures on MATH500. The plot shows Pass 5 and Average@5 scores, revealing three distinct regions: low temperatures with high accuracy but low diversity, high temperatures with preserved diversity but degraded accuracy, and the middle region offering the desired trade-off. We select T 1.2 for our RL training.
21

---

Published as a conference paper at ICLR 2026
G INPAINTING GENERATION QUALITATIVE EXAMPLES
Blue text: Question Prompt Green text: Injected Hints
Black text: Inpainted Generation
Circle has radius 6 em, How many square centimeters are the area of largest possible inscribed triangle having one side as a diameter circle C?
Problem:
Inpainting Input At Denosing Step 0:
Respond in the followwng format: <reasoning>
</reasonings
<answer>
\ boxedf<Your answer>}
</answer>
Cirole $C$ has radius cm, How many square centimeters are in the area largest passible
triangle having one side dianeter rele 3cs7<lect_idiil (dlp<//strt__heaae saadder_ddl
baass baas stantx/end_hheadeee
< nask /ass/><cmaa > mask| indn_nask]>
Since the dianster is twice the radius, the base of the triangle will have length $2 times
12$
To < -indd__nask ( jadn_maak 12. we want the height to be as large mdm mask|2, . (mdd, >ankl>
of the triangle is the distance from the midpoint of the base to the circumference
the circle.
This is equal the radius the circle, maadmmmakk) |>... ....... <Imch_mask|>>
mdm mimm_mmaa mank/ 12.. mndm_maakip <|mdm_ mack Dre & < mankm
times [ddd_nasx]2.... mdd_mask]>>
Inpainted Generation After Denoising:
<reasoning>
Ne know to find the largest posaible inseribed triangle in cirale $58, the base the triangle should the dianetes of the circle. Since the diameter is twice the radius, the base of the triangle will have length $2 times = 125 cm.
find the largest possible ares large as possible.
tsiangle,
height from midpoint
largest possible height the base to the circumference the circle.
triangle
This is equal the radius the circle, which is Therelore, the ares of the largest possible inscribed triangle is given by the formula
zriangle, which 84% frec(1H(2 times textbase
times \ textheight = frac(1](2) times 12 times 6 3689 aquare centimeters. </reasening>>
<answer>
\ boxed(36)
</answer>
22

---

Published as a conference paper at ICLR 2026
H EVALUATION HYPERPARAMETERS
We evaluate all models with fixed decoding setup per benchmark. Unless otherwise noted, the same prompts and decoding hyperparameters are applied to both LLaDA-8B-Instre and LLaDA-1.5.
Table 3: Evaluation hyperparameters by benchmark.
Benchmark
Temp.
Block Len.
Diffusion Steps
Gen. Len.
Metric
GSM8K
0.0
32
$12
512
pass @
MATH500
32
$12
pass @
AMC
0.5
128
512
512
avg @ 16
Minerva
0.0
128
512
512
pass@1
I PROMPT FOR SFT TRACES REVISION
Prompt for SFT traces revision sent to LLaMA Maverick
Please rewrite the original solution to make more concise and easier to understand without changing the details. Please put the explanation in the solution between <reasoning> and </ </reasort i ng> and put the final numerical answer between <answer> and </answer> in boxed format. Please shorten or rewrite the rewritten solution to a random length between 100 and 1000 words while keeping sufficient details of the reasoning steps. Please do not return anything other than the rewritten solution. Example:
<reasoning>
xxx </reasoning>
<answer>
\boxed(14)
</answer>
Original solution: {generations].
Your response:
23

---

Published as a conference paper at ICLR 2026
J ROBUSTNESS OF IGPO TO NOISY REASONING TRACES
We evaluate IGPO's robustness when the ground-truth reasoning traces used for hint injection are corrupted with simulated realistic noise: we design numeric errors (e.g., "8 15 130"), operator swaps (c.g., + + x), logical word inconsistencies (e.g., "therefore" + "however"), and replacing hesitation tokens (e.g.. "umm", "wait"). We vary the corruption rate p from to 0.5, where each token is independently corrupted with probability p. Examples of trace corruption is shown in Table 4.
Our evaluation results are shown in Figure 9 shows that IGPO remains effective even under introduced noise: although performance gradually degrades with noises, IGPO consistently outperforms GRPO. This robustness arises from IGPO's gating mechanisms: (i) only partial reasoning chunks are injected and the final answers must still be generated by the model, so only srrectness-verifie responses are used; and (ii) only the top 20% most uncertain injected tokens are allowed to contribute to the gradient, limiting exposure to incorrect hints.
We note that extremely corrupted traces violate the intended use of IGPO, whose goal to leverage mostly correct reasoning to guide exploration; in practice, IGPO should not be applied to heavily inaccurate reasoning datasets.
Robustness to Noisy Reasoning Traces
00.3
79.3
CPO 130% Ncby bakrr)
CPO 140% Nekr tskars
GPO (50% Noley Sarms
KPP 10% Mayy lueersl
28.5
28.0
27.3
77.3
a2o
Traiiinn Stops
Figure 9: @ass accuracy on GSMSK with generation length 256 under simulated reasoning-trace noise. A fraction p of inpainting tokens are randomly corrupted.
Type
Example
Original
Let's calculate: x 15 120, then 120 4 30. Therefore, the answer 30.
Number
Let's calculate: 8 x 15 130, then 130 4 32. Therefore, the answer is 32.
Operator
Let's calculate: 8 x 15 120, then 120x4 480. Therefore, the answer is 480.
Logic
Let's calculate: 8 x 15 120, then 120 + 1 = 30. However, the answer is 30.
Mixed
Umm let's calculate: 8 x 15 125, then 125 + 4 31. Wait, therefore the answer is 31.
Table 4: Example reasoning traces corrupted. Red text indicates corrupted tokens.
24

---

Published as a conference paper at ICLR 2026
K THEORETICAL ANALYSIS: GRADIENT RECOVERY AND KL-CONTROLLED INPAINTING
We analyze why IGPO improves policy optimization exactly the regime where vanilla GRPO fails: the all-wrong, zero-advantage casc. This is precisely the event that triggers inpainting in our algorithm. Throughout,
X' denotes a query, O a response sequence, and r(o) E {0, 1 verifiable reward. The policy xg is parameterized by 0.
Our main conclusions are: (1) In all-wrong groups, GRPO has a zero gradient, whereas IGPO restores non non-zero gradient whose magnitude scales as p p), where p is the effective replacement ratio of inpainted correct responses. (2) The same replacement mechanism admits mixture-policy view: replacing fraction of the response distribution with the correct solution o* yields mixture policy that weakly improves the expected reward and strictly improves it whenever the base policy is imperfect. (3) Partial hint injection controls the KL shift the current policy linearly in the mixture weight a, providing a soft trust-region-like stability guarantee. In our implementation, the batch-level replacement hyperparameter À upper-bounds bounds the effective mixture weight a p P X.
K.1 SETUP AND LERO-ADVANTEE DILEMMA
We aim to maximize the expected reward
((O) E, [[To;
[[T; = Bo ( [o)](
(9)
In GRPO, for each query we sample group for fo....., da i.dd from np(- x)) and compute group-normalized advantages
à r()),
A = r(o,) -
(10)
which yields the per-query policy gradient estimator
1 Σ A V Tol; x).
(cRPO(Z) =
(11)
We focus on the all-wrong event
Evrring = (r(())) = Vi (1..... ()).....))
(12)
Zero-gradient dilemma GRPO.
Conditioned on Ewnngg we have ğGnro 0. This follows
immediately since under Ewrong r(o,) for all so G E, r(o.) and A; r(o,) for all i. Substituting into equation 11 yields
@GRPO (x) = 0 V (og xx) O.
(13)
K.2 GRADIENT RECOVERY AND POLICY IMPROVEMENT
We now analyze IGPO under Ewrong. For fixed query æ, the IGPO batch construction proceeds as follows:
Sample G original responses (oi, d with r(o:) =
Generate an additional set (ōi,..., ) via inpainting using hint injection based on the ground- truth trace o*.
Keep only inpainted responses that pass conrectness verification, r(ō,) 1, and replace at most [AG] of the original failures, where E (0, is hyperparameter controlling the maximum replacement fractioo
25

---

Published as a conference paper at ICLR 2026
Let K be the number of inpainted correct responses used for replacement (K [AG]] and define the effective replacement ratio
K p
G [0.
(14)
The augmented group is
S' fo .o........ d [(:r(d)) 1} K, [iir((o)) = 031 G K.
(15)
Let
Tcorreat
(i: r(d) = 1},
Iwrong = (i:r(o)) = 0}.
(16)
The new group-average reward is
G r(,))
K G Pt
F new
(17)
i=1
and the corresponding advantages are
p. i E Ioonects
A
(18)
P. i E Twrung-
The per-query IGPO gradient estimator (omitting clipping and KI terms for clarity) is
G
OIGro (x)
Σ ==1
1, Vo log To(o) z))
(19)
G
Mixture policy interpretation. weight of an idealized mixture policy that puts mass the correct solution. For each query x with ground-truth solution o², define
The effective replacement ratio p = K K/G can be viewed as the
x- x) (1 a) TH x)) d a E [0, 1],
(20)
where dgr is the Dirac distribution at o". In this view, replacing K of the G samples by o* corresponds to sampling approximately from Ta with s p; our implementation enforces p S À via the cap K < [.G].
We next relate this mixture to reward improvement. For reference policy no. define the advantage
A (z, = r(o) (
(21)
our single-step, verifiable-reward setting, the performance-ddifferenee lemma (Kakade and Langford, 2002) simplifies to
J(T') - (=re = Eg Bown' (+1z) for any comparison policy x'. For n. in equation 20 and fixed x,
(z, (z,
(22)
(own To (x. o)] (1 (o (x, o]] + (z,
= (1 a) 0 a(((o') (TT;
a x(T
(23)
since r(o*)= 1 and (A (I, (I, o)] by definition in all-wrong case. Plugging into equation 22 yields
J(ma) ((To) = a E, [(mo;))]
(24)
Lemma (Closed-form expected IGPO gradient). Define the gradient expectations under the correct and wrong distributions separately:
Icorrect (1)) = Eegg Vo log To m;]] Vo log To(o") |x)),
(25)
gwrong (1) Egrr Vo log (o | x²)
(26)
where we note that r(o') 0 foror T- 1x) Then
(I)ro()) [[g p(1 p) (gcormer(z)) Iwrrong (wwnnn(x:))
(27)
26

---

Published as a conference paper at ICLR 2026
Proof. The expectation in (1) EÍgIGP( is taken over all possible sets S foi......0 oc) such that: (1) the set has size G, and (2) exactly K of the G outputs are correct (sampled from 6,-) while the remaining G K are wrong (sampled from no(- z) conditioned on being incorrect).
To compute this expectation, we first observe that for each valid set S', we can consider all G! permutations of its elements. By symmetry, each position E (1..... G} has the same marginal distribution when averaged over all permutations of all valid sets. Specifically, each of drawn from the mixture distribution:
K
of ~ a. with a =
(28)
Therefore, we can decompose the expectation over groups into a sum of G identical expectations, each over the mixture distribution ..
G A V log Told) m)
gTGPO()) Es.
G
i=1
1
Bof (12) [4] V log to [x)]
G
1=1
Bo' (+w)) [A] v log n (o' x)],
(29)
where the second equality follows from the symmetry argument and the third uses the fact that all G terms are identical.
Now, expanding the mixture distribution Ta (1 a)TT + ado with a pt
Eg' Ee' [A4 V log To(o x)]
= (1-p) was (-1z) [A Va log (o(o') x]]
+ +p [A V log To(o' x)].
(30)
For outputs sampled from (conditioned being wrong), we have r(o') 0, so from equation 18, A = p: (1 p) (r01))==0
[ Vo log no (o' x>] = (1 - p) (-p) = = - p) Swnmggg2 (r).
Vo mol [x)]
(31)
For outputs sampled from dor, have r(o') 1, so A! = = p:
@ [A] Vo (o o' x)] = p (1 -))lEe Vo log To Told x)]
p(1 p) (courea( (x).
(32)
Combining both terms:
(TGPO(#) = p(1 p) Icomret (x) - p(1 p) Swrong (x)
= p(1 p) (x) 9woong (1:).
(33)
Theorem 1 (Gradient recovery and policy improvement). Conditioned on Eurong and for 0 < the expected IGPO gradient satisfies
1,
9IGPO (x) = p(11 () =p(1-p)(grrreed Iwrong (z))
(34)
and is non-zero whenever Gcorrea (I) # gwrong (z). Furthermore, for the mixture policy Ta in equa-
tion 20,
J(Ta) - J( = a Ex -=<o;;:))) 2 0,
(35)
with strict inequality whenever there exists a query such that J x)) < 1. The scalar factor p(1 - () governing the gradient magnitude is maximized at p 1/2.
Proof. The expression and non-degeneracy of (TGPo(x) follow directly from Lemma 1. Equa- tion equation 24 implies J[Ta) - .](T)) 2 since 0 S J( < for all x (as r()) E {0, 1)). on a set of positive measure,
there exists least one with J ([TT; æ) < then 1 [[TT:;
so for any a > 0 the improvement is strictly positive. Finally, the quadratic f(p) = p(1 - p) has derivative f'( 1p2p and second derivative f" (m(() - < 0, so f maximized p 1/2. n
27

---

Published as a conference paper at ICLR 2026
Connection to implementation. In our method section, the hint injection ratio 1) controls how many ground-truth chunks are injected during inpainting and thus only affects how candidate inpainted responses {ō:} are generated. The replacement hyperparameter À caps how many conrectness- veriffied inpainted responses enter the RI. group: K < [. so the effective replacement ratio satisfies p K/G X. In the mixture-police view, the idealized mixture weight a is therefore realized in practice as random effective weight a z2 bounded by X.
K.3 KL CONTROL VIA PARTIAL HINT INJECTION
We now quantify the distributional shift induced by inpainting to argue that partial replacement leads to controlled KL change relative to the current policy.
Theorem 2 (KL control via partial hint injection). For the mixture policy Ta in equation 20, the KL divergence the current policy satisfies
DEL (T.( (:1 22) To( x)) DKI. (K, TT(- (2)) -a log (o" zz).
(36)
Proof. The KL divergence DRL(PIIQ) is convex in its first argument. For any distributions Pi, Pa and a E 1].
DEL - P: + aP: Q) s (1 ak + (Pal()))
(37)
Apply this with P1 = (- x), P3 8.. Q = Tp(- x)):
DKL (Ta(: ax) mo x)) s a) DKL (- x)) || x)) aDL To(. [x))
DDDi. (6.. m.( x)),
(38)
since D = For the Dirac distribution,
(ōor |o( x))
Σ 80 (0) log
5 (o) To(o x) TT (o x)
log
1 log To(o" [x),
(39)
which proves equation 36.
Interpretation and link to İ and 17. When Ta (o x) is small (the model is far from the correct solution), the full Dirac update (a 1) induces a large KI. shift and may destabilize training. The bound equation 36 shows that using mixture weight a < 1 scales the KI. divergence linearly with a, implementing soft trust-region-likk constraint: IGPO injects strong supervised-like corrections while keeping the updated policy within controlled KL neighborhood of the current iterate. Empiri- cally, we observe that intermediate values corrrrssoonding to partial hint injection and hence intermediate X piyld the best performance.
28

---

Published as a conference paper at ICLR 2026
L ABLATION: A SAMPLE-MATCHER GRPO BASELINE WITHOUT INPAINTING
To determine whether IGPO's improvements arise from its hint-guided exploration or merely from an increased number of generated trajectories, we compare against GRPO baseline that matches IGPO's total sampling budget.
In GRPO + resample, whenever GRPO group produces only incorrect responses, we resample additional trajectories from the current policy, matching the number of samples that IGPO would have generated. In GRPO resample, all trajectories remain fully on-policy, and no hint tokens are injected.
train/all,oonn paccudd
trainireward
1200 GIED
1270 G8T0 Haancl
PANO
0.6
0.35
0.3
1.5
02
Sargggooal stop
train/gloool $150
200
800 24
200 400
800
200
400
600
800
Figure 10: Training statistics comparing IGPO, standard GRPO, and GRPO with sample-matched resampling. We report the all-wrong rescued rate (the proportion of all-wrong groups in which resampling yields at least one correct response), the overall all- wrong ratio, and the training reward. IGPO achieves the lowest all-wrong ratio and the highest reward.
IGPO vs. GRPO with Oversampplnng
0RF0 wil reserctet resenses
79
335
200
Evaluation curves on GSM8K (w/ generation length 256, greedy decoding), comparing IGPO, GRPO, and GRPO + resample. IGPO achieves higher pass @ 1, showing that its improvements are not explained by increased sampling alone.
Figure 11:
As shown in Figure 10, despite using the same number of sampled trajectories, GRPO + resample consistently exhibits a higher all-wrong ratio and lower training rewards. Simply increasing the sampling budget does not resolve all-wrong cases, as the policy remains unable discover correct solutions on its own. In contrast, IGPO substantially reduces the frequency of all-wrong groups through its hint-guided exploration mechanism. As shown Figure 11, IGPO also outperforms both baselines in evaluation. These results confirm that IGPO's advantages stem from its inpainting- based guidance rather than from increased sampling volume. Partial hint injection provides targeted directional signals that pure resampling cannot replicate, while still preserving predominantly on- policy generation.
29

---

Published as a conference paper at ICLR 2026
SFT trace before rewritten by LLaMA 4 Maverick Problem: Find all primes between and 15.
<think>
Okay, so I need to find all the prime numbers between and 15. Hmm, let's start by remembering what a prime number is. From what I recall, a prime number natural number greater than that has no positive divisors other than 1 and itself. So, numbers like 2, 3, etc., are primes because you can't divide them evenly by any other number besides and the number itself. On the other hand, numbers like 4, 6, 8, and they can be divided by other numbers. For example, 4 can be divided by 2, so it's not prime. Alright, the task is to list all primes between and 15. Let list out all numbers from to 15 first to check them one by one. The numbers are: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15.
not primes because
Now, let's go through each number and see it's prime. Starting with 1. Wait, remember that by definition, prime numbers are greater than 1. So is not considered prime. So can skip 1.
Next up is Let's check if prime. The divisors of are and There are no other divisors, yes, 2 is prime number. Moving on to 3. Similarly, the divisors are and No other numbers divide into 3, is prime.
Next 4. Let's check divisors. divided by is 2, which is an integer, so is not prime. be divided by 2, it's composite.
Then 5. The divisors of are and No other numbers divide into 5, so 5 is prime.
6. Checking divisors. 6 divided by is 3, it's divisible by and 3. Not prime. Divisors are and 7. No other divisors, so 7 is prime.
Divided by 2 gives 4,
is not prime.
9. Divided by 3 gives 3, so 9 is not prime (since times 3 is 9).
10. Divided by 2 is 5, 10 not prime.
11. Let's see, 11. Divided by 2? 11 divided by is 5.5, not an integer. Divided by 3? 11 divided by is approximately 3.666... Not an integer. Divided by 4? 2.75, nope. 5? 2.2, nope.
Since 11 is a prime number because none of these divisions result in integers, 11 is prime.
12. Divided by is 6, so composite.
13. Checking divisors. 13 divided by is 6.5, 13 divided by 3 is about 4.333..., 4? 3,25,
2.6, 6? 26..6... None of these are integers. So 13 prime.
14. Divided by 2 is 7, composite.
15. Divided by 3 is 5, so composite.
Let recap the numbers between 1 and 15 and their primality:
1: Not prime
2: Prime
3: Prime
Not
5: Prime
Not
7: Prime
8: Not
9: Not
10: Not
11: Prime
12: Not
13: Prime
14: Not
15: Not
So compiling the primes from this list: 2, 3, 7, 11, 13.
Wait a second, let me verify each one again just to be sure didn't make mistake. Starting with 2: Definitely prime. It's the only even prime.
3: Prime, yes.
5: Prime, yes. Prime. Correct.
30

---

Published as a conference paper at ICLR 2026
11: Let's check again. 11 divided by 2 whole. 3+3 3*4 12. So no. 5? 5 is 10, 5 is 15. So 11 is prime.
divided by 2 6.5, 3 4.333, is 2.6. So, prime. Yes, those all check out. So primes between and 15 are 2, 3, 5, 11, and 13.
13: Similarly,
But hold on, sometimes people might make mistake with 9. Wait, is 3 times 3, so that's composite, which I had down. 15 is times 5, composite. So all the non-primes are accounted for.
Is there any number might have missed? Let's see. Let's list all numbers again:
2, 3, 5, 7, 11, 13. Are there any others? Well, is excluded. 4,6,8,9,10,12,14.155 primes between and 15.
composite. So no, those six numbers are all
Wait, but the question says "between and 15." Does that include 15? Hmm, the wording is "between and 15," which could be a bit ambiguous. But usually, when someone says "between and 15," they mean including both endpoints. However, since 15 composite, it's excluded either way. So primes are 2,3,5,7,11,.1. Yes, that seems correct. don't 't think made any mistakes here. Let me cross-verify with another method, like the Sieve of Eratosthenes. The Sieve is method where you list all numbers and cross out multiples. Starting with 2, cross out multiples: 1,611,10,12114. Then 3, cross out multiples: 6,9,12,15.
Then 5, cross 7, cross out 14.
10,15.
11 and 13 are primes, and their multiples are beyond 15. So remaining numbers are 2,3,5,7,11,13, Which matches our previous list. Therefore, the primes between and 15 2.3,5.7.11,13. **Final Answer*
The primes between 1 and 15 are 2 3 </think>>
11 and 13
To find all the prime numbers between and 15, we start by recalling that a prime number is natural number greater than I that has positive divisors other than and itself. We then check each number from I to 15:
** **
Not a prime number (by definition).
**2** Prime (divisors: 1, 2).
**3** Prime (divisors: 1, 3).
** Not prime (divisible by 2).
** **5** Prime (divisors: 5).
** Not prime (divisible by 2, 3).
#$ * Prime (divisors: 7).
**g**: Not prime (divisible by 2).
* **g**; Not prime (divisible by 3).
** 10 Not prime (divisible by 2, 5).
** ** Prime (divisors: 1, 11).
12** Not prime (divisible by 2, 3, 4, 6).
** * Prime (divisors: 1, 13).
14 Not prime (divisible by 2, 7).
15 Not prime (divisible by 3. 5).
Using the Sieve of Eratosthenes method to
non-prime numbers confirms the primes
listed above. Therefore, the prime numbers between and 15 are:
2
5
13
31

---

Published as a conference paper at ICLR 2026
SFT trace after rewritten by LLaMA 4 Maverick Problem: Find all primes between 1 and 15.
<reasoning>>
To find all the prime numbers between and 15, we recall that prime number is natural number greater than 1 that has no positive divisors other than and itself. then check each number from to 15 to determine if prime.
**; Not a prime number (by definition).
** Prime (divisors: 1, 2).
****# Prime (divisors: 1, 3).
**
Not prime (divisible by 2).
*8 #*5**
Prime (divisors: 5).
* Not prime (divisible by 2, 3).
* * Prime (divisors: 7).
***** Not prime (divisible by 2).
**g**; Not prime (divisible by 3).
** +0 Not prime (divisible by 2, 5).
** 11 Prime (divisors: 1, 11).
12** Not prime (divisible by 2, 3, 4, 6). ** 13**; Prime (divisors: 1, 13).
14 Not prime (divisible by 2, 7).
*** Not prime (divisible by 5).
The Sieve of Eratosthenes method confirms the primes listed above. </reasoning>>
<answer>
2,3,5, 7,1 13
</answer>
32


Reviews:
[{"review_id": "7Vmhs8ug9y", "reviewer": "Reviewer_NuRc", "summary": "This paper proposes a policy optimization for diffusion-based LLMs that bridges supervised fine-tuning and reinforcement learning. Specifically, it introduces Inpainting Guided Policy Optimization (IGPO), which leverages the unique inpainting capabilities of full-attention dLLMs by strategically injecting partial ground-truth reasoning traces into online policy optimization. With this design as well as some other techniques such as entropy-based filtering, they achieve better sample efficiency than GRPO and achieve SOTA among full-attention masked dLLMs across four mathematical reasoning benchmarks.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper presents the first work to utilize the unique inpainting capabilities of diffusion LLMs\nfor reinforcement learning, which is novel.\n+ The method achieves SOTA results on four benchmarks for full-attention based dLLMs.\n+ The ablation studies demonstrate insights of the proposed method, such as how self-generated inpainted traces provide a better learning signal than ground truth traces.", "weaknesses": "The proposed method is a hybrid between supervised learning (or imitation learning in the general RL field) and online reinforcement learning. The authors should compare or at least discuss other guided exploration methods in the general RL domain (especially in areas like diffusion policy) to give more insights into how this method is particularly good for dLLMs and how it might be or might not be transferable to tuning other diffusion-based models (say, video diffusion models or diffusion policies). Admittedly, the full-attention dLLMs have their unique properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1mzuqONykH", "reviewer": "Reviewer_raEu", "summary": "The paper introduces IGPO, a reinforcement learning framework that leverages the inpainting capability of masked diffusion large language models to overcome exploration inefficiency in RL finetuning. Instead of relying on full solutions, IGPO injects partial ground-truth reasoning hints during generation, guiding exploration toward promising reasoning paths while trying to maintain on-policy learning. Combined with a Length-Aligned Supervised Fine-Tuning stage using concise rewritten reasoning traces and entropy-based gradient filtering for stability, IGPO achieves state-of-the-art performance on GSM8K, Math500, AMC, and Minerva, improving sample efficiency and robustness over baseline methods.", "questions": "1. IGPO assumes access to accurate ground-truth reasoning traces during RL. How robust is the method to imperfect or noisy reasoning annotations? Have the authors considered evaluating IGPO with reasoning traces of varying quality to assess its sensitivity?\n\n2. Would it be possible to include a discussion or comparison between IGPO and more recent approaches addressing the zero-advantage problem, such as [1]?\n\n[1] Le, Thanh-Long V., et al. \"No prompt left behind: Exploiting zero-variance prompts in llm reinforcement learning via entropy-guided advantage shaping.\" arXiv preprint arXiv:2509.21880 (2025).", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and thoughtfully presented, making its core ideas clear and easy to follow.\n\n1. The proposed algorithm elegantly exploits the unique bidirectional and inpainting capabilities of diffusion LLMs to form a guided exploration strategy that resolves the zero-advantage problem in RL finetuning.\n\n2. It demonstrates clear empirical gains, achieving state-of-the-art results across multiple mathematical reasoning benchmarks.", "weaknesses": "1. The proposed method relies heavily on the unique characteristics of diffusion LLMs and is therefore not a general solution to the zero-advantage problem applicable to broader classes of language models.\n\n2. Compared with standard GRPO, IGPO additionally requires the RL dataset to include ground-truth reasoning paths, which further limits its applicability and use scenarios.\n\n3. The use of reasoning hints violates the on-policy requirement of underlying RL algorithms, potentially leading to bias in the optimization objective.\n\n4. Using reasoning hints during the RL process may constrain the model's ability to explore freely, making the final performance dependent on the quality of the provided expert reasoning hints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b34YLOV6FG", "reviewer": "Reviewer_L2eU", "summary": "The paper proposes Inpainting-Guided Policy Optimization (IGPO), a reinforcement learning framework designed to enhance exploration in diffusion-based large language models under sparse reward settings. The key idea is to inject partial ground-truth reasoning traces as inpainting hints into masked regions during RL sampling, allowing the model to receive guided feedback while retaining its own generative reasoning. This strategy effectively mitigates the zero-advantage problem in group-based policy optimization methods such as GRPO, where uniformly incorrect responses yield zero gradients and inefficient learning. Experiments on mathematical reasoning datasets demonstrate the effectiveness of the proposed method.", "questions": "Could the authors provide theoretical analysis to justify why the proposed inpainting mechanism improves policy optimization performance?\n\nHow robust is the proposed trace rewriting method? Have the authors evaluated it against simpler rewriting heuristics or with different base models for inpainting?\n\nFor domains lacking high-quality reasoning traces, how does IGPO’s performance degrade or adapt under less informative supervision?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed use of inpainting for guided exploration in RL for dLLMs is a creative and well-motivated exploitation of architecture-specific inductive bias.\n\nThe approach is validated with thorough quantitative results on four widely recognized mathematical reasoning benchmarks,substantial gains over both prior masked dLLM methods and non-diffusion LLM baselines are reported\nAblation studies are thorough.", "weaknesses": "The formulation of IGPO objective in the sampling procedure(Eq 5)  is difficult to follow. \n\nWhat does \"Advantages $A_{i}$ are computed normally\" mean?\n\nThe theoretical analysis is weak, provides limited justification for the proposed inpainting mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ixn5WrracJ", "reviewer": "Reviewer_LgBR", "summary": "This paper introduces IGPO (inpainting guided policy optimization) for optimizing diffusion language models with RL which uses ground truth correct trajectories in order to learn in questions where all of the generations are incorrect. To do this, the algorithm resamples more trajectories for questions where all generations are wrong, but inserting text from the correct ground truth trajectory to be inpainted around. Followed by a standard GRPO update on it. This provides \"hints\" for the model during generation. The authors show that this increase performance in math reasoning tasks.", "questions": "- how many of these new responses get clipped? I feel like clipping would null many of these since the trajectory would be off policy?\n- I believe that the fair comparison would actually be to GRPO which oversampled on incorrect trajectories. Ie. Do hint tokens actually matter?\n- does `without Inpaint` in Figure 4 mean that one adds oversampled correct trajectories into the batch as well?\n- does SFT on the rewritten trajectories move it closer to on policy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- this paper is well written\n- this paper considers an important problem, both improving overall performance and utilizing ground truth responses.\n- the results of this paper are encouraging for the method.", "weaknesses": "- while it does seem to work, the entropy-based gradient filtering seems like a method in order to deal with the off policy-ness of the hint responses. Being able to deal with this off-policy would be desirable instead (although harder).\n- the improvement is encouraging, but it is unclear whether it comes from increasing generation number or the hints (see questions)\n- My overall feeling is this seems to improve performance slightly, but there is more work to be done to show that this is the *right* way to use gold trajectories. Ie. How does it compare to distillation methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)