DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation
Authors: Shansan Gong
Ruixiang ZHANG
Huangjie Zheng
Jiatao Gu
Navdeep Jaitly
Lingpeng Kong
Yizhe Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: text diffusion model; diffusion large language model; code generation
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduce DiffuCoder 7B, show that higher temperature diversifies both token choices and generation order to aid RL, and propose coupled-GRPO, a diffusion-native RL method that avoids semi-AR and improves EvalPlus by 4.4%.

Abstract: Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models because their denoising models operate over the entire sequence. The global planning and iterative refinement features of dLLMs are particularly useful for code generation. However, current training and inference mechanisms for dLLMs in coding are still under-explored. To demystify the decoding behavior of dLLMs and unlock their potential for coding, we systematically investigate their denoising processes and reinforcement learning (RL) methods. We train a 7B dLLM, DiffuCoder, on 130B tokens of code. Using this model as a testbed, we analyze its decoding behavior, revealing how it differs from that of AR models: (1) dLLMs can decide how causal their generation should be without relying on semi-AR decoding, and (2) increasing the sampling temperature diversifies not only token choices but also their generation order. This diversity creates a rich search space for RL rollouts. For RL training, to reduce the variance of token log-likelihood estimates and maintain training efficiency, we propose coupled-GRPO, a novel sampling scheme that constructs complementary mask noise for completions used in training. In our experiments, coupled-GRPO significantly improves DiffuCoder's performance on code generation benchmarks (+4.4\% on EvalPlus) and reduces reliance on AR bias during decoding. Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework.

Published as a conference paper at ICLR 2026
DIFFUCODER: UNDERSTANDING AND IMPROVING MASKED DIFFUSION MODELS FOR CODE GENERATION
Shansan Gong 12 Ruixiang Zhang', Huangjie Zheng', Jiatao Gu', Navdeep Jaitly"
I Lingpeng Kong"i Yizhe Zhang' Apple 'The University of Hong Kong "Core advising sansa933ed connect. hku. hk; 1pkê hku. wtw.k/iih..hgggaapp.......
Work done during the internship at Apple
ABSTRACT
Diffusion large language models (dLLMs) are compelling alternatives to autoregres- sive (AR) models because their denoising models operate the entire sequence. The global planning and iterative refinement features of dLLMs are particularly useful for code generation. However, current training and inference mechanisms for dLLMs in coding are still under-explored. To demystify the decoding behavior of dLLMS and unlock their potential for coding, we systematically investigate their denoising processes and reinforcement learning (RL) methods. We train 7B dLLM, DiffuCoder, on 130B tokens of code. Using this model testbed, we analyze its decoding behavior, revealing how it differs from that of AR models: (1) dLLMs can decide how causal their generation should be without relying on semi-AR decoding, and (2) increasing the sampling temperature diversifies not only token choices but also their generation order. This diversity creates a rich search space for RL rollouts. For RL training, to reduce the variance of token log- likelihood estimates and maintain training efficiency, we propose coupled-GRPO, novel sampling scheme that constructs complementary mask noise for comple- tions used in training. In our experiments, coupled-GRPO significantly improves DiffuCoder's performance on code generation benchmarks (+4.4% on EvalPlus) and reduces reliance on AR bias during decoding. Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework. https: htt///gittt github. com/ apple/ e/meemmmm----- /mmmmmmmdmmmm ml-diff fucourr
Base Models
HumesFuul
Irotruct 2a
DIPO Le
Ser segurase Baren oroues Iearne strdeo: BEF3 uo Lisnlittr millan Liple
ceaarate thi fun
BCB liand
HE
ths popor o 20 ponpan Fo (TT- (1r
KBFP-
Careest coart aunontreees
HB-4
Hum anbull
Models
Iummnnvvll
Realt ae0ee
carnest
5CB Hard
MgPP
283 100 med
Sectliclle (ue and dim madibuiiile cecl
Generation
ecs
HBPP+
HumanEvs
(start)
(middle)
(mnd)
LiaDA
DifhuCodee
20
40
DiMfusion
Figure I: (a) A real example of DiffuCoder-Insshee 's decoding process with sampling temperature 1.2. (b) Results coding benchmarks. (c) When decoding steps are halved, DiffuCoder-Insteee trained with coupled-GRPO experiences a smaller performance drop, compared to Instruct itself.
INTRODUCTION
Large language models (LLMs) have shown remarkable capabilities across tasks, from dialogue to code generation (Touvron et al., 2023; OpenAI, 2023). While these successes are built predominantly

---

Published as a conference paper at ICLR 2026
on the autoregressive (AR) paradigm, masked diffusion models (MDMs) have recently emerged as compelling alternative (Zheng et al.. 2024; Shi et al., 2024; Sahoo et al., 2024), and are further scaled to diffusion LLMs (dLLMs) like LLaDA (Nie et al., 2024) and Dream (Ye et al., 2025), which achieve performance on par with similarly sized AR L.MSs. Rather than generating left-to-right, MDMs iteratively refine the entire sequence in parallel, which allows for global planning of content (Ye et al., 2024a; Zhang et al., 2023) and also improves flexibility during decoding (Kim et al., 2025).
Intuitively, code generation aligns well with the dLLM paradigm, as writing code often involves non- sequential back and forth refinement (Xie et al., 2025). Recent commercial-scale dLLMs (Inception Labs et al., 2025; DeepMind, 2025) show that a diffusion- based code generator can rival top AR code models. However, it remains unclear how open-source dLLMs perform on coding tasks, as their training and inference mechanisms are not yet fully interpreted. Existing post-training efforts for dI.LMs, such as LLaDA1.55 (Zhu et al., 2025) with DPO (Rafailov et al., 2023) training, and dl (Zhao et al., 2025), MMaDA (Yang et al., 2025) with GRPO (Shao et al., 2024) training, either show marginal gains rely heavily semi-AR decoding (i.e., block decoding with relatively small block size; see Arriola et al., 2025) which deviates from the global planning nature diffusion. To address these limitations, we first gain insight into decoding behaviors of dLLMs and then establish Iitlusionnnativv reinforcement learning (RL) methodology.
Our investigation is grounded in the analysis of DiffuCoder, 7B- MDM specialized for code generation (63), trained on 130B effective tokens (Huang et al., 2024). The model's performance is competitive with that of AR coders, providing strong testbed for understanding the behaviors of dLLMs and for developing diffusion-native post-training approaches.
To leverage the benefits of non-autoregressivennss in dLLMs, it is important to understand how non-autoregressiive the behavior of current dLLMSs actually is. To this end, we introduce local and global autoregreessvv-nnes (AR-ness) metrics ($4.1) measure how closely their generation follows a left-to-right pattern. We show that DiffuCoder can automatically decide how con-autoregressve it needs to be during decoding ($4.)). When the sampling temperature is increased from the default 0.2 to 1.2, DiffuCoder becomes more flexible in its token generation order, freeing itself from strict left-to-right constraints, as Figure ((a) shows. Unlike AR models, which primarily diversify token choices at higher temperatures, dLLMs additionally diversify the position of the generated token. With this increased diversity, DiffuCoder achieves higher pass@ 10 accuracy by changing the sampling temperature from 0.2 to 1.2 in our experiments ($4.3). The gain in pass@ 10 indicates the potential capacity of DiffuCoder, suggesting it can benefit from effective RL training to "elicit out" the most successful rollout samples.
Following this, we tailor GRPO (Shao et al., 2024) for dLLMs. Our design focuses on reducing vari- ance while maintaining efficiency in Monte Carlo estimations of token likelihoods. Specifically, propose coupled-GRPO, which employs a novel soupled-samplingg complementary mask noise the completion sequences generated by the model a temperature of 1.2. Unlike previous approaches (Zhao et al., 2025), our method does not rely on semi-AR decoding and it further improves the instruct-tuned model. After coupled-GRe GRPP training, the model exhibits a stronger non-AR generation pattern, as inferred from Figure (cc).
scheme. In detail, it adds paired
In summary, our contributions are as follows.
We introduce 7B dLLM for code generation, providing a foundation for developing diffusion- native training methods ($3).
We introduce local and global AR-ness metrics to demystify the decoding patterns of dLLMs and track how AR-ness evolves across different training stages ($4.2). Our analysis reveals that higher sampling temperatures encourage more parallel, non-AR generation ($4.3).
We design coupled-GRPO, an RL. algorithm for dL.LMs that avoids semi-AR decoding by using a novel coupled-sampling scheme for efficient and accurate policy gradient estimation ($5). We theoretically prove the variance reduction of coupled-GRPP using antithetic variates.
Experimentally, coupled GRRPO significantly improves DiffuCoder's performance, boosting its EvalPlus score by 4.4% with training on only 21K samples and demonstrating the effectiveness of RL aligned with diffusion principles.
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES AND NOTATIONS
2.1 MASK DIFFUSION MODELS
In difffusion models (Ho al., 2020; Song al., 2021), the forward process q( IT=1 ((<,/=/-1) gradually corrupts data zy ~ (aota(ao) into noisy variables T1.7. The back- ward process models the joint probability (oo.)))) pa (xT) II (o denoising to reconstruct xp. Discrete diffusion models (Hoogeboom et al., 2021; Zheng et al., 2024) define the for- ward process with categorical distribution 21 xx-1) = 2a Q, x1-1), where H, E {0, 1] a one-hot vector with vocabulary size K, and Qt E [0, 11 x is the transition matrix where [Qels represents the probability of transition from state j- For absorbing discrete diffusion (Austin et al., 2021a), Q: = (1 7 + 3, 1m where 1 is an all-one vector of size K and m is the one-hot encoding of a special [MASK] token. The parameters are learned by minimizing the negative log-likelihood of xo through the evidence lower bound (ELBO). For continuous time modeling (Shi al., 2024; Sahoo et al., 2024), the discrete timesteps f [I...TT T are scaled to a mask ratio within [0, 11, yielding the final ELBO at sampled time t as weighted cross-entrops loss (Ou et al., 2024):
N Σ
1 t
Eggee (20)
[-
(m))
log Sa N:N )m
where oag is the indicator function, and for represents the logits for the n-th token given the N-length input sequence. Please refer detailed related work in $6.
2.2 MARKOV DECISION PROCESS
DDPO (Black et al., 2024) and DPPO (Ren et al., 2025) reinterpret the denoising diffusion process as a Markov Decision Process (MDP). An MDP a tuple Menr = (S, A, Po, P. R) with a state space S, an action space A, an initial state distribution Po. transition probabilities P. and a reward function R. The probability of transitioning to state $141 is P( St, (t,aa) after taking an action a (a( se) and receiving reward R(s,, a.). The goal is to maximize the expected return [(TO) = EnD
Σl= L (t) R(st,
at)
by using the policy gradient method (Williams, 1992):
T
VoT(TT) = E,o
Ve log
at);; re(st,at) = ET(T)
(2)
t=0
In masked diffusion model for conditional generation, we set the state 8t (c. 1, ax.) (where is the condition) and the action to a, = -t-1, so that To (ag(())) = po(2;-1100 c). We sample E C and trajectories of xt, The reward is defined as R[so, ao) = r( c) at the final denoising step because fine-grained progressive reward is usually hard to quantify, especially in intermediate diffusion steps. Under this setting, the policy gradient becomes: VoJ = EEE Vo (x c) r r( c)].
2.3 GROUP RELATIVE POLICY OPTIMIZATION
GRPO (Shao et al., 2024) simplifies PPO (Schulman et al., 2017). It samples a group of outputs
{o,)e, from the old policy Tew. under given condition c, estimates the value bascline by averaging
rewards within the group, and defines the relative advantage for each output as A; = r(o:)
a EE r(o;). For token 1 < & < jo,], we denote p Raulo jo.0 as the token-level importance
xalo (t.o
ratio. The GRPO loss applies PPO-style clipping to p/. incorporating a KL-penalty to keep close to a reference policy net. and maximizes the following surrogate objective:
G lodl
(G) Bog
(pt A., clip(o,, e, e, +e)A,) - 33 DKL. (Tol
By estimating the group mean via Monte Carlo estimation, GRPO avoids training separate value function while fitting neatly into the MDP framework. As the diffusion process can be viewed as an MDP, the GRPO loss can be applied to MDMs by combining Eq. (2) and Eq. (3).
3

---

Published as a conference paper at ICLR 2026
Stages
Stage
Stage
Stage
Coupled-GRPO
Condiioo
Cemplation
Mask states
Tokm
D MADOOS
tokero
Descrip-
Adaptation-
Mid-
Instruction
tion
pretraining
training
tuning
DO COCSOS AAA
MANS
High
Coopled cmmpling Hrihh
Triinnng
RefineCade
OpenCrddrr
OpenCoder
Panki azk 06
DO RRSSS
Data
mixture
anneuling
SFT
POPOSSOO
Full mak for 1.0 A- Hdtd
Tradticaal High
Low
Training
Packed
Packed
Padded
GOSOSOO
estimaated probabiiiiis:
Complaation mosk High
Strategy
Unconditional
Unconditional
Conditional
Compleman made f 0. Wy-faratee
(P, P. Po)/2
Low
Figure 2: Pipeline of DiffuCoder training stages and an illustration of the GRPPled-GRPO algorithm. We sample complementary mask matrices for the same batch, so the coupling probability matrices can be merged into one full matrix. Coupled sampling reduces probability estimation variance while maintaining full token coverage, where each token is sampled exactly the same number of times.
3 DIFFUCODER
To build strong base model for RL training, follow common practices in LLM training and train our DiffuCoder model on a large-scale corpus (Lozhkov et al., 2024) with multiple training stages; the pipeline illustrated in Figure 2. We first conduct adaptation pre-training similar to the process followed by Dream (Ye et al., 2025). Mid-training (Wang et al., 2025) connects the pre-training and post-training stages, plays the role of an annealing phase as in OpenCoder (Huang et al., 2024), and has proven effective. This is followed by an instruction tuning stage to enhance the model's capability of following instructions. Finally, for post-training, we employ a novel coupled- method (introduced in $5) to further enhance the model's s pass @1 coding capabilities.
Training We adapt our model from Qwen-2.5- (Hui et al., 2024) as the base model to perform continual pre-training using the adaptation approach from Gong et al. (2025). During this pre-training, we use a 400B-token code pre-training corpus from RefineCode (Huang et al., 2024) and Stackv2 (Lozhkov et al., 2024). We adopt the code-to-text ratio suggested in Qwen- -wen-2.5-Coder and OpenCoder (Huang et al., 2024). We use 16B tokens of annealing code data during mid-training and 436K SFT samples during instruction tuning, both from OpenCoder (Huang et al., 2024). For RL training, we select 21K hard samples from Acecoder-87K (Zeng et al., 2025) with verifiable test cases. We build our post-training method upon the Open-R1' codebase. All experiments are conducted on 8 to 10 nodes, each with 8 H100 GPUs. We observed that training with 700B tokens in Stage led to worse performance than using only 65B tokens on the downstream validation sets, probably because the continual pre-training is sensitive to the data quality. Therefore, we perform early stopping for Stage 1, training on 65B tokens. In Stage 2, we train for epochs, totaling 65B tokens with repeats, as the mid-training data is less noisy. More details are listed in Appx. B.I.
Evaluation Our evaluation environments are built on three code benchmarks and their variants: Hu- manEval (Chen et al., 2021), MBPP (Austin et al., 2021b), EvalPlus (HumanEval+ and MBPP+; Liu et al. 2023), and BigCodeBench (Zhuo et al., 2024) with full and hard subsets completion (C) and instruction (I) query types. These benchmarks, with Python as the coding language, provide a diverse set of coding tasks for assessing code correctness and quality.
Performance We compare our models with AR code LLMs, including (wwe2.55CCder--77 (Hui et al., 2024), OpenCoder-8B (Huang et al., 2024); general dLLMss, such as Dream-7B (Ye al., 2025) and LLaDA-8B (Zhu et al., 2025); and commercial models, such GPT-4o", Mercury", and Gemini Diffusion As shown in Table 1, DiffuCoder, after being continually trained on 130B code tokens (Stages and 2), achieves performance on par with Qwen2.5-Code 5-Cder and OpenCoder. However, all dLLMS show only marginal improvement over their base models after instruction tuning, especially when compared Qwen2.5-Coder+-SFT which achieves large improvements from being instruct-tuned on the same data.
com htwp:////e.eeeeeeeeee/eeettt/o -o---oo-tsteerrdaaaooeee
'nttps://chat. inwpphatttiiiieiiioiii hhttps://deepmindd.gg http:///deeemi....ggggog///eee gemini-de - ffuaion/
4

---

Published as a conference paper at ICLR 2026
Table 1: Benchmark coding capacities of LLMs and dLLMs 7/8B scale. Different shaded colors indicates different generation paradigms (pink for AR, yellow for diffusion). denotes that the results are collected from public reports instead of evaluating by ourselves. We compute EvalPlus as the average of HE+ and MBPP+ and show the absolute score change (+) of each instruct model relative to its base. We bold scores when our model outperforms at least one LLMs.
HumanEval
MBPP
BigCodeBench (C)
Model
EvalPlus
Avg.
Plus
Plus
Full
Hard
Base Models
Qwen2.5- 5-Coder
61.6
51.8
75.9
61.4
56.6
46.1
16.2
52.2
OpenCoder'
66.5
63.4
79.9
70.4
66.9
40.5
9.5
55.0
LLaDA
35.4
30.5
50.1
42.1
36.3
18.9
4.1
30.2
Dream
56.7
50.0
68.7
57.4
53.7
23.6
4.1
43.4
DiffuCoder
67.1
60.4
74.2
60.9
60.6
40.2
12.8
52.6
Instruct Models
Wwen2.5-Cooder-nnsttui
90.2
85.4
83.9
72.0
78.7
50.7
21.6
67.3
Owen2.5-Cooder+SSP
82.9.21.
75.6.238
80.1,42
1.47
70.9.143
46.9.08
16.2-00
61.3...
OpenCoder- OpenCoder-Instructt
83.5+170
78.7+15.3
79.1-08
69.0
73.947.
40.3 02
16,9+74
61.3+63
LLaDA-IIntruu
35.4-00
31.7-12
31.5 18.6
28.6 13.5
30.2 6.1
16.5
2.7
24.4-5M 5.8
Dream-Instruct Instruct
57.9+12
53.743.7
68.3-04
56.1-13
54.9+1.2
10.6 13.0
0.7
41.2-22
DiffuCader-Instruse
72.0.49
65.2448
75.140.
61.941.
63.6x3.
35.7
12.2-06
53.7+11
+ coupled-GRPO
73.2.6.1
68.347.
78.6.44
67.5.66
67.947.
40.4.0.
10.8-20
56.5.39
Commercial Models
GPT 40"
90.2
82.2
82.4
49.9
Mercury"
90.0
77.1
80.4
45.5
Gemini Diffusion"
89.6
76.0
45.4
This improvement gap between AR and dLLMs at the instruction-tuning stage motivates us to explore RL-based post-training methods ($5). Previous RI. approaches for diffusion models (Zhao al., 2025; Yang et al., 2025) rely heavily on semi-AR decoding, which deviates from diffusion's 's global nature. To design RL methods aligned with diffusion's 's non-autoregressive principle, we first analyze the intrinsic decoding behavior of dLLMs and their differences from AR models in $4.
UNDERSTANDING MASK DIFFUSION MODELS BASED ON DIFFUCODER
Current dLLMs such as LLaDA (Nie et al., 2024) and Dream rely on low-confidence remasking decoding strategies (Chang al., 2022), and LLaDA achieves improved performance on certain tasks using semi-AR decoding methods (i.e., block diffusion decoding; see Arriola et al., 2025 and Fathi al., 2025). Another common practice among dLLLMs to set the number of diffusion timesteps equal to the sequence length, effectively resorting to token-by-toker generation to enhance performance. Given this, we introduce local and global autoregressive-nee systematically investigate the decoding order of dLLMs. Specifically, analysis aims to demystify: (1) how dLLMs' decoding patterns differ from those of AR models; (2) how data modality (e.g. code math) influences model behavior, and (3) how AR Reness evolves across different training stages.
(AR-ness) metrics to
4.1 AUTOREGRESSIVE-NESS IN GENERATION
In standard AR decoding, the model generates tokens strict left-to-right order, ensuring strong sequential coherence. However, diffusion-based decoding may choose to recover [MASK] out of order. Therefore, introduce two metries quantify how the unmasking schedule of diffusion model resembles AR pattern, including (i) "next token" pattern and (ii) "left first" pattern.
Local: Consecutive Next- Token Prediction Local AR-ness@ & is computed by the ratio pre- dicted sequence matching the pattern of next token prediction within range k. If all tokens in k-length
5

---

Published as a conference paper at ICLR 2026
0.6
LLaDA Local ARnees nReness
Code Math
1.0
Adapted DLM Local ARness
Code
Probs
Max probs of each token
0.4
AR MK-nessBB
Math
50
100
150
200
250
0.2
0.5
Token index (LLADA)
Neg Neg. Entropy
neg entropy
0.0
0.0
6
0.6
LlaDA Global AR. ness
Adapted DLM Global AR noss
100
200
300
400
500
ge
AV nessess
Code
Token index (Dreum)
0.4
Math
Local AecalAReee ARenees
Global AR-ness
0.5
0.2
Code
[M] love dogs and (M]
[M] love doge and [M]
Math
0.0
[X] love doga cats
Love doga [M]
(unmask next token)
(unmask first mask)
Figure 3: Left: Local and global AR-ness across different models and data modalities. Adapted dLLM refers Dream for the math task and DiffuCoder (Stage I trained with 65B tokens) for the code task. Right: (a) Confidence score for each position in the dLLM's ' first forward decoding step (Appx. C.1). (b) Local AR-ness @ the fraction of decoding steps where the newly unmasked token, together with the k immediately preceding predicted tokens, forms a strictly increasing consecutive sequence. Global AR-ness@A k: the fraction of decoding steps where the model chooses to unmask one of the carliest k positions among all remaining masked tokens.
Stage 1 Le nes
Stage 2- Local Afteess
Stage 3- Lccal Alless
Post Truining Lecal Altezes
0.0
Stape31len|
intruct Model
Stags177:05)
Stage315esl
Sape1 Afnees
Stog: Gbbb)A8.ncc
Stog: Gl.ba
Post Vrinng Olibal
Stags
Skage2lkksl
Slage31bus)
GAPO
Figure 4: AR-ness drifts on different training stages. Left: adaptation pre-training stage and mid- training stage. Right: instruction tuning and RL post-training stage.
span are immediate successors of the previously generated token, we count this span as casual. Local AR-ness decays as i grows, it is harder maintain longer consecutive spans.
Global: Earliest Mask Selection In step 1, if the predicted token lies in the first & masked positions, the global AR-ness scored. Global AR-ness@! k is the averaged ratio for each I, and measures the tendency to always unmask the carliest remaining token, capturing a left-to-right filling strategy. This ratio grows with k, since the criterion becomes casier to satisfy as more early positions are allowed. For the two metries, the higher value indicates that the generation is more autoregressive. Detailed formulations are listed in Appx. A.2.
4.2 DECODING ANALYSIS
We conduct AR-ness comparisons during conditional generation between: (I) different dLLMs,
including LLaDA trained from scratch and Dream or DiffuCoder adapted from AR LLMs; (2) different data modalities, including math and code; and (3) different training stages of DiffuCoder.
All inference settings based on the con-cofiieecce remasking strategy (Chang et al., 2022) with the same sequence length and diffusion timesteps (512). Math is evaluated on GSM8K using 8-shots (Cobbe et al., 2021), and code is evaluated using zero-shot HumanEval.
How do dLLMs decode differently from AR models? For AR decoding, both local and global R-nnsss are identically equal to 1 (i.e., 100% AR). In contrast, as illustrated in Figure 3, dLLMs do not always decode in purely AR manner. A significant fraction of tokens in dLLM decoding recovered from neither the leftmost masked token nor the next token. This observation indicates that dI.LMs adopt a more flexible decoding order compared regular AR models. Nevertheless, both

---

Published as a conference paper at ICLR 2026
local and global AR-ness closer to 1 than 0, demonstrating that text data inherently exhibit some AR structure, which diffusion based LMs, regardless of whether they are trained from scratch or adapted from AR models, naturally capture. For further validation, examine the confidence score of each recovered token in Figure 3(a) and reveal this phenomenon as the entropy sink (A (Appx. C.1), I), where the confidence distribution displays characteristic "I." "I-"-shaped pattern. We hypothesize that the entropy sink arises because the intrinsic nature of text biases the model toward tokens that lic immediately to the right of the given prefix: those positions receive stronger positional signals and closer context, leading the model to assign them disproportionately high confidence.
Entropy sink contributes to AR-ness of dLLMs. Empirically, adapted dLLMs tend to exhibit stronger AR-ness than those trained from scratch. This is because they inherit the left-to-right token dependencies from the original AR training. Lower AR-ness opens up additional opportunities for parallel generation by breaking this dependency (Appx. C.4). Higher AR-ness can also be beneficial; for example, L.LaDA often needs to resort to semi-AR (block-wise decoding; Arriola et al., 2025) generation to achieve higher overall performance. In that setting, the block decoder explicitly reintroduces causal bias into the generation process. In DiffuCoder, we argue that the model can decide how causal it is during generation by itself.
How do different data modalities affect the decoding paradigm? According to Figure 3, although math and code decoding exhibit different degrees of local AR-ness, consistent finding is that code generation has a lower mean and higher variance in global AR-ness. This indicates that when generating code, the model tends to produce later tokens first, leaving some early masked tokens un-recovered until much later (Appx. C.3). The reason might be that mathematical text is essentially sequential and usually requires left-to-right computation, whereas code has an intrinsic structure. Consequently, the model often plans token generation more globally, much like programmer jumping back and forth through code to refine a code implementation.
How does nR-ness change at different training stages? In Figure 4 (Stage 1), after training with 65B tokens, we already observe a relatively low AR-ness. However, when we extend the training to 700B tokens, AR-ness increases while overall performance drops (see the table in Appx. B. We suspect that the quality of the pre- pre-training data limits performance. Consequently, we choose the Stage 65B model as the starting point for Stage 2. During mid-training (Stage 2) and instruction tuning (Stage 3), on the first epoch of high-quality data, the model learns a high causal bias. As it sees more tokens, however, task performance improves (Appx. B.1), while the measured AR-ness starts to decline. This pattern implies that after the first epoch, dLLMs begin to capture dependencies beyond a pure AR order. After GRPO training, the model's global AR-ness also decreases and meanwhile shows less of a performance drop when decoding in half as many steps (Figure (cc); Appx. C.4).
Humaneval Humanevale
HBPP & MBPP
To Temp 2.2 Te Temp-).8 1.0
76-
Temp- 2.0
Temp-0.0 0.0
Tomp-0.2
Temp-00.2 2.2
T.mp=
Tempan.55
cook
Tem
Temp-0077
Trmp--0.2
Thmp-10
20
Tamp-12 122
Temp=10
Temp-1.2
Temp-1.0
Tomp=15
Temp-1.5
Figure 5: Affects of different sampling temperatures. Left: For base model and instruct model, changing temperature affects the AR-ness on HumanEval. Right: pass @k k curves are different for different temperatures, where triangles refer to score for plus version of each task.
4.3 GENERATION DIVERSITY
Post-training studies on AR LLMs (Yue et al., 2025) show that an RL model's reasoning paths are bounded by the base model's pass@k sampling capabilities. Therefore, we ex- amine generation diversity with pass@k accuracy in dLLMs. As Figure 5 (right) and Fig- ure 6 illustrate, for both the base and instruct versions of DiffuCoder, a low temperature yields high pass @ I but little growth in pass@k, indicating that the samples lack diversity.
7

---

Published as a conference paper at ICLR 2026
1.0 to 1.2), pass k rises significantly, revealing a latent capability in the model. In many RL settings (Bercovich et al., 2025; Liu et al., 2025a), the model must be able to sample diverse responses during rollouts before RI. can reinforce pass@) accuracy. The promising pass @ k curves of DiffuCoder indicate substantial room for improvement through RL, motivating the design of our coupled-GRPP algorithm ($5). Moreover, a higher temperature also sub- stantially lowers AR-ness as shown in Figure 5 (left) and Figure (a), meaning the model generates tokens a more random order, which operates differently from AR models. In AR models, temperature only affects token selection, while dLLMs it influences both token selection and their generated order. A visualization of the decoding process from real samples is provided in Appx. C.3.
By increasing the temperature to suitable range (e.g.,
$7.8
70.3
82.5 85.455.0
Pass Rate (%)
Trmp=62
Tamp-1.0
(aysde101
20-
1emp-1.2 (Pyes
Bost
Instruct HumanEval
Instruct HumanEval
HumsnEval
HumanEval+
Figure 6: Pass@ k scores for different models and temperatures.
COUPLED-GRPO
Reinforcement learning (RL) and GRPO (Shao et al., 2024) have proven critical for enhancing AR models (Bercovich et al., 2025; Shao et al., 2025), but their application dLLMs is less explored. As discussed in $2.3, formulating the mask diffusion process for a policy optimization approach akin to PPO (Schulman et al., 2017). To facilitate integration with GRPO (Shao et al., 2024), the approximation of token probabilities within diffusion models is necessary. Current masked diffusion models rely on Monte Carlo sampling (Zhu et al., 2025; Shi et al., 2024) for bogaprobabylity estimation. Specifically, the negative log-likelihood (NLL) is bounded by the ELBO, iee., P; = Beo.....TT::aaggl æ. lan)) Le (xx)), where L, is the cross-entropy loss introduced in Eq. (1), However, Monte Carlo sampling introduces significant overhead during the training of GRPO, as highlighted by dl (Zhao et al., 2025).
Markov Decision Process allows
Baseline methods To overcome this, dI chooses to mask all completion tokens and perform a single forward pass to compute each token's 's probability, which is equivalent to sampling once at diffusion step f = T: we call this sequence of log- bog-probubilities P,, with each element being log no(o (c, of = m). Thus, GRPO's update in Eq. (3) uses the ratio o =e(o"|e.0"= ==))) dl also
Tuw lo (o,oo c,))=m))
randomly masks p- 15% of the condition tokens increase sampling diversity, which, in practice, makes the completion-toke
probability estimates unreliable. In our code experiments, masking condition tokens does not yield a stable reward improvement (Figure 7), probably because code tasks demand higher token-level generation accuracy than math tasks. As result, we revert to the completion full mask version (p- 0%) and use Pi=T as our baseline. Even so, this baseline is biased: as shown in our entropy sink analysis ($4.2), high-entropy tokens tend to lie the left side, so RL training still ends up updating early tokens more aggressively.
CRRP In the L, computation, only the loss for positions involving masked tokens is counted, which introduces inefficiency and variance when sampling times are limited. To improve probability estimation while still covering every token, we introduce
coupled-sampling scheme (Figure 2). Concretely, we pick timestep pairs (t. f) with t+ T, then sample two complementary completion masks: each mask hides part of the tokens, and together they cover all completion tokens. In other words, every token is unmasked in exactly one of the two forward passes. This design guarantees that (1) each token's log- nog-probability is computed at least once (giving each token non-zero learning signal) and (2) these log- proobabbbiylity estimations token is evaluated under a realistic partial-maski context rather than always being masked, and we have 2 additional samples compared to the baseline. Combining Eqs. (1), (2), and (3), we have:
more accurate because every
G logl
To To |c, ofte)) Te (oflc, ofee))
To(o lc, of.ee) Team (ofle, ofter)
(o) -
min
Ai, clip(
1- 1- +))))) 3DKL]
with log To lo, ofte))= ofter))) x+1
Σ T
(Z((z)) + Cc(x;)) (xer)
Sarm + de 8m 1.
(4)
8

---

Published as a conference paper at ICLR 2026
In practice, we choose À = 1. For a fair comparison, introduce de-coupled baseline with the same number of samples but without the complementary constraint. For advantage score computation, we also consider leave-one-out (LOO) strategies (Ahmadian et al., 2024; Kool et al., 2019) to determine the baseline score: A; r(o.) r(o E r(o,), which creates an unbiased estimate. We show that our coupled-sampling scheme can viewed an application of the Antithetic Variates (Hammersley & Mauldon, 1956) variance reduction technique in Appx. A.3, where we also list detailed designs for verified rewards, including a code format reward and the execution pass rate over test cases as a correctness reward.
2.0
2.0
2.00
1.75
Roward
Reward
1.5
Reward
1.50
1.0-
1.25
(g-0.15)
1.0
Coupled GRPO
Full mask (p=0)
Coupled de-Coudled
1,00
Coupled temp-1.
Coupled temp=
0.0
0.2
Training Epoch
0.4
0.6
0.8
1.0
0.2
Training Epoch
0.75
0.0
Training Epech
0.4
0.6
Figure 7: Reward curves during GRPO training. Left: Comparison between coupled GRPO and dI baselines. Middle: Decoupled GRPO uses the same number of samplings but with randomly sampled mask noise. Right: Coupled-GRPO is sensitive to the rollout temperature.
HumanEval Plus
MBPP Plus
BigCodeBench (C)
BigCodeBench (I) Hard
Model
Full
Hard
Full
Owenn.5-Cooler-SF
82.9
75.6
66.1
46.9
16.2
39.5
14.9
+ GRPO
80.5-24
75.0-a6
84.4.43
72.8
49.7.28
16,2
40.0.os
10.2-47
DiffiCCoder-Instruct
72.0
65.2
75.1
61.9
35.7
12.2
34.0
8.8
+ coupled GRPO
73.2
68.3.3.
78.6.3.5
67.5.5.
40.4:47
10.8-1.4
37.5.35
10.8
+ coupled GRPO (LOO)
70.7 13
62.2 3.0
79.6-1.
68.5+6.6
41.2+5.
13.5-13
37.6-36
12,8+40
w/ full mask completion
66.5.5.
59.1-6.1
77.0-19
65.1
38.0a2a
8.8-14
35.6-166
13.5447
w/ decoupled sampling
68.9.11
62.8-24
78.3432
66.4.4.
40.4.4.
10.8-1.
36.5+2.
10.8+20
Table 2: Evaluation results for GRPO post-training across multiple benchmarks and models. We report the best results from the sampling temperature set f0.2, 0.3, 0.4).
Experiment Results Table 2, together with Figure 7, demonstrates the effectiveness of coupled- GRPO training. In contrast, the bascline variants: d1, full-mask completion, and decoupled sampling, exhibit unstable reward learning. The rollout sampling temperature is also critical: as shown in Figure 6, SefffCoddr-Innstruue attains higher @ 10 at temperature 1.2 than at 1.0, mirroring the trend observed during coupled-GRPP training. Notably, RL fine-tuning shifts the optimal sampling temperature from 0.2 to larger value, such as 0.3 or 0.4, during evaluation, indicating that training sharpens the per-token distribution. This finding aligns with recent results for AR LLMs (Cui et al., 2025; Liu et al., 2025a; Agarwal al., 2025), suggesting that the approaches proposed in these works may also be generalizable to dLLMs. Finally, at the new optimal temperature, global decoding AR-ness decreases, as shown Figure 4 (right). more interesting finding is that, as shown in Figure (c)), when we use 0.5x fewer decoding steps (equivalent to a 2x generation speedup), training with coupled-GRPO results in a smaller performance drop compared to the model before training, suggesting that AR-ness is reduced and parallelism is increased (Wu et al., 2025). Detailed discussions are provided in Appx. C.4. The comparison for different is in Appx. C.5
RELATED WORK
Text Diffusion Models Early explorations of text diffusion models were based on a continuous
space (Li et al., 2022; Gong al., 2023b; Chen et al., 2023). Subsequently, discrete diffusion mod- els (Hoogeboom et al., 2021; Austin et al., 2021a) directly introduced discrete noise to accommodate the discrete nature of text, demonstrating significant potential (Zheng et al., 2024; Lou al., 2024)
9

---

Published as a conference paper at ICLR 2026
and were further developed into mask diffusion models (Shi et al., 2024; Ou al., 2024; Sahoo et al., 2024). Recent work has explored scaling these models significantly, with DiffuLLaMA (Gong et al., 2025) being adapted from pretrained AR LLMs, and LLaDA (Nie et al., 2024) and Dream (Ye et al., 2025) being the first open-source diffusion L.Mss to achieve performance comparable to AR LLMs. Block diffusion (Arriola et al., 2025) proposes hybrid approach that applies diffusion within each block (Han et al., 2023), serving as a midpoint between autoregressive and diffusion models. Multimodal models such as Va (Li et al., 2025), MMaDA (Yang et al., 2025), and Dimple (Yu et al., 2025) combine text diffusion models with vision models. Liu et al. (2025b); Hu et al. (2025); Ma et al. (2025); Wu et al. (2025); Sahoo et (2025) introduce caching and parallel decoding algorithms for dLLMs, significantly improving inference efficiency.
Code Generation Code generation is a crucial domain for LLMs (Roziere et al., 2023; Sun et al., 2024), exemplified by state-of-the-ate art open-source models like Qwen-2.5-Code and OpenCoder (Huang et al., 2024), with wide applications in areas such as coding assistants and agents (Xu et al., 2024). CodeFusion (Singh et al., 2023) was the first to combine diffusion models with code generation, but it was limited to small-scale models and simple tasks. Recent commercial-scnn dLLMs, such as Mercury (Inception Labs et al., 2025) and Gemini (DeepMindd, 2025), have demonstrated that diffusion-bars to leading autoregressive code models while offering significantly faster generation speeds.
(Hui et al., 2024)
code generators can achieve performance comparable
Reinforcement Learning Reinforcement learning with verifiable reward (RLVR) using GRPO (Shao et al., 2024; OpenR1, 2025; Guo et al., 2025; Bercovich et al., 2025) is highly effective in enhancing a language model's 's math reasoning (Shao et al., 2025) and code generation abilities (Xie al., 2025). Wang et al. (2025) show the importance of mid-training during RL scaling. Combining RL and diffusion models, VRPO (Zhu et al., 2025) introduces the efficient sampling algorithm from DPO (Rafailov al., 2023) for dLLMs. d1 (Zhao al., 2025) and MMaDA (Yang et al., 2025) optimize math reasoning in dLLMs using GRPO, but they rely heavily on block diffusion decoding during rollout and evaluation. LLadDou (Huang et al., 2025) trains additional module to predict the rank score of tokens. For small text diffusion models, Zhang et al. (2025) propose the target concrete score matching framework, and Zekri & Boullé (2025) introduces score entropy policy optimization (SEPO). Earlier, DDPO (Black et al., 2024) and DPPO (Ren et al., 2025) formulated the diffusion process as a Markov Decision Process and performed policy optimization for continuous diffusion models.
CONCLUSION
In this work, we present DiffuCoder, a 7B-scale open-source diffusion model for code with strong performance, and its complete training recipe. We also present comprehensive analysis of dLLMs for code generation. Our investigation into their decoding patterns reveals fundamental differences from AR models; notably, sampling temperature affects not only token selection but also the generation order, creating rich sample diversity for optimization. Capitalizing on this, we introduce coupled- GRPO, reinforcement learning algorithm that respects the non-autoregressive nature of dLLMs. By using a novel GRPO significantly boosts DiffuCoder's performance, demonstrating the effectiveness of RI. methods aligned with diffusion principles. Our work provides the community with a deeper understanding of dLLLMs and lays strong foundation for future explorations of dLLMs in complex reasoning and generation tasks.
strategy. our method provides a more accurate likelihood estimation.
Coupled- CoPPled-GRPO
ETHICS STATEMENT
This work develops a diffusion large language model for code generation. Such models could be misused to produce harmful insecure code. We focus on methods and train only on public datasets; to our knowledge, they do not include personal or sensitive data. We follow standard filtering and manually review representative generations used in the paper to avoid toxic misleading content.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng. The unreasonable effec- tiveness of entropy minimization 11m reasoning. ArXiv preprint, (ob//2055.1114, 2025. URL
https:/ atw.v..ivvvvvv////i//.../.a.aa org/ abs 2os/25005 2505. 15134..
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzich Fadace. Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back basics: Revisiting reinforce style optimization for learning from human feedback in llms. ArXiv preprint, 12//240..14740, 2024. URL https:
//arxiv. wtg. abs /4022 14740.
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang. Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. The Thirteenth International Conference on Learning Representations, 2025.
Jacob Austin, Daniel D. Johnson, Jonathan Ho. Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. In Marc' Aurelio Ran- zato, Alina Beygelzimer. Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, 17981-17993, 2021a. URL https: pttpp//ecccen......ee/pepp nww.ips.r.ppp.ee.. wttppppp..pppppp22....... Abstract
70t. aet.ee99dd-bbbratttt.eaaa html.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2108.0777 07733, 2021b. URL https: /ttp//axvvvv../////e//e/ae org/ abs 2108.
07732.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El- Yamiv,,, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, al. Llama-nemotron: Efficient reasoning models. ArXiv preprint, 00/225 2025. URL https: /
abs /2505 9494..
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:://peeneeiiee net "oerumiiddYYWihGrr..
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 1 11305-11315. IERE, 2022, doi: 10. 2022.
/109/CVPP52288.2222..1111 01103.
URL https: //to///ooooo//ooo wtw.doo..o///oooooooooooooooeo
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, 0bs/2210.00377 2021. URL https: /aaxxiv. org/abs /21071112 03374.
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton. Analog bits: Generating discrete data using diffusion models with coffiiondiio.ing. In The Eleventh International Conference on Learning Representutions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net 2023. URL.
net met/pdffidee d
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, 100//110..1468. 2021. URL htttps://arxiv.oot abs/ fass/210... 14168.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang. Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. ArXiv preprint, abs/2505.22617, 2025. URL https: //arxiv. org//bs/ 2256b2261..
22617.
11

---

Published as a conference paper at ICLR 2026
DeepMind. Gemini diffusion. gemini-difffsion/
2025.
URL https://deepminde wtwp.//./o.ooooo..oooo/oo//
Nima Fathi, Torsten Scholak, and Pierre-André Noël. Unifying autoregressive and diffusion- basedd
sequence generation. arXiv preprint arXiv:2504.064416 2025.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. DiffuSeq-v2 v2: Bridging
discrete and continuous text spaces for accelerated Seq2Seq diffusion models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 9868-9875, Singapore, 2023a. Association for Computational Linguistics. URL https: http:/aaaaaaatto......... wrg/20233
doi: 10.18653/v 1inn.innii........nnn.ni pandings-emnll..oo findings-emnip 660.
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b.
URL httpt:htttppseeeeeeeeee http:////pppeeeeeeeee/ee//e/eeee net /ptp:/ddd-jjj--11----
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. When attention sink emerges in language models: An empirical view. ArXiv preprint, Abs/2410.10781, 2024. URL https:// perxiivorrg abs/bb/// /211//2110022 10781.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. ArXiv preprint, abs/2501.12948,, 2025. URL https: //axxvv. org/abs
John Michael Hammersley and James G Mauldon. General principles of antithetic variates. In Mathematical proceedings of the Cambridge philosophical society, volume 52, pp. 476-481. Cambridge University Press, 1956.
John Michael Hammersley and Keith William Morton. A new monte carlo technique: antithetic variates. In Mathematical proceedings of the Cambridge philosophical society, volume 52, pp. 449-475. Cambridge University Press, 1956.
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov. SSD-LM: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11575-11596, Toronto, Canada, 2023. Association for Computational Linguistics. doi: https: /acl anthologyy org/ 2023 a.23.accl110 acl- 647.
URL
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http:: http:t/ppp///poooeedndge..e meeeoeeeeednnnnnnneeeeeeeeeee.e wttccc/ppaaa htcp/appr/2222///aa// 4c5bcfec85 Sabpccfcess440000TTTaaaa01111111
html.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Marc' Aurelio Ran- zato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 12454-1244 2021. URL https: http:://ea////ppeppeeeeeeeeeeeee neurips wwtppppeppppccccccccccccc /221/ 202h/hash/ 67d96d45 (eeedd588bbeee f21 322 Abst html.
12

---

Published as a conference paper at ICLR 2026
Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S. Abdelfattah, Jae-sun Seo, Zhiru Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided diffusion. ArXiv preprint, abs/2505.21467, 2025. URL https:::: /atp/vxivv////////////////////aa org/ aas///b///// 2505. 21467.
Siming Huang. Tianhao Cheng. Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, Yang. JH Liu, Chenchen Zhang. Linzheng Chai, et al. Opencoder: The open cookbook for top-tier code large language models. ArXiv preprint, abs/241 1. 1.04905, 2024. URL htwps: /taptpss//arxx...../ org/ abs/2411. 04905.
Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, and Guo-Jun Qi. Reinforcing the diffusion chain of lateral thought with diffusion language models. ArXiv preprint, abs/2505. 10446, 2025. URL https: arxiv. org/ abs abs/2505 10446.
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. Qwen2. 5-coder technical report. ArXiv preprint, abs/2409. 12186, 2024. URL https: arxiv. org attpa//s///.....//// 12186.
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov. Mercury: Ultra-fast language models based on diffusion. 2025. URL
https:/ inceptionaabs
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. Forty-second International Conference on Machine Learning, 2025. URL https: http:///wppneeeeeee.e.eeee www.teeetfffitttteoooooo. DjJmre5IkP.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! DeepRLStructPred Workshop ICLR, 2019.
Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover. Lavida: A large diffusion language model for multimodal understanding. ArXiv preprint, abs/2505.16839, 2025. URL https:
//arxiv. org/ abs 25005.55... .....666666 16839.
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion-Im improves controllable text generation. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: http://papers. /tppp/apprrssss///e nips . cc htcepaper__iiie filerfiiel//ele httllee/pppe//222/ /hash/ Abst Conference html..
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 nips ca wwtpppppr.filliecceee files httpp/aappp///aaeeeeeeeeeaeeeaae 2 2023/ 3/hash/ Abstract Conference html.
16, 2023, 2023. http:/ att.aaaaaaaaaaae--eeeaeatttttttt
4de9 4
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. ArXiv preprint, 0aw.22555.24666 2025a. URL httts::///aaatt
org/ abs /2005 24864.
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjic Chen, Chang Zou, Qingyan Wei, Shaobo Wang, and Linfeng Zhang. dilm-cache: Accelerating diffusion large language models with adaptive caching, 2025b. URL https: / thub. com/ m oma MLLM- cache.
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. pecreason-nemotree 1.1: Advancing math and code reasoning through sft and rl synergy. ArXiv preprint, abs/2506. 01122506.12844. 13284, 2025c. URL ht / axvvi org abs /2006. 13284.
13

---

Published as a conference paper at ICLR 2026
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. In International Conference on Machine Learning. ICML, 2024.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. ArXiv preprint, 19s//2402.1917, 2024. URL https: / /ttxiaiivvco///// org/ abs/
2402. 19.
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. ArXiv preprint, abs/2505.15781, 2025. URL 2505. 15781.
org / abs/
Shen Nie, Fenggi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng. Min Lin, and Chongxuan Li. Scaling up masked diffusion models on text. ArXiv preprint, ab8/2410.18514, 2024. URL https: //rrii//arvvv.// org/ abs/ 2410. 18514.
OpenAl. Gpt-4 technical report. ArXiv preprint, abs/2303.08774, 2023, URL httph::t/axttttt org/abs/2300 08774.
Hugging Face OpenR1. Open rl: A fully open reproduction of deepseek-rl, 2025. URL https:
//ggithubb.. wtt.ccoo///hhoe innnaaaa face/ open- rl.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. ArXiv preprint, abs/2406.03736, 2024. URL https: aaxivv. org/ abs 2406. 03736.
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference Neural Informa- tion Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http:://apest nips cctpaper_fiiiiccceceee solffiiesss wwtps//pppp/222222 Bash/370df50ccfdree (et.eeeeffeeee11f888ee222222 Benchmarks Track. html.
Saleeeefffffcc (tweee1fff99929111111 -
_nd
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 -37441, 2023.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Allen Z Ren, Justin Lidard, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy optimization. In International Conference on Learning Representations, ICLR, 2025.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. ArXiv preprint, abs/2308. 12950, 2023. URL https:
www.xiioorrgo abs abs/aasss. 2308 12950.
Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffu- sion language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, An- gela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neurul Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL http:// http:///ppppp.eee nips ec htc/paper_fiieet http///ppep/////// /haah/ 3o0b13cc515724abBB e20011cc111772880000000000000000 http:e/eeeeeeeeeaaaaaataateaeaae Ceteraact-ooffeennceeeee html.
Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat. Esoteric language models, 2025.
14

---

Published as a conference paper at ICLR 2026
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.00347,, 2017. URL abs/1707. 06347.
Rulin Shao, Shayue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Si- mon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannanch Hajishirzi, Pang Wei Koh, and Luke Zettlemoyrr. Spurious rewards: Rethinking training signals rlvr.
rlver not. mon.oon. site/
Bourious-Rewards-Rae -ta.-eetttcnnna--taaooonn-iiiiia
2025. Notion
Blog.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. ArXiv preprint, abs/2402.03300, 2024. URL https: //arxiv. www.or///aao abs 2402. 03300.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias. Simplified and generalized masked diffusion for discrete data. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL /tatt::/papprre nips wt/ppppp.pppecccceeeeeee hacilpapp_feleeeee patass/papr//22/ 202pp///222//ahhh hash/ Dbad233br 4994995599 901 9ww.eee5ece000eeee000500 -otpaaeeeeeo----aaaaa--------oo- Conffeeeeett html..
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, and Gust Verbruggen. CodeFusion: A pre-trained diffusion model for code generation. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11697-11708, Singapore, 2023. Association for Computational Linguistics. doi: 111.663//1/2023.mmlppp. 2023.
URL https: //t////aaanttt/ //aaaaaaaaanoooyyooeeeeaa org/
716.
Yang Song, Jascha
Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In 9th Interna- tional Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:tpp/// www.ooeeeeieeeeeoo ew. net. for forum? d=PxTTG1 2RRHS.
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengeheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint 2403 14734, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302. 13971, 2023. URL https:
//arxiv. www.xiii.rggaba////ooaaaaa 2302.1
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu. Octothinker: Revisiting mid mid-training in the com/ hotm///coohiiii nker, 2025. Notion Blog.
era of rl scaling. fttps://tine
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 8:229-256. 1992.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast dllm: Training-free acceleration of diffusion Ilm by enabling kv cache and parallel decoding. ArXiv preprint, abs/2505.226 2025. URL https: // /twxxv.vvoorr.. abs/2505. 22618.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httt::://cpenreviie net htwprrrrwwweeeee//ooiiiiieeeeeee 555717
15

---

Published as a conference paper at ICLR 2026
Zhihui Xie, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, et al. Teaching language models to critique via reinforcement learning. ArXiv preprint, abs/2502.03492, 2025. URL https: /aarivivvv/../////// htarg//bs/25000 03492.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: hNtwSmtXRh.
hattps::/openreeeeee http///ppeeeeeieeeeeeeeeeeeeeeee net forum?ids
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong. and Mengdi Wang. Mmada: Multimodal large diffusion language models. ArXiv preprint, hls/2505.15809 2025. URL
tttps://arxivve org/ aabs/////////aaooo//ooo/a/ss//// 2505 15809.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2024a. URL
https: /apppi//aaxxv..o// org/ alss 2410 0 14157.
Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng. Jiahui Gao, Han Shi, Chuan Wu, Zhenguo Li, Wei Bi, and Lingpeng Kong. Diffusion of thoughts: Chain-of-thought reasoning in diffusion language models. ArXiv preprint, abs/2402.07754, 2024b. URL httptsst/sstii.....t///pptt/ org abb//
2402. 07754.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https: //hk://////lpppp github io ho/blog/////ooooooooooooooootooo 2025/ /rream.
Runpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding. ArXiv preprint, abs/2505.16990 2025. URL htRp::ts:://aree org/
abs/b/////// brrg/bs// 2505.555... 1 1990.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? ArXiv preprint, abs/2504.13877,, 2025. URL https: aawaaaa.vvv..........a..aa. org. abs 2504 13837.
Oussama Zekri and Nicolas Boullé, Fine-tuning discrete diffusion models with policy gradient methods. ArXiv preprint, abs/2502.01384,, 2025. URL sittps://arttt http://awwwwooaaa////3//// 01384.
Huaye Zeng. Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. ArXiv preprint, abs/2502.01718, 2025, URL https::/aat /ttp://aaavvv//aa///e/eeeeaawavr wwwoorgg//aoooo..oooo als 2502. 01718.
Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang, James Thornton, Zijing Ou, Joshua Susskind, and Navdeep Jaitly. Target concrete score matching: A holistic framework for discrete diffusion. ArXiv preprint, abs/2504. 16431, 2025. URL https: / /awrivivvvv./////////aaa////aavv org/ abs/ 6431.
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua M. Susskind, and Navdeep Jaitly. generating diversified paragraph via latent language diffusion model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- nips cc/paper.ppppee/pppppee
PLANNER:
16, 2023, 2023. URL Edba5e0a 9b57 Fce Fato5a/00eO00000000000000aaa2000 Aattaaa4----ttttttt----eoeeeeeee
hatpperr//2///hhhh. /hash/ html.
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. ArXiv preprint, 0bbs/2504.12216 2025. URL
https: /ataas////xiiii
org/ /250 12216. 6.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott. Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao. Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proc. VLDB Endow., 16(12):3848-384) -8600 2023. ISSN 21 2150-8097. doi: 10.14778/361 1540.3611550
16

---

Published as a conference paper at ICLR 2026
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusion model for text generation. In Conferenec on Language Modeling, COLM, October 7-9, 2024, Philadelphia, PA. 2024.
Fan Zhou, Zengzhi Wang. Qian Liu, Junlong Li, and Pengfei Liu. Programming every example: Lifting pre-training data quality like experts at scale. ArXiv preprint, abs/2409.17115 2024. URL https: arxiv. org/ abs 2200949..0000 17115.
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. ArXiv preprint, abs/2505. 2025. URL
https: /tte://aaae //tp///aaavv.///////////a..aaaaa org/ abas 220..1122.........1
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. ArXiv preprint, abs/2406. 15877, 2024. URL https: / arxiv. org/ abs /40/24466.. 15877.
17

---

Published as a conference paper at ICLR 2026
A PROBLEM FORMULATION
A.1 MASK DIFFUSION MODELS
Diffusson models (Ho et al., 2020; Song et al., 2021) contain forward process that gradually
corrupts data 36 Pdtata (36) into noisy variables .:7 through g( (o) I (( xr-1)
and a backward process that models the joint probability as (o(errr) = po(#r) I Po ( (2t). denoising 2, to reconstruct 20. The parameters
learned by minimizing the negative log-
likelihood via the evidence lower bound (ELBO):
- loo (x I By [mm) log po (zo x1) +
Lr,
(5)
with CT
Σ Ba (KL [1 ( [ 2b) (pe (x(-1|1))))
(6)
Hoogeboom et al. (2021); Austin et al. (2021a) first proposed discrete diffusion models, where they define the forward process with categorical distribution g( x2-1) Cat(ar;; Q, xt-1), where 2, E {0, 1j is one-hot vector of vocabulary size K, and Q2 E [0, 1 K is the transition matrix. A special case is absorbing discrete diffusion, where Q. (1 B.) 3. with 1 being an all-ones vector of size K and m the one-hot encoding of a special [MASK] token. Starting from Zo. the /-step marginal distribution is q(2x)(00) = Cat p = Q: vo.), where the cumulative product Q, = -I=II:- Q: = a + (1 - am)mm and a; = II 1
3t). We expect or to approach 0 such
that the fully noised data øy equals with probability 1.
For any two arbitrary time points < q(x,)ar,) Cat(#,; T. where Qaj: Q. Q. a, I _
< t
1, the transition distribution between them is a,
1m This allows to
compute the transition probability between any two timesteps using the ratio of their alpha values: 1-(1a.)
q( 22, xo) = g )q (xo)
1-2a =
a.-as I-ac
af D. m.
-t ).0.
g(0,, (20)
0,-ai
if = m + v.
I-a:
T-a;
ag at fo(x)) +
1-as
p=(22((0))
m.
(8)
at
1 att
(kL(zoll5o(arr)
for 0; m:
DKI(q(x./20 2ollo
T-a:
for e2 f m.
Thus,
03 Oz
CT
(t s) a1 Sx, mt log fo()))
(10)
(=2
where oæ,, is the indicator function, and fo( represents the logits of the tokens. In the continuous time limit where T + we set small timestep A, = t + (0, 1). The sum over timesteps becomes integral, and we have af مe=a,, Following the noise schedule a, = 1 which is widely adopted by Shi et al. (2024); Sahoo et al. (2024); Lou al. (2024); Ou et al. (2024), get
I-s
-ai
I-a; }. This can be substituted into Eq. (6), yielding the final ELBO at a sampled time t as a weighted cross cross-entropy loss:
1 Eyt (mm)
C.
Σ 8mp (xx) log fo(
(11)
A.2 GENERATION AUTORECRESSIVE-NES
We formally define local and global autoregresssivenss (AR-ness) here.
Problem Setup Inference for dLLMs often utilizes low-confidence remasking with the number of diffusion timesteps set equal to the sequence length to ensure high performance. this setting, let the target sequence length be L, and at each diffusion decoding iteration t 1,..., T. the set of still-masked positions just before step t M,- S {1, 2....... L}. We denote by P: E M,-1 the single position unmasked at step I, thereby producing the full decoding order (PI,..-,Pr].
18

---

Published as a conference paper at ICLR 2026
Local Autoregressive-nsss: Contiguous TokenToken Prediction
For any integer & 2 1, define
[1, if (pr-sk-m {Pr i:i 0, otherwise.
Iocat (t, 1))
The Local AR-ness @K k is then computed as NTP@k = + Ei= (t, k). Local AR-ness measures the tendency decode the immediate successor of previously generated token, capturing sequential continuity. It is non-increasing with k, as it becomes harder to maintain longer consecutive spans.
Global Autoregressiv--nes:: Earliest-Firse Mask Selection m²1 < xm- <... m (1Me-1l) Then
At step t. sort the masked positions
Tgloball (t, 1,
(2) pt E {m mi-1h
(K)
0, otherwise.
The Global FMS-ratio@k is FMS dency to always unmask the earliest remaining token, capturing front-to-back filling strategy. Together, these ratios reveal how the model behaves during generation. The ratio is non-decreasing with k as the criterion becomes easier to satisfy when more early positions are allowed.
E k). Global AR-ness measures the ten-
(t,
A.3 COUPLED GRPO
In this section, we provide a detailed formulation of our Coupled GRPO algorithm. As discussed in $5, we improve upon the baseline GRPO by introducing a coupled sampling scheme for more accurate probability estimation. The complete Coupled GRPO algorithm presented Algorithm 1.
Probability Estimation with Coupled Sampling For a given completion sequence of length L, we select À timestep pairs (t, where & + f T. For each pair, we create two complementary masks Mj and Mj defined binary vectors in (0, 1)4, such that:
M, V Mj 1, M, A Mj 0,
(12)
where V and A denote element-wine wise OR and AND, respectively, and 1, 0 {0, 10,1) the all-ones and all-zeros vectors. The probability estimation for token o (also marked as Zg)) then computed as:
no c, oier) X+1
Σ [C((xt) + L,(00) + LT(ØT))
(13)
++i=T
Le the loss term from Eq (11) at timestep f. In detail, we have Le(xr)) = M2 CE((,, mo) where CE stands for the cross entropy loss.
The coupled sampling scheme provides several benefits: (i) Full Coverage: Each token guaranteed to be evaluated exactly once in each coupled pair, ensuring complete coverage of the sequence. (ii) Reduced Variance: By evaluating each token under realistic mertial-maskigg contexts, we reduce the variance in probability estimates compared to full masking, (iii) Computational Efficiency: The coupled sampling requires only two additional forward passes per update compared to the d1 (Zhao et al., 2025) baseline when 1. The variance reduction can be formally quantified in the next section, Appx. A.4.
Analysis
A.4 THEORETICAL ANALYSIS OF COUPLED-GRPO
In this section, we provide formal analysis of the coupled sampling scheme used to estimate the per- token log- og-probability proxies within our coupled-GRPO framework. GRPO requires stable estimates of these per-token quantities to compute the importance sampling ratios for the policy gradient update. We demonstrate that our coupled approach can viewed as a direct and powerful application of the Antithetic Variates (Hammersley & Morton, 1956; Hammersley & Mauldon, 1956) variance reduction technique. We prove that it provides an unbiased estimator for the desired per-token quantity and, critically, that it is guaranteed to reduce estimation variance.
19

---

Published as a conference paper at ICLR 2026
Algorithm 1 Coupled GRPO:
Policy Optimization with Coupled Sampling
1: Input: Reference model ref condition set C, number of completions per condition G, code test
cases T. hyperparameters (4, 33 and
2: Initialize <- fref
3: while not converged do
4: update reference model Fref <- Te
5: for step = 1,...,
T do
6:
Wold
7:
Sample a batch condition C, ~
8:
Sample G completions {o: Totd(-le), for each Cb
9:
For each o,, compute reward r r(o)) by execute test cases To of each c
10:
Get advantage A, r(o,)
Σ r(o,) or LOO A, r(o,) G-T Eg E(zr
11:
for GRPO iteration j = 1, , do
12:
Randomly sample a timestep pair (4,, where i, + 1 TT
13:
Create complementary masks M, and Mr, for batch Compute Lt,, Li, and Lr
14:
15:
Compute coupled probability estimates in Eq (13) and importance ratios p Update To by gradient descent on JGRPO Eq (4)
16:
17:
end for
end for
19:
end while
return T
The core challenge is obtain stable estimate of a score for each token in generated sequence, where this score serves as proxy for its log pro--abbbbilty.. This score is defined as an expectation over random process involving diffusion timestep and mask M.
Assuming the linear noise schedule, the sampling process as follows: timestep / is drawn from distribution [0, 1]. typically f (0, 1). Different from Appx. A.3 which formulates the loss computation batch-wise in this section we examine the token-wise formulation. Conditional on t, mask for each component in the sequence sampled independently from Bernoulli distribution M₁ ~ Bernoullitt). k I.
For each token in a sequence o, we define per-token scoring function, g(t, M, k), which is non-zero only if token l is masked:
g(,, M, M,K) Mr + ((-
(14)
where E(.) is the cross-entropy loss for token o² given the condition and the unmasked context 01-M- The quantity we wish to estimate for each token k is its expected score:
Uk = E [g((, (, M, A:]]
(15)
This estimated Ug is then used to compute the policy probability ratio / / in the GRPO objective.
A.4.1 STANDARD vs. COUPLED MONTE CARLO ESTIMATORS
Standard MC Estimator. To estimate the vector of scores (Ui...., VL), one can draw 2N i.i.d. pairs {tt, M. The estimator for each token & is:
2N glt. M).
0K MC
Σ
(16)
2N
In any given sample g(t, M;, k) is non-zero only for the subset of tokens & where Mi 1. Many samples are needed to obtain a reliable, non-zero estimate for all tokens.
Coupled (Antithetic) Estimator. Our coupled sampling method generates antithetic pairs. We draw N pairs ((4, M and deterministically create their counterparts (f,, M,) 4, 1 M,).
20

---

Published as a conference paper at ICLR 2026
The antithetic variates (AV) estimator for 2y is:
N
g(t,, Mr, k) - g(li, Mi,h)
(17)
N
2
A key structural property emerges here: for any given token & and sample i, exactly of the two terms in the inner sum is non-zero. This is because M and M complements. This guarantees that every token receives non-zero score contribution from every coupled pair, ensuring full coverage and making the estimation process vastly more efficient.
Mi, are binary
A.4.2 PROOFS
Unbiasedness. We show that ūK is an unbiased estimator of vk-
Proof. The proof relies on showing that the joint probability distribution of (t, MM is identical to that of its antithetic counterpart (f, .M. As proven in the previous version, under the symmetric sampling scheme for t and the dependent Bernoulli sampling for M, we have p(t, p(t, M).
Since the random variables (1, M) and (f, M) are identically distributed, the expectation of any function of these variables is the same:
Elg(t, M, k)] E[g((, M, Uk.
(18)
By linearity of expectation, the expectation of the AV estimator is:
1
Al
Σ
(Eig(t, M, A)] Elg(f M, k)) 2N
Σ Ukk + uk) uk-
(19)
2N
Thus, the coupled estimator is unbiased.
Variance Reduction. We now provide direct and rigorous proof that the coupled estimator has lower variance than the standard MC estimator.
Proof. The variance of the AV estimator for token A given by:
Var(ūk. Var(g(t, M, k)) + Cov(g(t, M, M. g(t, M))
(20)
2N
Variance is reduced if and only if the covariance term is negative. Let us analyze the covariance:
Cov(g(t, M, k), gf,, M, k)) Eig(t, M, k) o(f. M, A)] E[g(t, M, k:]] Elg(f, R)).
(21)
Consider the product term inside the first expectation:
g M, M, = ( (Ms- tec...)
Mr -ib.)
(22)
The crucial insight is that the product of the mask indicators M Mr is always zero, since M, 1 - M2 and Mr is either or 1. Therefore, the product of the scoring functions identically zero for all possible values of t and M. This means its expectation is also zero:
Elg(1, M, k) g(i, M, k)] = 0.
(23)
Substituting this back into the covariance formula:
Cov(g(t, M, gf. M, k)) 0 (E(g(t, M,)) (E[g(E, k))
(24)
= Uk Uk
(25)
= -
(26)
Since the loss f(.) is non-negative, the scoring function is negative.. Its expectation, Uk must therefore be non-negative. Assuming there is some configuration where a loss is incurred (i.e., en 0), we have:
Cov(g(t, M,k), (f, M)) - 0.
(27)
21

---

Published as a conference paper at ICLR 2026
The covariance is guaranteed to be negative. The variance reduction is therefore not just plausible but a mathematical certainty of this estimation scheme. The amount of reduction is:
03
2
Var(ôk,Mc) - Var(Uk..v)
9
0.
(28)
2.V
2.N
2NV
This result stems directly from the mutually exclusive nature of the estimators for given token within coupled pair, direct consequence of the complementary masks. It is worth noting that in antithetic sampling, the key requirement negative correlation between paired samples, not symmetry of the underlying distribution. Pairing with T - t ensures that the two masks have complementary masking ratios over the sequence. This holds for general monotonic schedules, even if a is not symmetric around T/2.
A.5 RELATIONS BETWEEN AR-NESS AND COUPLED SAMPLING
The AR-ness analysis directly influenced the design of Coupled-GRPO in two ways: (1) AR-ness reveals that diffusion models over-update early (left-most) positions. Our measurements show a strong entropy sink at the prefix: the model produces very confident logits there, standard di dI-style sampling causes GRPO to repeatedly update only these early tokens. This leads to biased assignment and slows down learning for later positions. Coupled-GRPO introduces complementary masking to ensure that later tokens are also sampled and updated, directly addressing this imbalance uncovered by AR-ness. (2) AR-ness decreases with higher temperature, enabling more parallel decoding which is important for RL. We observe that higher sampling temperatures reduce AR-ness and produce more diverse, arbitrary-order decoding. Such diversity is essential for RL rollouts. However, this shifts probability mass to later positions, making it even more important to sample and train on those positions reliably. Coupled- goupled-GRPO exactly provides the balanced sampling across the entire sequence instead of focusing on the early positions.
Coupled-GRe -apled-GRPOOOO not only uses AR-ness insights but also feeds back to produce more parallel, less AR-biased diffusion behavior. As in Appx. C.4, after training, we find that Coupled-GRPO reduces AR-ness and makes the model more robust to step-truncation: using 0.5x fewer diffusson steps (2x speedup) incurs smaller accuracy drop compared to the pre-RL model.
B IMPLEMENTATION DETAILS
B.1 TRAINING DETAILS
Adaptation pretraining During pre-training, we filter the code pre-training corpus from Re- fineCode (Huang et al., 2024) and Stackv2 (Lozhkov et al., 2024). Since RefineCode only provides each item's 's index in Stackv2, we built a local index engine to download the raw data from Stackv2. We also used text and math data from Fineweb (Penedo et al., 2024) and DCLM-Pre pro (Zhou et al., 2024). The final processed dataset contains around 400B tokens, as shown in Table 3. We adopted the code-to-text ratio suggested in Qwen-2.5-Codee (Hui et al., 2024) and OpenCoder (Huang et al., 2024). The training was conducted 10 nodes of 8 H100 GPUs each, using BF 6 and full-shard FSDP (Zhao et al., 2023). The total wall-clock time for training on 65B tokens (100,000 global steps) was approximately 40 hours. The single-GPU batch size Following LLaDA (Nie et al., 2024), we truncated 1% of the data to a random length to improve handling of variable-length inputs. Additionally, for another 1% of the data, we used random-length prefix as the condition during the diffusion process, which was kept unnoised. We used the Adam optimizer with maximum learning rate of 2e-5, with linear warmup of 20,000 steps followed by a cosine decay schedule to 10% of its peak value at the end of training. The attention mask annealing was performed over 10,000 steps, following DiffuLLaMA (Gong et al., 2025). In our experiments, we observed that training on more tokens in Stage 1 did not improve performance on downstream tasks (Table 4), nor did it lead to further improvements in Stage 2. Therefore, we used the model trained with 65B tokens as Stage 1 model. This is a well-known behavior in continual pre- pre-training: when a strong base model is trained on high- proprictary data, extending training on lower-quality or distribution- iisstibution-miimaaccee mismatched data can hurt downstream performance. In our case, Qwen2.5-Codl is
2, and the context window was 4096.
22

---

Published as a conference paper at ICLR 2026
trained with high-quality, non-public data. Stage 1 adaptation uses an open-source mixture whose quality is inevitably lower, so degradation at long training horizons is expected. With a higher-quality pre-training corpus, might have reached different conclusions.
Source
# tokens
Sample weight
Percentage
RefineCode from Stackv2 (code)
330B
78%
Fineweb code page (text)
55B
20%
DCLM subset (text)
33B
Fineweb math page (math)
3B
2%
Table 3: Adaptation Training Data Recipes on Stage
Mid-training We used around 16B tokens of annealing code data (Huang et al., 2024) during mid-training. This dataset contains an algorithmic corpus and synthetic data, such as high-quality code snippets and code textbooks. This high-quality mid-training data significantly improves the capacity of the base model, and we chose the model trained on 65B tokens (roughly 4 epochs over the training data) as the final version of DiffuCoder Base. The training was carried out on 8 nodes with 8 A A100 GPUs each, using BF1 6 and full-shard FSDP (Zhao et al., 2023), which took 90 hours of wall-clock time. We used the Adam optimizer with a maximum learning rate of 1e-5 and linear warmup of 2,000 steps. Other settings were the same as in Stage 1.
Instruction tuning We conducted classifier-free guidance SFT (Gong ct al., 2023a; Nie et al., 2024; Ye al., 2024b) for DiffuCoder using 436K instruction tuning samples from OpenCoder (Huang et al., 2024). The classifier-fre guidance approach uses a conditional mask to prevent the diffusion process from adding noise to the condition prefix. We used the same chat template as Qwen2.5-Code (Hui et al., 2024). We trained a new padding token that was used to pack each sample to fixed length to control the generation length. We compared different SFT strategies for DiffuCoder instruction tuning, including: (i) using fixed sequence length of 2048 for each sample with conditional mask; (ii) using a fixed 2048 length but mixing conditional and unconditional training; and (iii) flexible padding, where padded to the maximum sequence length in the current batch instead of fixed 2048. Finally, we progressively added a conditional mask for the first epoch, considering that Stages I and 2 were trained unconditionally, and then trained the remaining epochs using a fixed sequence length of 2048. The SFT code is based on LLaMA-Factrry The training was performed on 8 nodes of 8 H100 GPUs each, using BF16 and ZeRO2 (Rajbhandari et al., 2020), taking around 24 hours. We used the Adam optimizer with maximum learning rate of le-5 5 and linear warmup ratio of 0. 1, followed by a cosine decay schedule to 10% of its peak value at the end of training.
Coupled GRPO For RL training, for both our model and the ablation baselines, we filtered 21 hard samples from Acecoder-87k (Zeng et al., 2025) with verifiable test cases and trained for one epoch. We used the pass rate of reference solutions in this dataset to filter for questions with a low average (bottom 20%) and high variance (top 40%) pass rate. These were marked as hard samples, yielding the final 21K samples used for GRPO training. We found that filtering the training samples by difficulty into a proper range is important. We used the online sandbox E2B for code execution and reward verification. We trained the models on a single node with 8 H100 GPUs for wall-clock time of 40 hours. The default GRPO training parameters were: reference model syne steps 64, number of iterations = 2, B = 0.01, E = 0.5, learning rate 1e-6, and maximum completion length 256. The rollout parameters were: diffusion timesteps 256, rollout samples G = 10. and sampling temperature 1.2. When sampling coupled i, we empirically chose range of [0.2, 0.8) instead of [0, 1.0) to avoid extreme loss values (Figure 8). Despite training for only one epoch with coupled
GRPO, we observed that DiffuCCder-IIsttree
performance continued to rise as we added more
training steps.
We designed a weighted reward for each completion o as r(o:) = 2.0 X Tende (o:) + 0.5 Froemar (o:). Teode (o,) is the pass rate on the test cases, evaluated only if Tioemat (o;) = 1. Ttormat (o:) =
hhttps://git wtt.huu..coo/h/oom com/hi (aapgg//iiaaMA-acttoo httpp: //tes.//ebbbee. dev/
23

---

Published as a conference paper at ICLR 2026
Table Raw results on coding tasks for LLMs and dLLMs in 7/8B scale. denotes that the results are collocated from public reports instead of evaluating by ourselves. BigCodeBench has completion
(C) and instruction (I) version of template during evaluation. +SFT means conduct the same instruction tuning with DiffuCodde-Instrruc...
HvaanEval
MBPP
BigCodeBench (C)
BigCodeBench ())
Model
Plus
Full
Hard
Full
Hard
Base Models
Qwen2.5-Code
61.6
51.8
75.9
61.4
46.1
16.2
40.2
14.2
OpenCoder"
66.5
63.4
79.9
70.4
40.5
9.5
LLaDA
35.4
30.5
50.1
42.1
18.9
4.1
Dream
56.7
50.0
68.7
57.4
23.6
4.1
DiffuCoder (Stage 65B)
67.1
60.4
74.2
60.9
40.2
12.8
1.8
0.7
Stage 1 65B
39.0
31.7
48.4
38.3
Stage
720B
23.3
38.8
31.3
Stage 2 16B
66.5
61.0
71.9
57.1
36.7
8.8
Instruct Models
Qwen2.5 -.onl-5-Coer--nstruc
90.2
85.4
83.9
72.0
21.6
42.2
18.2
Owen2.5-Coder+SFF
82.9
75.6
80.1
66.1
39.5
14.9
OpenCoder-Instuune
83.5
78.7
79.1
69.0
40.3
16.9
LLLDD-Instruct
35.4
31.7
31.5
28.6
16.5
2.7
14.1
1.4
Instrc-nsttuctt
57.9
53.7
68.3
56.1
10.6
0.7
11.4
2.7
Dream+SFT
56.7
50.6
71.7
58.7
27.4
25.9
DiffuCCoder-Instruut (S ep)
72.0
65.2
75.1
61.9
35.7
12.2
34.0
8.8
1 epoch
67.1
60.4
75.7
61.9
31.1
8.1
32.0
6.1
- flexible padding
65.2
58.5
71.2
59.5
35.4
11.5
32.4
10.1
+ coupledGRPO (Tep)
73.2
68.3
78.6
67.5
40.4
10.8
37.5
10.8
+ coupledGRPO (2ep)
65.2
79.1
65.3
42.8
14.9
39.4
13.5
Commercial Models
GPT 4o"
90.2
82.2
49.9
Mercury"
90.0
77.1
45.5
Gemini Diffusion"
89.6
76.0
45.4
20.0
unweighted less avg unweighted loss weighted loss average weighted
17.5
15.0
9 12.5
10.0
7.5
SWoot spot
0.0
0.2
0.4
0.5 timestep t
0.
1.0
Figure 8: Validation loss distribution of DiffuLLaMA (Gong et al., 2025) for different timesteps. The weighted loss refers to L. in Eq. (11), while the unweighted loss refers to the cross-entropy term in the masked diffusion loss without the 1/t weighting. Timesteps that are too large or too small will lead to extreme values. The sweet spot is in the span of [0.2, 0.8), which is also consistent for DiffuCoder.
0.5 if o; contains a valid Markdown code block and passes a Python syntax check;
0.25 if the code block format is correct but the code has syntax error;
0 if the Markdown block format is not matched.
Concretely, first require the completion to contain valid Markdown code block and to pass Python syntax check (the format reward), before evaluating the test pass rate as the main correctness signal. This gating is important: if code correctness rewarded without enforcing format, the
24

---

Published as a conference paper at ICLR 2026
model could receive positive rewards while ignoring formatting, leading to progressively degraded structure and harder parsing. In practice, the model quickly saturates the format reward, so the relative upweighting of the test-based component (2.0 0.5) becomes essential for providing meaningful learning signal.
The results are in Table 4. Our design choices in Appx. B.1 rely these results. DiffuCoder Base underperforms on the instruction-uued BigCodeBench (I), likely because the pre-training corpus teaches the base model to focus on completion rather than instruction following. Consequently, fares better on the subset BigCodeBench (C) with completion query. After instruction tuning, DiffuCoder- Instruct achieves reasonable performance on BigCodeBench (I), indicating that instruction tuning teaches the model to follow instructions.
Table Chat template during the evaluation for HumanEval and BigCodeBench (C).
Table 6: Chat template during the evaluation for MBPP and BigCodeBench (I).
im.start [ >system
immstaart <|m.start|>syysen >system
You are a helpful assistant.<limmeng
You are helpful assistant. <mnmeen < | >user
<I [im.start/>use
Please complete the following
...
problem:
{prompt)
...
{prompt}
<| immend|>>>
"''
< im.start >asssistat
<limendd|>
Here is the code to solve this problem:
<I limsttart Passistaat
Here is the code to solve this problem:
" 'python
" 'pyyhon
B.2 EVALUATION DETAILS
By default, LLaDA (Nie et al., 2024) employs low confidence remasking with temperature 0, while Dream (Ye et al., 2025) uses top negative entropy remasking with temperature 0.2. Both models use maximum sequence length and diffusion timesteps of 512. GSM8K tests are conducted using Im-harness and code benchmarks are based on Qwen2.5-Coder's evaluation suitcase DiffuCoder shares the architecture (Qwen2.5-7B) tokenizer, and inference implementation of Dream-7B. The chat templates we used during the inference are listed below.
C ADDITIONAL RESULTS
C.I ENTROPY SINK
When dLLMs perform conditional generation, the first diffusion step starts with a fully masked completion given a prefix prompt and attempts recover the completion sequence. At this step, we record the confidence score of each recovered token in Figure 3(a). Appx. C.2 also lists the entropy heatmap across all decoding timesteps. The default decoding algorithm from LLLDD and Dream selects the token with the highest confidence while remasking the rest. LLaDA uses log probabilities while Dream uses negative entropy to measure confidence, where a larger value indicates that the model is highly confident about that token.
Remarkably, the resulting distribution displays characteristic "L""shaped pattern. refer to this phenomenon the entropy sink. We hypothesize that the entropy sink arises because the intrinsic
'ttpps: wtw....oo/...ttteeeeeeeeeeeeeeee http:///eteeeeee/meeeeeaaaaaorra 'https: //g thub. coom/wennnn http:///eeeea//eeeeeeeoo/oeeeeee /etpe////e..coeeeeeee//////iiiii coder-eval
25

---

Published as a conference paper at ICLR 2026
nature of text biases the model toward tokens that lie immediately the right of the given prefix: those positions receive stronger positional signals and closer context, leading the model to assign them disproportionately high confidence. This phenomenon may be related the cause of the attention sink (Gu et al., 2024; Xiao et al., 2024), but its underlying cause requires further analysis and verification. This entropy bias toward locally adjacent tokens explains why dLLMs still maintain a non-trivial level of AR-ness.
C.2 ENTROPY PATTERN
In Appx. C.I, present the phenomenon entropy sink and illustrate the L-shaped distribution of the confidence score for a single time step (the first forward), In Figure 9, we extend the visualization to 2D heatmaps. We can still observe casual bias in these examples, but more token information and flexibility are involved compared to the strict causal mask.
20
Decoding steos
Decoding steos
15
Decoding steos
29
so 70 so 50 40
99
20
50
8
60
80
300115
Index Tokens
Index Takens
Index of Tokens
Figure 9: Visualization of the decoding entropy for random samples. The x-axis is the index of generated token, while y-axis refers decoding steps. Here the diffusion timestep and generation length to be equal.
C.3 DECODING SAMPLES
Figure 10 demonstrates the generation order for different temperatures. The background color runs monotonically from red (earliest) through the spectrum to purple (latest). we can see, the higher temperature leads to less AR sequence generation. The model tends to determine the right-hand side first, including pad tokens which decide the generation length, and the key parts of this code snippet are generated near the end.
Fooe mposre Sepore
FE3M PSESIS ANCUPE LEEE
sterents
Sert TV
de aadae BaMArs S000000001
retur Falue
Tea
recum
Trae
Conal ressus scrtis
Figure 10: Visualization of the decoding trajectory of DiffuCoder- InstrCcde--Intrrucct under different sampling temperatures. Each character's background is colored from red purple according the recover order of the [MASK). Left: temperature is 0.2; Right: temperature is 1.2.
26

---

Published as a conference paper at ICLR 2026
Instruct (lep)
(lep)
Instruct
Scores (96
GRPO 1x temp0.2
temp0.2
3
GRPO 1x temp0.4
tempa.4
HumanEval
HumanEval+
MБPP
M6PP+
Figure 11: Different model variants act differently when changing decoding timesteps to 1/2 sequence length. 1x means the default setting where decoding timesteps equal the sequence length while 2x means fewer steps which will result in 2x speedup.
C.4 DECODING TIMESTEPS
Another key motivation for measuring AR-ness is its relationship with generation parallelism. A model with very high AR-ness implies strong left-to-right token dependencies, which limits op- portunities for parallel decoding. Conversely, lower AR-ness suggests that the model can generate multiple tokens more independently, enabling fewer diffusion steps and thus faster generation. In other words, by monitoring AR-ness, also uncover how much headroom remains for accelerated, parallel decoding schedules. Figure 11, if we correlate the performance drop from 1x to 2x with the non-AR (parallelism) decoding pattern, where higher drop indicates higher AR-ness, then can draw the following conclusions. (1) Compared with the Instruct model (the starting point of RL training), GRPO training improves DiffuCoder Instruct'ss parallelism. (2) Compared with one epoch of instruction tuning, training for more epochs (5ep here) can reduce the AR-ness of the model. (3) For different sampling temperatures in GRPO-trained DiffuCoder, the higher temperature (0.4) brings less AR-ness and thus smaller performance drop at 2x speed.
2.0
1.8
2.0
Reward
1.6
Reward
1.5
1.4
Baseline (p=0.1
1.4
1.0
Full mask (p-0)
Coupled
Coupled
1.2
Coupled GRPO
do-pled
0.0
Training Epoch
Training Epoch
Training Epoch
Figure 12: Reward curves during GRPO training. Left: Comparison between coupled GRPO and dl baselines (based on an early version biffuooder-Insrruee Middle: Decoupled GRPO uses the same number of sampling times but with randomly sampled masks (based on an early version of DiffuCCoder-Inttrucy Right: Coupled-GRPP on DiffuCoder compared with regular GRPO for the AR model Qwen2.5Coder+SFF..
C.5 COUPLED GRPO TRAINING
We monitored the completion length during GRPO training but did not observe consistent increase in length as seen in AR GRPO training (Shao et al., 2024; OpenR1, 2025). A possible reason is that do encourage long-chain reasoning generation (Liu et al., 2025c), which could be future research direction. In our experimental environment, the end-to-end GRPO training time for DiffuCoder twice that of the AR model Qwen2.5-Codee Same with pre- pre-training, post-training of diffusion LLMs is also more compute-intense intensive than training their AR counterpart due to the sampling and learning efficiency ds.
We further include two supplementary experiments to clarify the generality of coupled GRPO.
Effeet of larger coupling parameter (A = 2). We add results for = to examine whether increasing the number of complementary mask pairs yields additional benefits. The setting follows
27

---

Published as a conference paper at ICLR 2026
standard GRPO training single node with 8XH100 GPUs. As shown in Table 7, X provides only marginal accuracy gains over 1, while substantially increasing wall-clock time (from 40 to 54 hours). This confirms that À offers the best yatranne-accuurcc-commpp trade-off.
Table 7: Performance and cost comparison across A.
X
Theoretical Variance
Training Reward Curve
Wall-clock Time
MBPP
MBPP+
0 (d1)
High
Unstable Stable
34 hrs
0.0
0.0
1 (ours)
Low
40 hrs
78.6
67.5
2
Low
Stable
54 hrs
78.8
67.2
Applying Coupled-GRPO to Dream 7B. To demonstrate that coupled GRPO is model-agnostic and not tied to DiffuCoder, we apply the same RL recipe to an independent diffusion model Dream- 7B-Instruct (Ye et al., 2025). We follow the identical reward design and rollout setting. Table 8 shows that coupled GRPO also improves Dream 7B particularly on MBPP and MBPP+. This supports that method generalizes beyond single architecture training pipeline.
Table 8: Applying coupled-GRPO on Dream-7B-Instreen
Model
HE
HE+
MBPP
MBPP+
Dream-7B-Instee In-ttrruct
57.9
53.7
68.3
56.
coupled-GRPO 57.9
52.3
72.5
58.2
Overall, these additional results reinforce (i) the diminishing returns and increased cost large A, and (ii) the broad applicability of coupled-GRPO across diffusion LLMs. Although our experiments are on code, coupled-GRPO is also domain-agnostic. The method can be applied directly to any diffusion LLM, including text and multimodal settings, with curated domain-specifie RL data.
28


Reviews:
[{"review_id": "X00QWuwNmW", "reviewer": "Reviewer_ZNHR", "summary": "This paper is a systematic study and methodological work on applying diffusion language models (dLLMs) to code generation; it introduces AR-ness metrics to quantify decoding order and a diffusion-native RL approach (coupled-GRPO), delivers consistent gains across multiple code benchmarks, and reveals the key phenomenon that temperature affects both token selection and generation order.", "questions": "1.\tPlease evaluate λ>1 (multi-pair coupling) and alternative t-samplings (e.g., Beta/log-uniform), and report the variance–accuracy–compute trade-offs.\n2.\tPlease add a compute-parity comparison against semi-autoregressive RL (block diffusion / re-masking) with matched rollouts and wall-clock time to quantify the benefit of avoiding AR bias.\n3.\tPlease demonstrate robustness by sweeping syntax/format/test weights, adding static analysis and richer tests, and reporting mean ± σ over 3 seeds.\n4.\tPlease report practicality: end-to-end throughput (tokens/s) and training/inference cost, and comment on whether your method stacks with existing dLLM accelerations.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tProposes local/global AR-ness@k to compare dLLMs vs. AR LLMs across stages and modalities. \n2.\tHigher temperature makes decoding less AR-like and raises pass@k; real sample trajectories are visualized. \n3.\tCoupled-GRPO lowers variance via complementary masks and avoids semi-AR bias, yielding stable post-training gains. \n4.\tFrom adaptation → mid-training → SFT → RL, with compute setups that aid reproducibility. \n5.\tWhen halving steps (2× speed), performance drops less after GRPO.", "weaknesses": "1.\tGeneralization to multi-language, multi-file, or agentic tasks is unclear. \n2.\tMulti-stage large-scale training plus RL rollouts; dLLM GRPO takes ~2× AR’s wall time. \n3.\tEven with variance reduction, the training still approximates token log-likelihoods.\n4.\tStage-1 at ~700B tokens hurts downstream performance, implying sensitivity to data quality and early stopping. \n5.\tNo λ>1 (multi-pair coupling) or alternative t-distributions; limited analysis of reward weighting and verifiers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q2fgFirOOn", "reviewer": "Reviewer_eErP", "summary": "This paper presents *DiffuCoder*, a large-scale diffusion language model (DLM) designed for code generation. Unlike traditional DLMs that struggle with fixed decoding order and limited self-correction capability, DiffuCoder introduces a complete four-stage training framework including pretraining, mid-tuning, instruction fine-tuning, and reinforcement learning. A major contribution lies in the proposed coupled-GRPO, a variance-reduced reinforcement learning algorithm that enables effective self-reflective sampling under high-temperature decoding. The authors also conduct detailed behavioral analysis, demonstrating DiffuCoder's flexible, non-autoregressive decoding strategy and strong performance across major code and math benchmarks.", "questions": "1. Have you considered applying coupled-GRPO to existing diffusion backbones like LLaDA or Dream to assess its modularity and transferability?\n2. Given that the proposed method enhances performance with only two diffusion blocks, have you observed any performance degradation or instability with longer generation lengths or more complex prompts?\n3. Do you expect the entropy-sink decoding behavior to generalize to non-code domains, or is it specific to code and math where token dependencies are more structured?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Proposes the first full pipeline for large-scale masked diffusion models in code generation, demonstrating competitive performance with AR models.\n* Introduces coupled-GRPO, a principled, theoretically justified improvement over prior RL-based DLM training.\n* Deep behavioral analysis offers valuable insight into how diffusion decoding diverges from autoregressive patterns.\n* Experiments are comprehensive, including comparisons with prior DLMs, autoregressive baselines, and multiple ablations.\n* The paper is clearly written and well-structured, making technical innovations accessible.", "weaknesses": "This work is strong across the board — in terms of methodological innovation, clarity of explanation, and thorough ablation studies. I did not find any substantive weaknesses, and I believe the paper easily meets the bar for poster acceptance at ICLR. However, given that the core method could likely generalize to domains beyond code (e.g., open-ended language modeling, multimodal reasoning), it would have been valuable to see some discussion of this potential in the main paper. Including such analysis or preliminary results could elevate the work to oral-level significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c3UgGw5v99", "reviewer": "Reviewer_vwfe", "summary": "This paper introduces DiffuCoder, a 7B-parameter diffusion LLM (dLLM) for code, and uses it to investigate the decoding behavior of dLLMs. The authors propose \"autoregressive-ness\" (AR-ness) metrics to quantify how left-to-right dLLM generation is and analyze the effect of sampling temperature on diversity. Based on these insights, they introduce Coupled-GRPO, a reinforcement learning method that uses complementary mask pairs to reduce variance in policy gradient estimation. Experiments show DiffuCoder is competitive with AR baselines and that Coupled-GRPO provides modest performance gains (+4.4% on EvalPlus) and improved decoding parallelism.", "questions": "Please refer to **Weaknesses**.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1- Provides the **first systematic measurement of AR-ness** in diffusion LLMs, with clear metrics and visualizations.\n\n2- **Coupled-GRPO** is theoretically motivated (antithetic variates) and supported by formal variance-reduction proof.\n\n3- Offers a **complete open-source training recipe** for a diffusion-based code LLM, potentially useful for the community.\n\n4- Connects decoding order, sampling temperature, and parallelism, yielding practical insights for efficient generation.\n\n5- Writing and figures are generally clear and reproducible; appendices are thorough.", "weaknesses": "### **Incremental Analytical Novelty** \nThe key observations from the AR-ness analysis (e.g., the effect of temperature, the difference between adapted and from-scratch models) confirm properties that are largely expected from the principles of masked diffusion. The contribution is in the measurement, not the discovery of new phenomena. I would suggest that the authors tone the it down in the contribution sections.\n\n### **Modest and Inconsistent Algorithmic Gains**\nThe improvements from Coupled-GRPO are relatively small and not uniform across all evaluated benchmarks (Table 2), raising questions about its overall impact.\n\n### **Insufficient Ablations**\nThe paper lacks critical ablations for its core method, including:\n- A sweep of the coupling parameter λ to test if the benefits of \"coupled\" sampling scale.\n- A direct comparison against a \"2x decoupled\" baseline to isolate the effect of complementary sampling from simply doubling the number of forward passes.\n- A clear analysis of the computational cost/overhead of Coupled-GRPO.\n\n### **Missing Baseline**\nI understand that **Dream-Coder-7B** is relatively a new baseline, but the absence of a comparison with that model gives an unjust favor to DiffuCoder over other open-sourced dLLMs. I would love to see that comparison added to the tables in the rebuttal period, as the model was released almost two months prior to the submission deadline.\n\n### **Overstated Claims** \nSeveral passages (e.g., on temperature-driven diversity) could be more accurately framed as measurements of known model properties rather than novel discoveries.\n\n### **Missing Citations**\nThe paper does not fully contextualize its work within the broader literature. Specifically:\n- When discussing early masked diffusion formulations (e.g., around line 52), the paper should cite \"Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data\" (Ou et al., 2024)[1] as a concurrent and relevant work in this space.\n- The discussion of semi-AR decoding and Block Diffusion (Arriola et al., 2025)[2] would be strengthened by also citing the concurrent work \"Unifying Autoregressive and Diffusion-Based Sequence Generation\" (Fathi et al., 2025)[3], which explores similar hybrid modeling ideas. Properly crediting these concurrent works is important for a complete scholarly narrative.\n\n### **Practical Utility of AR-ness** \nBeyond interpretability, what are the actionable insights or design recommendations from the AR-ness analysis? For example, could a model's AR-ness score be used to dynamically adjust the number of diffusion steps during inference for speedup without a major performance drop?\n\n### Minor Typographical or formatting issues\n1- Page 4, Section 3: \"We observed that training with 700B tokens in Stage 1 led to worse performance than using only 65B tokens\" -> This is a key observation. The reasoning is plausible, but could be slightly expanded.\n\n2-Page 5, Table 1 Footnote: \"Scores are bolded when our model outperforms LLMs specialized for code (excluding LLaDA or Dream).\" -> This is an unusual and potentially confusing bolding criterion. It would be clearer to bold the best score in each column or simply state the comparison in the text.\n\n3- Page 8, Section 5: \"w. full mask completion\" -> Should be \"w/ full mask completion\" or \"with full mask completion\" for formal writing.\n\n### **References**\n[1]  Ou et al., \"Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data\"\n\n[2] Arriola et al., \"Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models\"\n\n[3] Fathi et al., \"Unifying Autoregressive and Diffusion-Based Sequence Generation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PGWjUzUHJs", "reviewer": "Reviewer_17D4", "summary": "The authors trained a 7B diffusion large language model (dLLM), DiffuCoder, on 130B tokens of code and analyzed its decoding behavior on the “AR-ness” of the its sampling process. The authors’ contributions are two-fold: 1. proposed a global and a local AR-ness metric for dLLMs and used the metrics to analyze the sampling process; 2. proposed coupled-GRPO, a RL technique that adapts GRPO of autoregressive LLMs to dLLMs.", "questions": "See Weaknesses for major concerns.\n\nMiscellaneous:\nThe figure legends overlap with the plots, which makes them difficult to read. Please consider reformatting the figures to improve clarity and presentation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed AR-ness metrics are intriguing and provide insightful analysis of how and when discrete diffusion models exhibit autoregressive-like behavior. While some obervations may seem intuitive, concluding them in a formal way is a meaningful contribution.\n2. The introduction of Coupled-GRPO is a simple yet effective technique for adapting GRPO to discrete diffusion models. Table 1 presents a comprehensive benchmark across various LLMs and dLLMs, showing that DiffuCoder clearly outperforms other diffusion-based counterparts, and that DiffuCoder + Coupled-GRPO further improves upon the base model. I also agree that approaches which make dLLMs overly autoregressive undermine their purpose, so it is encouraging to see these results.", "weaknesses": "The two main contributions of the paper are valuable in their own right, but I find it difficult to fully understand how they connect as part of a unified method. While the AR-ness metrics are conceptually interesting and insightful, it remains unclear how they relate to or motivate the proposed Coupled-GRPO. Specifically, how does Coupled-GRPO address or influence the behavior measured by the AR-ness metrics? There is only a brief discussion around line 468 and a few related observations in Appendix D.4, but it would be helpful if the authors could provide a clearer explanation or reference that explicitly connects how Coupled-GRPO contributes to improving non-AR behavior in diffusion models.\n\nFor Coupled-GRPO, the method pairs timesteps whose indices sum to T, the total number of diffusion steps. This design implicitly assumes a symmetric noising schedule, where the masking ratios at t and T−t are approximately complementary. It would be helpful for the authors to clarify whether this pairing choice is arbitrary or theoretically justified, and whether they have explored or evaluated alternative coupling strategies under non-symmetric schedules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)