DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation

## Abstract

Abstract Diffusion large language models (dLLMs) are compelling alternatives to autoregressive (AR) models because their denoising models operate over the entire sequence. The global planning and iterative refinement features of dLLMs are particularly useful for code generation.
However, current training and inference mechanisms for dLLMs in coding are still under-explored.
To demystify the decoding behavior of dLLMs and unlock their potential for coding, we systematically investigate their denoising processes and reinforcement learning (RL) methods.
We train a 7B dLLM, DiffuCoder , on 130B tokens of code.
Using this model as a testbed, we analyze its decoding behavior, revealing how it differs from that of AR models: (1) dLLMs can decide how causal their generation should be without relying on semi-AR decoding, and (2) increasing the sampling temperature diversifies not only token choices but also their generation order. This diversity creates a rich search space for RL rollouts.
For RL training, to reduce the variance of token log-likelihood estimates and maintain training efficiency, we propose coupled-GRPO , a novel sampling scheme that constructs complementary mask noise for completions used in training.
In our experiments, coupled-GRPO significantly improves DiffuCoder’s performance on code generation benchmarks (+4.4% on EvalPlus) and reduces reliance on AR bias during decoding.
Our work provides deeper insight into the machinery of dLLM generation and offers an effective, diffusion-native RL training framework. https://github.com/apple/ml-diffucoder

## 1 Introduction

Large language models (LLMs) have revolutionized natural language processing, achieving remarkable results across tasks from dialogue to code generation (Touvron et al., 2023; OpenAI, 2023).
While these successes are built predominantly on the autoregressive (AR) paradigm, masked diffusion models (MDMs) have recently emerged as a compelling alternative (Zheng et al., 2024; Shi et al., 2024; Sahoo et al., 2024), and are further scaled to diffusion LLMs (dLLMs) like LLaDA (Nie et al., 2024) and Dream (Ye et al., 2025), which achieve performance on par with similarly sized AR LLMs.
Rather than generating left-to-right, MDMs iteratively refine the entire sequence in parallel, which allows for global planning of content (Ye et al., 2024a; Zhang et al., 2023).

Intuitively, code generation aligns well with the dLLM paradigm, as writing code often involves non-sequential back and forth refinement (Xie et al., 2025).
Recent commercial-scale dLLMs, Mercury (Inception Labs et al., 2025) and Gemini (DeepMind, 2025), show that a diffusion-based code generator can rival top AR code models.
However, it remains unclear how open-source dLLMs perform on coding tasks, as their training and inference mechanisms are not yet fully interpreted.
Existing post-training efforts for dLLMs, such as LLaDA1.5 (Zhu et al., 2025) with DPO (Rafailov et al., 2023) training, and d1 (Zhao et al., 2025), MMaDA (Yang et al., 2025) with GRPO (Shao et al., 2024) training, either show marginal gains or rely heavily on semi-AR decoding (i.e., block decoding with a relatively small block size; see Arriola et al., 2025) which deviates from the global planning nature of diffusion.
To address these limitations, we first gain insight into decoding behaviors of dLLMs and then establish a diffusion-native reinforcement learning (RL) methodology.

Our investigation is grounded in the analysis of DiffuCoder, a 7B-scale MDM specialized for code generation (§[3](https://arxiv.org/html/2506.20639v2#S3)), trained on 130B effective tokens (Huang et al., 2024). The model’s performance is competitive with that of AR coders, providing a strong testbed for understanding the behaviors of dLLMs and for developing diffusion-native post-training approaches.

To leverage the benefits of non-autoregressiveness in dLLMs, it is important to understand how non-autoregressive the behavior of current dLLMs actually is. To this end, we introduce local and global *autoregressive-ness* (AR-ness) metrics (§[4.1](https://arxiv.org/html/2506.20639v2#S4.SS1)) to measure how closely their generation follows a left-to-right pattern.
Our analysis reveals that dLLMs exhibit an *entropy sink* phenomenon (§[4.2](https://arxiv.org/html/2506.20639v2#S4.SS2)), which causes a strong causal bias during conditional generation using low-confidence remasking decoding (Chang et al., 2022). We show that DiffuCoder can automatically decide how non-autoregressive it needs to be during decoding. When the sampling temperature is increased from the default $0.2$ to $1.2$, DiffuCoder becomes more flexible in its token generation order, freeing itself from strict left-to-right constraints, as Figure [1](https://arxiv.org/html/2506.20639v2#S0.F1)(a) shows.
Unlike AR models, which primarily diversify token choices at higher temperatures, dLLMs additionally diversify the position of the generated token.
With this increased diversity, DiffuCoder achieves higher pass@10 accuracy by changing the sampling temperature from $0.2$ to $1.2$ in our experiments (§[4.3](https://arxiv.org/html/2506.20639v2#S4.SS3)).
The gain in pass@10 indicates the potential capacity of DiffuCoder, suggesting it can benefit from effective RL training to “elicit out” the most successful rollout samples.

Following this, we tailor GRPO (Shao et al., 2024) for dLLMs. Our design focuses on reducing variance while maintaining efficiency in Monte Carlo estimations of token likelihoods. Specifically, we propose coupled-GRPO, which employs a novel coupled-sampling scheme. In detail, it adds paired complementary mask noise to the completion sequences generated by the model at a temperature of $1.2$. Unlike previous approaches (Zhao et al., 2025), our method does not rely on semi-AR decoding and it further improves the instruct-tuned model. After coupled-GRPO training, the model exhibits a stronger non-AR generation pattern, as inferred from Figure [1](https://arxiv.org/html/2506.20639v2#S0.F1)(c).

In summary, our contributions are as follows.

- •
We introduce a 7B dLLM for code generation, providing a foundation for developing diffusion-native training methods (§[3](https://arxiv.org/html/2506.20639v2#S3)).
- •
We introduce local and global *AR-ness* metrics to demystify the decoding patterns of dLLMs and track how AR-ness evolves across different training stages (§[4.2](https://arxiv.org/html/2506.20639v2#S4.SS2)). Our analysis reveals that higher sampling temperatures encourage more parallel, non-AR generation (§[4.3](https://arxiv.org/html/2506.20639v2#S4.SS3)).
- •
We design coupled-GRPO, an RL algorithm for dLLMs that avoids semi-AR decoding by using a novel coupled-sampling scheme for efficient and accurate policy gradient estimation (§[5](https://arxiv.org/html/2506.20639v2#S5)). We theoretically prove the variance reduction of coupled-GRPO using antithetic variates.
- •
Experimentally, coupled-GRPO significantly improves DiffuCoder’s performance, boosting its EvalPlus score by 4.4% with training on only 21K samples and demonstrating the effectiveness of RL aligned with diffusion principles.

## 2 Preliminaries and Notations

### 2.1 Mask Diffusion Models

In diffusion models (Ho et al., 2020; Song et al., 2021), the forward process $q(\bm{x}_{1:T}|\bm{x}_{0})=\prod_{t=1}^{T}q(\bm{x}_{t}|\bm{x}_{t-1})$ gradually corrupts data $\bm{x}_{0}\sim p_{data}(\bm{x}_{0})$ into noisy variables $\bm{x}_{1:T}$. The backward process models the joint probability as $p_{\theta}(\bm{x}_{0:T})=p_{\theta}(\bm{x}_{T})\prod_{t=1}^{T}p_{\theta}(\bm{x
}_{t-1}|\bm{x}_{t})$, denoising $\bm{x}_{t}$ to reconstruct $\bm{x}_{0}$.
Discrete diffusion models (Hoogeboom et al., 2021; Zheng et al., 2024) define the forward process with a categorical distribution $q(\bm{x}_{t}|\bm{x}_{t-1})=\text{Cat}(\bm{x}_{t};\bm{Q}_{t}^{\top}\bm{x}_{t-1})$, where $\bm{x}_{t}\in\{0,1\}^{K}$ is a one-hot vector with vocabulary size $K$, and $\bm{Q}_{t}\in[0,1]^{K\times K}$ is the transition matrix where $[\bm{Q}_{t}]_{ij}$ represents the probability of transition from state $i$ to $j$. For absorbing discrete diffusion (Austin et al., 2021a), $\bm{Q}_{t}=(1-\beta_{t})I+\beta_{t}\mathbf{1}\bm{m}^{\top}$, where $\mathbf{1}$ is an all-one vector of size $K$ and $\bm{m}$ is the one-hot encoding of a special [MASK] token.
The parameters $\theta$ are learned by minimizing the negative log-likelihood of $\bm{x}_{0}$ through the evidence lower bound (ELBO). For continuous time modeling (Shi et al., 2024; Sahoo et al., 2024), the discrete timesteps $t=\{1\dots T\}$ are scaled to a mask ratio within $[0,1]$, yielding the final ELBO at a sampled time $t$ as a weighted cross-entropy loss:

$$ $\mathcal{L}_{t}^{1:N}=\frac{1}{t}\mathbb{E}_{q(\bm{x}_{t}|\bm{x}_{0})}\left[- \sum_{n=1}^{N}\delta_{\bm{x}_{t}^{n},\bm{m}}(\bm{x}_{0}^{n})^{\top}\log f_{ \theta}(\bm{x}_{t}^{1:N})_{n}\right],$ (1) $$

where $\delta_{\bm{x}_{t}^{n},\bm{m}}$ is the indicator function, and $f_{\theta}(\bm{x}_{t}^{1:N})_{n}$ represents the logits for the $n$-th token given the $N$-length input sequence.

### 2.2 Markov Decision Process

DDPO (Black et al., 2024) and DPPO (Ren et al., 2025) reinterpret the denoising diffusion process as a Markov Decision Process (MDP). An MDP is a tuple $\mathcal{M}_{env}=(\mathcal{S},\mathcal{A},P_{0},P,R)$ with a state space $\mathcal{S}$, an action space $\mathcal{A}$, an initial state distribution $P_{0}$, transition probabilities $P$, and a reward function $R$. The probability of transitioning to state $s_{t+1}$ is $P(s_{t+1}\mid s_{t},a_{t})$ after taking an action $a_{t}\sim\pi_{\theta}(a_{t}\mid s_{t})$ and receiving a reward $R(s_{t},a_{t})$. The goal is to maximize the expected return $\mathcal{J}(\pi_{\theta})=\mathbb{E}_{\pi_{\theta}}\Bigl{[}\sum_{t=0}^{T}
\gamma(t)\,R(s_{t},a_{t})\Bigr{]}$ by using the policy gradient method (Williams, 1992):

$$ $\nabla_{\theta}\mathcal{J}(\pi_{\theta})=\mathbb{E}_{\pi_{\theta}}\bigl{[}\sum _{t=0}^{T}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\,r_{t}(s_{t},a_{t}) \bigr{]};\quad r_{t}(s_{t},a_{t})=\sum_{\tau\geq t}\gamma(\tau)\,R(s_{\tau},a_ {\tau}).$ (2) $$

In a masked diffusion model for conditional generation, we set the state to $s_{t}=(c,t,\bm{x}_{t})$ (where $c$ is the condition) and the action to $a_{t}=\bm{x}_{t-1}$, so that $\pi_{\theta}(a_{t}|s_{t})=p_{\theta}(\bm{x}_{t-1}|\bm{x}_{t},c)$. We sample $c\in\mathcal{C}$ and trajectories of $\bm{x}_{t}$. The reward is defined as $R(s_{0},a_{0})=r(\bm{x}_{0},c)$ at the final denoising step because a fine-grained progressive reward is usually hard to quantify, especially in intermediate diffusion steps. Under this setting, the policy gradient becomes:
$\nabla_{\theta}\mathcal{J}=\mathbb{E}\bigl{[}\sum_{t=0}^{T}\nabla_{\theta}\log
p
_{\theta}(\bm{x}_{t-1}|\bm{x}_{t},c)\,r(\bm{x}_{0},c)\bigr{]}$.

### 2.3 Group Relative Policy Optimization

GRPO (Shao et al., 2024) simplifies PPO (Schulman et al., 2017). It samples a group of outputs $\{o_{i}\}_{i=1}^{G}$ from the old policy $\pi_{\theta_{\text{old}}}$ under a given condition $c$, estimates the value baseline by averaging rewards within the group, and defines the relative advantage for each output as $A_{i}=r(o_{i})-\frac{1}{G}\sum_{j=1}^{G}r(o_{j})$. For token $1\leq k\leq|o_{i}|$, we denote $\rho_{i}^{k}=\frac{\pi_{\theta}(o_{i}^{k}\mid c,o_{i}^{<k})}{\pi_{\text{old}}(
o_{i}^{k}\mid c,o_{i}^{<k})}$ as the token-level importance ratio.
The GRPO loss applies a PPO-style clipping to $\rho_{i}^{k}$, incorporating a KL-penalty to keep $\pi_{\theta}$ close to a reference policy $\pi_{\text{ref}}$, and maximizes the following surrogate objective:

$$ $\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{o_{i}\sim\pi_{\theta_{\text{old} }}}\Bigl{[}\sum_{i=1}^{G}\sum_{k=1}^{|o_{i}|}\min\bigl{(}\rho_{i}^{k}A_{i}, \mathrm{clip}(\rho_{i}^{k},1-\varepsilon,1+\varepsilon)A_{i}\bigr{)}-\beta\,D_ {\mathrm{KL}}\bigl{(}\pi_{\theta}\|\pi_{\text{ref}}\bigr{)}\Bigr{]}.$ (3) $$

By estimating the group mean via Monte Carlo estimation, GRPO avoids training a separate value function while fitting neatly into the MDP framework. As the diffusion process can be viewed as an MDP, the GRPO loss can be applied to MDMs by combining Eq. ([2](https://arxiv.org/html/2506.20639v2#S2.E2)) and Eq. ([3](https://arxiv.org/html/2506.20639v2#S2.E3)).

## 3 DiffuCoder

To build a strong base model for RL training, we follow common practices in LLM training and train our DiffuCoder model on a large-scale corpus (Lozhkov et al., 2024) with multiple training stages; the pipeline is illustrated in Figure [2](https://arxiv.org/html/2506.20639v2#S3.F2).
We first conduct adaptation pre-training similar to the process followed by Dream (Ye et al., 2025).
Mid-training (Wang et al., 2025) connects the pre-training and post-training stages, plays the role of an annealing phase as in OpenCoder (Huang et al., 2024), and has proven effective.
This is followed by an instruction tuning stage to enhance the model’s capability of following instructions. Finally, for post-training, we employ a novel coupled-GRPO method (introduced in §[5](https://arxiv.org/html/2506.20639v2#S5)) to further enhance the model’s pass@1 coding capabilities.

Figure: Figure 2: Pipeline of DiffuCoder training stages and an illustration of the coupled-GRPO algorithm. We sample complementary mask matrices for the same batch, so the coupling probability matrices can be merged into one full matrix. Coupled sampling reduces probability estimation variance while maintaining full token coverage, where each token is sampled exactly the same number of times.
Refer to caption: x2.png

##### Training

We adapt our model from Qwen-2.5-Coder (Hui et al., 2024) as the base model to perform continual pre-training using the adaptation approach from Gong et al. (2025).
During this pre-training, we use a 400B-token code pre-training corpus from
RefineCode (Huang et al., 2024)
and Stackv2 (Lozhkov et al., 2024). We adopt the code-to-text ratio suggested in Qwen-2.5-Coder and OpenCoder (Huang et al., 2024).
We use 16B tokens of annealing code data during mid-training and 436K SFT samples during instruction tuning, both from OpenCoder (Huang et al., 2024). For RL training, we select 21K hard samples from Acecoder-87K (Zeng et al., 2025) with verifiable test cases. We build our post-training method upon the Open-R1(^1^11[https://github.com/huggingface/open-r1](https://github.com/huggingface/open-r1)) codebase. All experiments are conducted on 8 to 10 nodes, each with 8 H100 GPUs.
We observed that training with 700B tokens in Stage 1 led to worse performance than using only 65B tokens on the downstream validation sets. Therefore, we perform early stopping for Stage 1, training on 65B tokens. In Stage 2, we train for 4 epochs, totaling 65B tokens with repeats, as the mid-training data is less noisy.
More details are listed in Appx. [B.1](https://arxiv.org/html/2506.20639v2#A2.SS1).

##### Evaluation

Our evaluation environments are built on three code benchmarks and their variants: HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021b), EvalPlus (HumanEval+ and MBPP+; Liu et al. 2023), and BigCodeBench (Zhuo et al., 2024) with full and hard subsets in completion (C) and instruction (I) query types.
These benchmarks, with Python as the coding language, provide a diverse set of coding tasks for assessing code correctness and quality.

##### Performance

We compare our models with AR code LLMs, including Qwen2.5-Coder-7B (Hui et al., 2024), OpenCoder-8B (Huang et al., 2024); general dLLMs, such as Dream-7B (Ye et al., 2025) and LLaDA-8B (Zhu et al., 2025); and commercial models, such as GPT-4o(^2^22[https://openai.com/index/gpt-4o-system-card/](https://openai.com/index/gpt-4o-system-card/)), Mercury(^3^33[https://chat.inceptionlabs.ai/](https://chat.inceptionlabs.ai/)), and Gemini Diffusion(^4^44[https://deepmind.google/models/gemini-diffusion/](https://deepmind.google/models/gemini-diffusion/)).
As shown in Table [1](https://arxiv.org/html/2506.20639v2#S3.T1), DiffuCoder, after being continually trained on 130B code tokens (Stages 1 and 2), achieves performance on par with Qwen2.5-Coder and OpenCoder.
However, all dLLMs show only marginal improvement over their base models after instruction tuning, especially when compared to Qwen2.5-Coder+SFT, which achieves large improvements from being instruct-tuned on the same data.

This improvement gap between AR and dLLMs at the instruction-tuning stage motivates us to explore RL-based post-training methods (§[5](https://arxiv.org/html/2506.20639v2#S5)).
Previous RL approaches for diffusion models (Zhao et al., 2025; Yang et al., 2025) rely heavily on semi-AR decoding, which deviates from diffusion’s global nature.
To design RL methods aligned with diffusion’s non-autoregressive principle, we first analyze the intrinsic decoding behavior of dLLMs and their differences from AR models in §[4](https://arxiv.org/html/2506.20639v2#S4).

**Table 1: Benchmark coding capacities of LLMs and dLLMs in 7/8B scale. Different shaded colors indicates different generation paradigms (pink for AR, yellow for diffusion). ^∗ denotes that the results are collected from public reports instead of evaluating by ourselves. In our evaluation settings, we compute EvalPlus as the average of HE+ and MBPP+. We show the absolute score change (±) of each instruct model relative to its base. Scores are bolded when our model outperforms LLMs specialized for code (excluding LLaDA or Dream).**
| Model | HumanEval | MBPP | EvalPlus | BigCodeBench (C) | Avg. |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| - | Plus | - | Plus | Full | Hard |  |  |  |
| Base Models |  |  |  |  |  |  |  |  |
| Qwen2.5-Coder | 61.6 | 51.8 | 75.9 | 61.4 | 56.6 | 46.1 | 16.2 | 52.2 |
| OpenCoder^∗ | 66.5 | 63.4 | 79.9 | 70.4 | 66.9 | 40.5 | 9.5 | 55.0 |
| LLaDA | 35.4 | 30.5 | 50.1 | 42.1 | 36.3 | 18.9 | 4.1 | 30.2 |
| Dream | 56.7 | 50.0 | 68.7 | 57.4 | 53.7 | 23.6 | 4.1 | 43.4 |
| DiffuCoder | 67.1 | 60.4 | 74.2 | 60.9 | 60.6 | 40.2 | 12.8 | 52.6 |
| Instruct Models |  |  |  |  |  |  |  |  |
| Qwen2.5-Coder-Instruct | 90.2 | 85.4 | 83.9 | 72.0 | 78.7 | 50.7 | 21.6 | 67.3 |
| Qwen2.5-Coder+SFT | 82.9+21.3 | 75.6+23.8 | 80.1+4.2 | 66.1+4.7 | 70.9+14.3 | 46.9+0.8 | 16.2–0.0 | 61.3+9.1 |
| OpenCoder-Instruct^∗ | 83.5+17.0 | 78.7+15.3 | 79.1–0.8 | 69.0–1.4 | 73.9+7.0 | 40.3–0.2 | 16.9+7.4 | 61.3+6.3 |
| LLaDA-Instruct | 35.4–0.0 | 31.7+1.2 | 31.5–18.6 | 28.6–13.5 | 30.2–6.1 | 16.5–2.4 | 2.7–1.4 | 24.4–5.8 |
| Dream-Instruct | 57.9+1.2 | 53.7+3.7 | 68.3–0.4 | 56.1–1.3 | 54.9+1.2 | 10.6–13.0 | 0.7–3.4 | 41.2–2.2 |
| DiffuCoder-Instruct | 72.0+4.9 | 65.2+4.8 | 75.1+0.9 | 61.9+1.0 | 63.6+3.0 | 35.7–4.5 | 12.2–0.6 | 53.7+1.1 |
| + coupled-GRPO | 73.2+6.1 | 68.3+7.9 | 78.6+4.4 | 67.5+6.6 | 67.9+7.3 | 40.4+0.2 | 10.8–2.0 | 56.5+3.9 |
| Commercial Models |  |  |  |  |  |  |  |  |
| GPT 4o^∗ | 90.2 | – | 82.2 | – | 82.4 | 49.9 | – | – |
| Mercury^∗ | 90.0 | – | 77.1 | – | 80.4 | 45.5 | – | – |
| Gemini Diffusion^∗ | 89.6 | – | 76.0 | – | – | 45.4 | – | – |

## 4 Understanding Mask Diffusion Models based on DiffuCoder

Current dLLMs such as LLaDA (Nie et al., 2024) and Dream rely on low-confidence remasking decoding strategies (Chang et al., 2022), and LLaDA achieves improved performance on certain tasks using semi-AR decoding methods (i.e., block diffusion decoding; see Arriola et al., 2025). Another common practice among dLLMs is to set the number of diffusion timesteps equal to the sequence length, effectively resorting to token-by-token generation to enhance performance. Given this context, we introduce local and global *autoregressive-ness* (AR-ness) metrics to systematically investigate the decoding order of dLLMs. Specifically, our analysis aims to demystify: (1) how dLLMs’ decoding patterns differ from those of AR models; (2) how data modality (e.g., code or math) influences model behavior; and (3) how AR-ness evolves across different training stages.

### 4.1 Autoregressive-ness in Generation

Figure: Figure 3: Left: Local and global AR-ness across different models and data modalities. Adapted dLLM refers to Dream for the math task and DiffuCoder (Stage 1 trained with 65B tokens) for the code task. Right: (a) Confidence score for each position in the dLLM’s first forward decoding step. (b) Local AR-ness@k: the fraction of decoding steps where the newly unmasked token, together with the $k$ immediately preceding predicted tokens, forms a strictly increasing consecutive sequence, at $k=1$ for next-token prediction. Global AR-ness@k: the fraction of decoding steps where the model chooses to unmask one of the earliest $k$ positions among all remaining masked tokens.
Refer to caption: x3.png

Figure: Figure 4: AR-ness drifts on different training stages. Left: adaptation pre-training stage and mid-training stage. Right: instruction tuning and RL post-training stage.
Refer to caption: x5.png

In standard AR decoding, the model generates tokens in strict left‐to‐right order, ensuring strong sequential coherence.
However, diffusion-based decoding may choose to recover [MASK] out of order.
Therefore, we introduce two metrics to quantify how the unmasking schedule of a diffusion model resembles an AR pattern, including (i) “next token” pattern and (ii) “left first” pattern.

##### Local: Consecutive Next‐Token Prediction

*Local AR-ness@$k$* is computed by the ratio of predicted sequence matching the pattern of next token prediction within range $k$.
If all tokens in $k$-length span are immediate successors of the previously generated token, we count this span as casual.
Local AR-ness decays as $k$ grows, as it is harder to maintain longer consecutive spans.

##### Global: Earliest Mask Selection

In step $t$, if the predicted token lies in the first $k$ masked positions, the global AR-ness is scored. *Global AR-ness@$k$* is the averaged ratio for each $t$, and it measures the tendency to always unmask the earliest remaining token, capturing a left-to-right filling strategy.
This ratio grows with $k$, since the criterion becomes easier to satisfy as more early positions are allowed.
For the two metrics, the higher value indicates that the generation is more autoregressive.
Detailed formulations are listed in Appx. [A.2](https://arxiv.org/html/2506.20639v2#A1.SS2).

### 4.2 Decoding Analysis

We conduct AR-ness comparisons during conditional generation between: (1) different dLLMs, including LLaDA trained from scratch and Dream or DiffuCoder adapted from AR LLMs; (2) different data modalities, including math and code; and (3) different training stages of DiffuCoder.
All inference settings are based on the low-confidence remasking strategy (Chang et al., 2022) with the same sequence length and diffusion timesteps ($512$). Math is evaluated on GSM8K using 8-shots (Cobbe et al., 2021), and code is evaluated using zero-shot HumanEval.

##### How do dLLMs decode differently from AR models?

For AR decoding, both local and global AR-ness are identically equal to $1$ (i.e., 100% AR).
In contrast, as illustrated in Figure [3](https://arxiv.org/html/2506.20639v2#S4.F3), dLLMs do not always decode in a purely AR manner.
A significant fraction of tokens in dLLM decoding are recovered from neither the leftmost masked token nor the next token.
This observation indicates that dLLMs adopt a more flexible decoding order compared to regular AR models.
Nevertheless, both local and global AR-ness are closer to $1$ than 0 0, demonstrating that text data inherently exhibit some AR structure, which diffusion-based LMs, regardless of whether they are trained from scratch or adapted from AR models, naturally capture.
Empirically, adapted dLLMs tend to exhibit stronger AR-ness than those trained from scratch.
This is because they inherit the left-to-right token dependencies from the original AR training.
Lower AR-ness opens up additional opportunities for parallel generation by breaking this dependency (Appx. [11](https://arxiv.org/html/2506.20639v2#A3.F11)).
Higher AR-ness can also be beneficial; for example, LLaDA often needs to resort to semi-AR (block-wise decoding; Arriola et al., 2025) generation to achieve higher overall performance.
In that setting, the block decoder explicitly reintroduces causal bias into the generation process. In DiffuCoder, we argue that the model can decide how causal it is during generation by itself.

##### How do different data modalities affect the decoding paradigm?

According to Figure [3](https://arxiv.org/html/2506.20639v2#S4.F3), although math and code decoding exhibit different degrees of local AR-ness, a consistent finding is that code generation has a lower mean and higher variance in global AR-ness.
This indicates that when generating code, the model tends to produce later tokens first, leaving some early masked tokens un-recovered until much later (Appx. [C.2](https://arxiv.org/html/2506.20639v2#A3.SS2)).
The reason might be that mathematical text is essentially sequential and usually requires left-to-right computation, whereas code has an intrinsic structure.
Consequently, the model often plans token generation more globally, much like a programmer jumping back and forth through code to refine a code implementation.

##### How does AR-ness change at different training stages?

In Figure [4](https://arxiv.org/html/2506.20639v2#S4.F4) (Stage 1), after training with 65B tokens, we already observe a relatively low AR-ness.
However, when we extend the training to 700B tokens, AR-ness increases while overall performance drops (see the table in Appx. [B.1](https://arxiv.org/html/2506.20639v2#A2.SS1)).
We suspect that the quality of the pre-training data limits performance.
Consequently, we choose the Stage 1 65B model as the starting point for Stage 2.
During mid-training (Stage 2) and instruction tuning (Stage 3), on the first epoch of high-quality data, the model learns a high causal bias.
As it sees more tokens, however, task performance improves (Appx. [B.1](https://arxiv.org/html/2506.20639v2#A2.SS1)), while the measured AR-ness starts to decline.
This pattern implies that after the first epoch, dLLMs begin to capture dependencies beyond a pure AR order.
After GRPO training, the model’s global AR-ness also decreases and meanwhile shows less of a performance drop when decoding in half as many steps (Figure [1](https://arxiv.org/html/2506.20639v2#S0.F1)(c); Appx. [11](https://arxiv.org/html/2506.20639v2#A3.F11)).

##### Entropy Sink

When dLLMs perform conditional generation, the first diffusion step starts with a fully masked completion given a prefix prompt and attempts to recover the completion sequence.
At this step, we record the confidence score of each recovered token in Figure [3](https://arxiv.org/html/2506.20639v2#S4.F3)(a). Appx. [C.1](https://arxiv.org/html/2506.20639v2#A3.SS1) also lists the entropy heatmap across all decoding timesteps.
The default decoding algorithm from LLaDA and Dream selects the token with the highest confidence while remasking the rest.
LLaDA uses log probabilities while Dream uses negative entropy to measure confidence, where a larger value indicates that the model is highly confident about that token.

Remarkably, the resulting distribution displays a characteristic “L”-shaped pattern.
We refer to this phenomenon as the *entropy sink*.
We hypothesize that the entropy sink arises because the intrinsic nature of text biases the model toward tokens that lie immediately to the right of the given prefix: those positions receive stronger positional signals and closer context, leading the model to assign them disproportionately high confidence. This phenomenon may be related to the cause of the attention sink (Gu et al., 2024; Xiao et al., 2024), but its underlying cause requires further analysis and verification.
This entropy bias toward locally adjacent tokens explains why dLLMs still maintain a non-trivial level of AR-ness.

Figure: Figure 5: Affects of different sampling temperatures. Left: For base model and instruct model, changing temperature affects the AR-ness on HumanEval. Right: pass@k curves are different for different temperatures, where triangles refer to score for plus version of each task.
Refer to caption: x7.png

### 4.3 Generation Diversity

Figure: Figure 6: Pass@$k$ scores for different models and temperatures.
Refer to caption: x9.png

Post-training studies on AR LLMs (Yue et al., 2025) show that an RL model’s reasoning paths are bounded by the base model’s pass@k sampling capabilities.
Therefore, we examine generation diversity with pass@$k$ accuracy in dLLMs.
As Figure [5](https://arxiv.org/html/2506.20639v2#S4.F5) (right) and Figure [6](https://arxiv.org/html/2506.20639v2#S4.F6) illustrate, for both the base and instruct versions of DiffuCoder, a low temperature yields high pass@1 but little growth in pass@$k$, indicating that the samples lack diversity.
By increasing the temperature to a suitable range (e.g., $1.0$ to $1.2$), pass@$k$ rises significantly, revealing a latent capability in the model.
In many RL settings (Bercovich et al., 2025; Liu et al., 2025a), the model must be able to sample diverse responses during rollouts before RL can reinforce pass@1 accuracy.
The promising pass@$k$ curves of DiffuCoder indicate substantial room for improvement through RL, motivating the design of our coupled-GRPO algorithm (§[5](https://arxiv.org/html/2506.20639v2#S5)).
Moreover, a higher temperature also substantially lowers AR-ness, as shown in Figure [5](https://arxiv.org/html/2506.20639v2#S4.F5) (left) and Figure [1](https://arxiv.org/html/2506.20639v2#S0.F1)(a), meaning the model generates tokens in a more random order, which operates differently from AR models.
In AR models, temperature only affects token selection, while in dLLMs it influences both token selection and their generated order. A visualization of the decoding process from real samples is provided in Appx. [C.2](https://arxiv.org/html/2506.20639v2#A3.SS2).

## 5 Coupled-GRPO

Reinforcement learning (RL) and GRPO (Shao et al., 2024) have proven critical for enhancing AR models (Bercovich et al., 2025; Shao et al., 2025), but their application to dLLMs is less explored.
As discussed in §[2.3](https://arxiv.org/html/2506.20639v2#S2.SS3), formulating the mask diffusion process as a Markov Decision Process allows for a policy optimization approach akin to PPO (Schulman et al., 2017).
To facilitate integration with GRPO (Shao et al., 2024), the approximation of token probabilities within diffusion models is necessary.
Current masked diffusion models rely on Monte Carlo sampling (Zhu et al., 2025; Shi et al., 2024) for log-probability estimation.
Specifically, the negative log-likelihood (NLL) is bounded by the ELBO, i.e., $\mathcal{P}_{i}=\mathbb{E}_{t\sim 1,...T;\bm{x}_{t}\sim q(\bm{x}_{t}|\bm{x}_{0
})}[\mathcal{L}_{t}(\bm{x}_{t})]$, where $\mathcal{L}_{t}$ is the cross-entropy loss introduced in Eq. ([1](https://arxiv.org/html/2506.20639v2#S2.E1)).
However, Monte Carlo sampling introduces significant overhead during the training of GRPO, as highlighted by d1 (Zhao et al., 2025).

##### Baseline methods

To overcome this, d1 chooses to mask all completion tokens and perform a single forward pass to compute each token’s probability, which is equivalent to sampling once at diffusion step $t=T$; we call this sequence of log-probabilities $\mathcal{P}_{i}$, with each element being $\log\pi_{\theta}(o_{i}^{k}|c,o_{i}^{k}=\bm{m})$.
Thus, GRPO’s update in Eq. ([3](https://arxiv.org/html/2506.20639v2#S2.E3)) uses the ratio $\rho_{i}^{k}=\frac{\pi_{\theta}(o_{i}^{k}|c,o_{i}^{k}=\bm{m})}{\pi_{\theta_{
\text{old}}}(o_{i}^{k}|c,o_{i}^{k}=\bm{m})}$.
d1 also randomly masks $p=15\%$ of the condition tokens to increase sampling diversity, which, in practice, makes the completion-token probability estimates unreliable.
In our code experiments, masking condition tokens does not yield a stable reward improvement (Figure [7](https://arxiv.org/html/2506.20639v2#S5.F7)), probably because code tasks demand higher token-level generation accuracy than math tasks.
As a result, we revert to the *completion full mask* version ($p=0\%$) and use $\mathcal{P}_{t=T}$ as our baseline.
Even so, this baseline is biased: as shown in our *entropy sink* analysis (§[4.2](https://arxiv.org/html/2506.20639v2#S4.SS2)), high-entropy tokens tend to lie on the left side, so RL training still ends up updating early tokens more aggressively.

##### Coupled-GRPO

In the $\mathcal{L}_{t}$ computation, only the loss for positions involving masked tokens is counted, which introduces inefficiency and variance when sampling times are limited.
To improve probability estimation while still covering every token, we introduce a coupled-sampling scheme (Figure [2](https://arxiv.org/html/2506.20639v2#S3.F2)).
Concretely, we pick $\lambda$ timestep pairs $(t,\hat{t})$ with $t+\hat{t}=T$, then sample two complementary completion masks: each mask hides part of the tokens, and together they cover all completion tokens.
In other words, every token is unmasked in exactly one of the two forward passes.
This design guarantees that (1) each token’s log-probability is computed at least once (giving each token a non-zero learning signal) and (2) these log-probability estimations are more accurate because every token is evaluated under a realistic partial-masking context rather than always being masked, and we have $2\lambda$ additional samples compared to the baseline. Combining Eqs. ([1](https://arxiv.org/html/2506.20639v2#S2.E1)), ([2](https://arxiv.org/html/2506.20639v2#S2.E2)), and ([3](https://arxiv.org/html/2506.20639v2#S2.E3)), we have:

$$ $\displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}\Bigl{[}\sum_{i=1}^{G }\sum_{k=1}^{|o_{i}|}\min\Bigl{(}\frac{\pi_{\theta}(o_{i}^{k}|c,o_{i,t<T}^{k}) }{\pi_{\theta_{\text{old}}}(o_{i}^{k}|c,o_{i,t<T}^{k})}A_{i},\mathrm{clip} \bigl{(}\frac{\pi_{\theta}(o_{i}^{k}|c,o_{i,t<T}^{k})}{\pi_{\theta_{\text{old} }}(o_{i}^{k}|c,o_{i,t<T}^{k})},1-\varepsilon,1+\varepsilon\bigr{)}A_{i}\Bigr{) }-\beta D_{\mathrm{KL}}\Bigr{]},$ $\displaystyle\text{with }\log\pi_{\theta}(o_{i}^{k}|c,o_{i,t<T}^{k})=\frac{1}{ \lambda+1}\bigg{[}\sum_{t+\hat{t}=T}^{\lambda}[\mathcal{L}_{t}(\bm{x}_{t})+ \mathcal{L}_{\hat{t}}(\bm{x}_{\hat{t}})]+\mathcal{L}_{T}(\bm{x}_{T})\bigg{]}_{ i}^{k},\;\delta_{\bm{x}_{t},\bm{m}}+\delta_{\bm{x}_{\hat{t}},\bm{m}}=\bm{1}.$ (4) $$

In practice, we choose $\lambda=1$. For a fair comparison, we introduce a de-coupled baseline with the same number of samples but without the complementary constraint.
For advantage score computation, we also consider leave-one-out (LOO) strategies (Ahmadian et al., 2024; Kool et al., 2019) to determine the baseline score: $A_{i}=r(o_{i})-\frac{1}{G-1}\sum_{j\neq i}^{G}r(o_{j})$, which creates an unbiased estimate.
We show that our coupled-sampling scheme can be viewed as an application of the Antithetic Variates (Hammersley & Mauldon, 1956) variance reduction technique in Appx. [A.3](https://arxiv.org/html/2506.20639v2#A1.SS3), where we also list detailed designs for verified rewards, including a code format reward and the execution pass rate over test cases as a correctness reward.

Figure: Figure 7: Reward curves during GRPO training. Left: Comparison between coupled GRPO and d1 baselines. Middle: Decoupled GRPO uses the same number of samplings but with randomly sampled mask noise. Right: Coupled-GRPO is sensitive to the rollout temperature.
Refer to caption: x10.png

**Table 2: Evaluation results for GRPO post-training across multiple benchmarks and models. We report the best results from the sampling temperature set $\{0.2,0.3,0.4\}$.**
|  | HumanEval | MBPP | BigCodeBench (C) | BigCodeBench (I) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | – | Plus | – | Plus | Full | Hard | Full | Hard |
| Qwen2.5-Coder+SFT | 82.9 | 75.6 | 80.1 | 66.1 | 46.9 | 16.2 | 39.5 | 14.9 |
| + GRPO | 80.5–2.4 | 75.0–0.6 | 84.4+4.3 | 72.8+6.7 | 49.7+2.8 | 16.20.0 | 40.0+0.5 | 10.2–4.7 |
| DiffuCoder-Instruct | 72.0 | 65.2 | 75.1 | 61.9 | 35.7 | 12.2 | 34.0 | 8.8 |
| + coupled GRPO | 73.2+1.2 | 68.3+3.1 | 78.6+3.5 | 67.5+5.6 | 40.4+4.7 | 10.8–1.4 | 37.5+3.5 | 10.8+2.0 |
| + coupled GRPO (LOO) | 70.7–1.3 | 62.2–3.0 | 79.6+4.5 | 68.5+6.6 | 41.2+5.5 | 13.5+1.3 | 37.6+3.6 | 12.8+4.0 |
| \cdashline1-9[0.8pt/2pt]<br>w. full mask completion | 66.5–5.5 | 59.1–6.1 | 77.0+1.9 | 65.1+3.2 | 38.0+2.3 | 8.8–3.4 | 35.6+1.6 | 13.5+4.7 |
| w. decoupled sampling | 68.9–3.1 | 62.8–2.4 | 78.3+3.2 | 66.4+4.5 | 40.4+4.7 | 10.8–1.4 | 36.5+2.5 | 10.8+2.0 |

##### Experiment Results

Table [2](https://arxiv.org/html/2506.20639v2#S5.T2), together with Figure [7](https://arxiv.org/html/2506.20639v2#S5.F7), demonstrates the effectiveness of coupled-GRPO training.
In contrast, the baseline variants: d1, full-mask completion, and decoupled sampling, exhibit unstable reward learning.
The rollout sampling temperature is also critical: as shown in Figure [6](https://arxiv.org/html/2506.20639v2#S4.F6), DiffuCoder-Instruct attains a higher pass@10 at temperature $1.2$ than at $1.0$, mirroring the trend observed during coupled-GRPO training.
Notably, RL fine-tuning shifts the optimal sampling temperature from $0.2$ to a larger value, such as $0.3$ or $0.4$, during evaluation, indicating that training sharpens the per-token distribution.
This finding aligns with recent results for AR LLMs (Cui et al., 2025; Liu et al., 2025a; Agarwal et al., 2025), suggesting that the approaches proposed in these works may also be generalizable to dLLMs.
Finally, at the new optimal temperature, global decoding AR-ness decreases, as shown in Figure [4](https://arxiv.org/html/2506.20639v2#S4.F4) (right). A more interesting finding is that, as shown in Figure [1](https://arxiv.org/html/2506.20639v2#S0.F1)(c), when we use $0.5\times$ fewer decoding steps (equivalent to a $2\times$ generation speedup), training with coupled-GRPO results in a smaller performance drop compared to the model before training, suggesting that AR-ness is reduced and parallelism is increased (Wu et al., 2025). Detailed discussions are provided in Appx. [11](https://arxiv.org/html/2506.20639v2#A3.F11) and Appx. [D](https://arxiv.org/html/2506.20639v2#A4).

## 6 Related Work

##### Text Diffusion Models

Early explorations of text diffusion models were based on a continuous space (Li et al., 2022; Gong et al., 2023b; Chen et al., 2023). Subsequently, discrete diffusion models (Hoogeboom et al., 2021; Austin et al., 2021a) directly introduced discrete noise to accommodate the discrete nature of text, demonstrating significant potential (Zheng et al., 2024; Lou et al., 2024) and were further developed into mask diffusion models (Shi et al., 2024; Ou et al., 2024; Sahoo et al., 2024).
Recent work has explored scaling these models significantly, with DiffuLLaMA (Gong et al., 2025) being adapted from pretrained AR LLMs, and LLaDA (Nie et al., 2024) and Dream (Ye et al., 2025) being the first open-source diffusion LLMs to achieve performance comparable to AR LLMs. Block diffusion (Arriola et al., 2025) proposes a hybrid approach that applies diffusion within each block (Han et al., 2023), serving as a midpoint between autoregressive and diffusion models. Multimodal models such as LaViDa (Li et al., 2025), MMaDA (Yang et al., 2025), and Dimple (Yu et al., 2025) combine text diffusion models with vision models. Liu et al. (2025b); Hu et al. (2025); Ma et al. (2025); Wu et al. (2025); Sahoo et al. (2025) introduce caching and parallel decoding algorithms for dLLMs, significantly improving inference efficiency.

##### Code Generation

Code generation is a crucial domain for LLMs (Roziere et al., 2023; Sun et al., 2024), exemplified by state-of-the-art open-source models like Qwen-2.5-Coder (Hui et al., 2024) and OpenCoder (Huang et al., 2024), with wide applications in areas such as coding assistants and agents (Xu et al., 2024).
CodeFusion (Singh et al., 2023) was the first to combine diffusion models with code generation, but it was limited to small-scale models and simple tasks. Recent commercial-scale dLLMs, such as Mercury (Inception Labs et al., 2025) and Gemini (DeepMind, 2025), have demonstrated that diffusion-based code generators can achieve performance comparable to leading autoregressive code models while offering significantly faster generation speeds.

##### Reinforcement Learning

Reinforcement learning with verifiable reward (RLVR) using GRPO (Shao et al., 2024; OpenR1, 2025; Guo et al., 2025; Bercovich et al., 2025) is highly effective in enhancing a language model’s math reasoning (Shao et al., 2025) and code generation abilities (Xie et al., 2025). Wang et al. (2025) show the importance of mid-training during RL scaling. Combining RL and diffusion models, VRPO (Zhu et al., 2025) introduces the efficient sampling algorithm from DPO (Rafailov et al., 2023) for dLLMs.
d1 (Zhao et al., 2025) and MMaDA (Yang et al., 2025) optimize math reasoning in dLLMs using GRPO, but they rely heavily on block diffusion decoding during rollout and evaluation. LLadDou (Huang et al., 2025) trains an additional module to predict the rank score of tokens. For small text diffusion models, Zhang et al. (2025) propose the target concrete score matching framework, and Zekri & Boullé (2025) introduces score entropy policy optimization (SEPO). Earlier, DDPO (Black et al., 2024) and DPPO (Ren et al., 2025) formulated the diffusion process as a Markov Decision Process and performed policy optimization for continuous diffusion models.

## 7 Conclusion

In this work, we present DiffuCoder, a 7B-scale open-source diffusion model for code with strong performance, and its complete training recipe. We also present a comprehensive analysis of dLLMs for code generation.
Our investigation into their decoding patterns reveals fundamental differences from AR models; notably, sampling temperature affects not only token selection but also the generation order, creating rich sample diversity for optimization.
Capitalizing on this, we introduce coupled-GRPO, a reinforcement learning algorithm that respects the non-autoregressive nature of dLLMs.
By using a novel coupled-sampling strategy, our method provides a more accurate likelihood estimation.
Coupled-GRPO significantly boosts DiffuCoder’s performance, demonstrating the effectiveness of RL methods aligned with diffusion principles.
Our work provides the community with a deeper understanding of dLLMs and lays a strong foundation for future explorations of dLLMs in complex reasoning and generation tasks.

#### Acknowledgments

We thank the following researchers for their stimulating discussions and valuable suggestions during the preparation of this manuscript: Shivam Agarwal from the University of Illinois Urbana-Champaign; Wenhao Chai from Princeton University; Richard Bai from Apple; Lin Zheng, Chenxin An and Zhihui Xie from the University of Hong Kong.

## References

- Agarwal et al. (2025)
Shivam Agarwal, Zimin Zhang, Lifan Yuan, Jiawei Han, and Hao Peng.
The unreasonable effectiveness of entropy minimization in llm reasoning.
*ArXiv preprint*, abs/2505.15134, 2025.
URL [https://arxiv.org/abs/2505.15134](https://arxiv.org/abs/2505.15134).
- Ahmadian et al. (2024)
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker.
Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.
*ArXiv preprint*, abs/2402.14740, 2024.
URL [https://arxiv.org/abs/2402.14740](https://arxiv.org/abs/2402.14740).
- An et al. (2025)
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong.
Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025.
URL [https://hkunlp.github.io/blog/2025/Polaris](https://hkunlp.github.io/blog/2025/Polaris).
- Arriola et al. (2025)
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov.
Block diffusion: Interpolating between autoregressive and diffusion language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Austin et al. (2021a)
Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg.
Structured denoising diffusion models in discrete state-spaces.
In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), *Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual*, pp.  17981–17993, 2021a.
URL [https://proceedings.neurips.cc/paper/2021/hash/958c530554f78bcd8e97125b70e6973d-Abstract.html](https://proceedings.neurips.cc/paper/2021/hash/958c530554f78bcd8e97125b70e6973d-Abstract.html).
- Austin et al. (2021b)
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.
Program synthesis with large language models.
*ArXiv preprint*, abs/2108.07732, 2021b.
URL [https://arxiv.org/abs/2108.07732](https://arxiv.org/abs/2108.07732).
- Bercovich et al. (2025)
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al.
Llama-nemotron: Efficient reasoning models.
*ArXiv preprint*, abs/2505.00949, 2025.
URL [https://arxiv.org/abs/2505.00949](https://arxiv.org/abs/2505.00949).
- Black et al. (2024)
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.
Training diffusion models with reinforcement learning.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=YCWjhGrJFD](https://openreview.net/forum?id=YCWjhGrJFD).
- Chang et al. (2022)
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman.
Maskgit: Masked generative image transformer.
In *IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022*, pp.  11305–11315. IEEE, 2022.
doi: 10.1109/CVPR52688.2022.01103.
URL [https://doi.org/10.1109/CVPR52688.2022.01103](https://doi.org/10.1109/CVPR52688.2022.01103).
- Chen et al. (2021)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.
Evaluating large language models trained on code.
*ArXiv preprint*, abs/2107.03374, 2021.
URL [https://arxiv.org/abs/2107.03374](https://arxiv.org/abs/2107.03374).
- Chen et al. (2023)
Ting Chen, Ruixiang Zhang, and Geoffrey E. Hinton.
Analog bits: Generating discrete data using diffusion models with self-conditioning.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023.
URL [https://openreview.net/pdf?id=3itjR9QxFw](https://openreview.net/pdf?id=3itjR9QxFw).
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*ArXiv preprint*, abs/2110.14168, 2021.
URL [https://arxiv.org/abs/2110.14168](https://arxiv.org/abs/2110.14168).
- Cui et al. (2025)
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al.
The entropy mechanism of reinforcement learning for reasoning language models.
*ArXiv preprint*, abs/2505.22617, 2025.
URL [https://arxiv.org/abs/2505.22617](https://arxiv.org/abs/2505.22617).
- DeepMind (2025)
DeepMind.
Gemini diffusion.
2025.
URL [https://deepmind.google/models/gemini-diffusion/](https://deepmind.google/models/gemini-diffusion/).
- Gong et al. (2023a)
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong.
DiffuSeq-v2: Bridging discrete and continuous text spaces for accelerated Seq2Seq diffusion models.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp.  9868–9875, Singapore, 2023a. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.660.
URL [https://aclanthology.org/2023.findings-emnlp.660](https://aclanthology.org/2023.findings-emnlp.660).
- Gong et al. (2023b)
Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and Lingpeng Kong.
Diffuseq: Sequence to sequence text generation with diffusion models.
In *The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023*. OpenReview.net, 2023b.
URL [https://openreview.net/pdf?id=jQj-_rLVXsj](https://openreview.net/pdf?id=jQj-_rLVXsj).
- Gong et al. (2025)
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong.
Scaling diffusion language models via adaptation from autoregressive models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Gu et al. (2024)
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin.
When attention sink emerges in language models: An empirical view.
*ArXiv preprint*, abs/2410.10781, 2024.
URL [https://arxiv.org/abs/2410.10781](https://arxiv.org/abs/2410.10781).
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*ArXiv preprint*, abs/2501.12948, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Hammersley & Mauldon (1956)
John Michael Hammersley and James G Mauldon.
General principles of antithetic variates.
In *Mathematical proceedings of the Cambridge philosophical society*, volume 52, pp.  476–481. Cambridge University Press, 1956.
- Hammersley & Morton (1956)
John Michael Hammersley and Keith William Morton.
A new monte carlo technique: antithetic variates.
In *Mathematical proceedings of the Cambridge philosophical society*, volume 52, pp.  449–475. Cambridge University Press, 1956.
- Han et al. (2023)
Xiaochuang Han, Sachin Kumar, and Yulia Tsvetkov.
SSD-LM: Semi-autoregressive simplex-based diffusion language model for text generation and modular control.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp.  11575–11596, Toronto, Canada, 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.647.
URL [https://aclanthology.org/2023.acl-long.647](https://aclanthology.org/2023.acl-long.647).
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), *Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual*, 2020.
URL [https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html](https://proceedings.neurips.cc/paper/2020/hash/4c5bcfec8584af0d967f1ab10179ca4b-Abstract.html).
- Hoogeboom et al. (2021)
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling.
Argmax flows and multinomial diffusion: Learning categorical distributions.
In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), *Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual*, pp.  12454–12465, 2021.
URL [https://proceedings.neurips.cc/paper/2021/hash/67d96d458abdef21792e6d8e590244e7-Abstract.html](https://proceedings.neurips.cc/paper/2021/hash/67d96d458abdef21792e6d8e590244e7-Abstract.html).
- Hu et al. (2025)
Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S. Abdelfattah, Jae-sun Seo, Zhiru Zhang, and Udit Gupta.
Accelerating diffusion language model inference via efficient kv caching and guided diffusion.
*ArXiv preprint*, abs/2505.21467, 2025.
URL [https://arxiv.org/abs/2505.21467](https://arxiv.org/abs/2505.21467).
- Huang et al. (2024)
Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, JH Liu, Chenchen Zhang, Linzheng Chai, et al.
Opencoder: The open cookbook for top-tier code large language models.
*ArXiv preprint*, abs/2411.04905, 2024.
URL [https://arxiv.org/abs/2411.04905](https://arxiv.org/abs/2411.04905).
- Huang et al. (2025)
Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, and Guo-Jun Qi.
Reinforcing the diffusion chain of lateral thought with diffusion language models.
*ArXiv preprint*, abs/2505.10446, 2025.
URL [https://arxiv.org/abs/2505.10446](https://arxiv.org/abs/2505.10446).
- Hui et al. (2024)
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al.
Qwen2. 5-coder technical report.
*ArXiv preprint*, abs/2409.12186, 2024.
URL [https://arxiv.org/abs/2409.12186](https://arxiv.org/abs/2409.12186).
- Inception Labs et al. (2025)
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, Aditya Grover, and Volodymyr Kuleshov.
Mercury: Ultra-fast language models based on diffusion.
2025.
URL [https://inceptionlabs.ai](https://inceptionlabs.ai).
- Kool et al. (2019)
Wouter Kool, Herke van Hoof, and Max Welling.
Buy 4 reinforce samples, get a baseline for free!
*DeepRLStructPred Workshop ICLR*, 2019.
- Li et al. (2025)
Shufan Li, Konstantinos Kallidromitis, Hritik Bansal, Akash Gokul, Yusuke Kato, Kazuki Kozuka, Jason Kuen, Zhe Lin, Kai-Wei Chang, and Aditya Grover.
Lavida: A large diffusion language model for multimodal understanding.
*ArXiv preprint*, abs/2505.16839, 2025.
URL [https://arxiv.org/abs/2505.16839](https://arxiv.org/abs/2505.16839).
- Li et al. (2022)
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B. Hashimoto.
Diffusion-lm improves controllable text generation.
In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), *Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022*, 2022.
URL [http://papers.nips.cc/paper_files/paper/2022/hash/1be5bc25d50895ee656b8c2d9eb89d6a-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2022/hash/1be5bc25d50895ee656b8c2d9eb89d6a-Abstract-Conference.html).
- Liu et al. (2023)
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.
Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html).
- Liu et al. (2025a)
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong.
Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.
*ArXiv preprint*, abs/2505.24864, 2025a.
URL [https://arxiv.org/abs/2505.24864](https://arxiv.org/abs/2505.24864).
- Liu et al. (2025b)
Zhiyuan Liu, Yicun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyan Wei, Shaobo Wang, and Linfeng Zhang.
dllm-cache: Accelerating diffusion large language models with adaptive caching, 2025b.
URL [https://github.com/maomaocun/dLLM-cache](https://github.com/maomaocun/dLLM-cache).
- Liu et al. (2025c)
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping.
Acereason-nemotron 1.1: Advancing math and code reasoning through sft and rl synergy.
*ArXiv preprint*, abs/2506.13284, 2025c.
URL [https://arxiv.org/abs/2506.13284](https://arxiv.org/abs/2506.13284).
- Lou et al. (2024)
Aaron Lou, Chenlin Meng, and Stefano Ermon.
Discrete diffusion language modeling by estimating the ratios of the data distribution.
In *International Conference on Machine Learning, ICML*, 2024.
- Lozhkov et al. (2024)
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al.
Starcoder 2 and the stack v2: The next generation.
*ArXiv preprint*, abs/2402.19173, 2024.
URL [https://arxiv.org/abs/2402.19173](https://arxiv.org/abs/2402.19173).
- Ma et al. (2025)
Xinyin Ma, Runpeng Yu, Gongfan Fang, and Xinchao Wang.
dkv-cache: The cache for diffusion language models.
*ArXiv preprint*, abs/2505.15781, 2025.
URL [https://arxiv.org/abs/2505.15781](https://arxiv.org/abs/2505.15781).
- Nie et al. (2024)
Shen Nie, Fengqi Zhu, Chao Du, Tianyu Pang, Qian Liu, Guangtao Zeng, Min Lin, and Chongxuan Li.
Scaling up masked diffusion models on text.
*ArXiv preprint*, abs/2410.18514, 2024.
URL [https://arxiv.org/abs/2410.18514](https://arxiv.org/abs/2410.18514).
- OpenAI (2023)
OpenAI.
Gpt-4 technical report.
*ArXiv preprint*, abs/2303.08774, 2023.
URL [https://arxiv.org/abs/2303.08774](https://arxiv.org/abs/2303.08774).
- OpenR1 (2025)
Hugging Face OpenR1.
Open r1: A fully open reproduction of deepseek-r1, 2025.
URL [https://github.com/huggingface/open-r1](https://github.com/huggingface/open-r1).
- Ou et al. (2024)
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li.
Your absorbing discrete diffusion secretly models the conditional distributions of clean data.
*ArXiv preprint*, abs/2406.03736, 2024.
URL [https://arxiv.org/abs/2406.03736](https://arxiv.org/abs/2406.03736).
- Penedo et al. (2024)
Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf.
The fineweb datasets: Decanting the web for the finest text data at scale.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html](http://papers.nips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html).
- Rafailov et al. (2023)
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
*Advances in Neural Information Processing Systems*, 36:53728–53741, 2023.
- Rajbhandari et al. (2020)
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.
Zero: Memory optimizations toward training trillion parameter models.
In *SC20: International Conference for High Performance Computing, Networking, Storage and Analysis*, pp.  1–16. IEEE, 2020.
- Ren et al. (2025)
Allen Z Ren, Justin Lidard, Lars L Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz.
Diffusion policy policy optimization.
In *International Conference on Learning Representations, ICLR*, 2025.
- Roziere et al. (2023)
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al.
Code llama: Open foundation models for code.
*ArXiv preprint*, abs/2308.12950, 2023.
URL [https://arxiv.org/abs/2308.12950](https://arxiv.org/abs/2308.12950).
- Sahoo et al. (2024)
Subham S. Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov.
Simple and effective masked diffusion language models.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/eb0b13cc515724ab8015bc978fdde0ad-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/eb0b13cc515724ab8015bc978fdde0ad-Abstract-Conference.html).
- Sahoo et al. (2025)
Subham Sekhar Sahoo, Zhihan Yang, Yash Akhauri, Johnna Liu, Deepansha Singh, Zhoujun Cheng, Zhengzhong Liu, Eric Xing, John Thickstun, and Arash Vahdat.
Esoteric language models, 2025.
- Schulman et al. (2017)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
*ArXiv preprint*, abs/1707.06347, 2017.
URL [https://arxiv.org/abs/1707.06347](https://arxiv.org/abs/1707.06347).
- Shao et al. (2025)
Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannaneh Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer.
Spurious rewards: Rethinking training signals in rlvr.
[https://rethink-rlvr.notion.site/Spurious-Rewards-Rethinking-Training-Signals-in-RLVR](https://rethink-rlvr.notion.site/Spurious-Rewards-Rethinking-Training-Signals-in-RLVR), 2025.
Notion Blog.
- Shao et al. (2024)
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
*ArXiv preprint*, abs/2402.03300, 2024.
URL [https://arxiv.org/abs/2402.03300](https://arxiv.org/abs/2402.03300).
- Shi et al. (2024)
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K. Titsias.
Simplified and generalized masked diffusion for discrete data.
In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), *Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024*, 2024.
URL [http://papers.nips.cc/paper_files/paper/2024/hash/bad233b9849f019aead5e5cc60cef70f-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2024/hash/bad233b9849f019aead5e5cc60cef70f-Abstract-Conference.html).
- Singh et al. (2023)
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, and Gust Verbruggen.
CodeFusion: A pre-trained diffusion model for code generation.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp.  11697–11708, Singapore, 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.716.
URL [https://aclanthology.org/2023.emnlp-main.716](https://aclanthology.org/2023.emnlp-main.716).
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021*. OpenReview.net, 2021.
URL [https://openreview.net/forum?id=PxTIG12RRHS](https://openreview.net/forum?id=PxTIG12RRHS).
- Sun et al. (2024)
Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al.
A survey of neural code intelligence: Paradigms, advances and beyond.
*arXiv preprint arXiv:2403.14734*, 2024.
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*ArXiv preprint*, abs/2302.13971, 2023.
URL [https://arxiv.org/abs/2302.13971](https://arxiv.org/abs/2302.13971).
- Wang et al. (2025)
Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu.
Octothinker: Revisiting mid-training in the era of rl scaling.
[https://tinyurl.com/OctoThinker](https://tinyurl.com/OctoThinker), 2025.
Notion Blog.
- Williams (1992)
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning.
*Machine learning*, 8:229–256, 1992.
- Wu et al. (2025)
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie.
Fast dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding.
*ArXiv preprint*, abs/2505.22618, 2025.
URL [https://arxiv.org/abs/2505.22618](https://arxiv.org/abs/2505.22618).
- Xiao et al. (2024)
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
Efficient streaming language models with attention sinks.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=NG7sS51zVF](https://openreview.net/forum?id=NG7sS51zVF).
- Xie et al. (2025)
Zhihui Xie, Liyu Chen, Weichao Mao, Jingjing Xu, Lingpeng Kong, et al.
Teaching language models to critique via reinforcement learning.
*ArXiv preprint*, abs/2502.03492, 2025.
URL [https://arxiv.org/abs/2502.03492](https://arxiv.org/abs/2502.03492).
- Xu et al. (2024)
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu.
Lemur: Harmonizing natural language and code for language agents.
In *The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024*. OpenReview.net, 2024.
URL [https://openreview.net/forum?id=hNhwSmtXRh](https://openreview.net/forum?id=hNhwSmtXRh).
- Yang et al. (2025)
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang.
Mmada: Multimodal large diffusion language models.
*ArXiv preprint*, abs/2505.15809, 2025.
URL [https://arxiv.org/abs/2505.15809](https://arxiv.org/abs/2505.15809).
- Ye et al. (2024a)
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong.
Beyond autoregression: Discrete diffusion for complex reasoning and planning, 2024a.
URL [https://arxiv.org/abs/2410.14157](https://arxiv.org/abs/2410.14157).
- Ye et al. (2024b)
Jiacheng Ye, Shansan Gong, Liheng Chen, Lin Zheng, Jiahui Gao, Han Shi, Chuan Wu, Zhenguo Li, Wei Bi, and Lingpeng Kong.
Diffusion of thoughts: Chain-of-thought reasoning in diffusion language models.
*ArXiv preprint*, abs/2402.07754, 2024b.
URL [https://arxiv.org/abs/2402.07754](https://arxiv.org/abs/2402.07754).
- Ye et al. (2025)
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong.
Dream 7b, 2025.
URL [https://hkunlp.github.io/blog/2025/dream](https://hkunlp.github.io/blog/2025/dream).
- Yu et al. (2025)
Runpeng Yu, Xinyin Ma, and Xinchao Wang.
Dimple: Discrete diffusion multimodal large language model with parallel decoding.
*ArXiv preprint*, abs/2505.16990, 2025.
URL [https://arxiv.org/abs/2505.16990](https://arxiv.org/abs/2505.16990).
- Yue et al. (2025)
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang.
Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?
*ArXiv preprint*, abs/2504.13837, 2025.
URL [https://arxiv.org/abs/2504.13837](https://arxiv.org/abs/2504.13837).
- Zekri & Boullé (2025)
Oussama Zekri and Nicolas Boullé.
Fine-tuning discrete diffusion models with policy gradient methods.
*ArXiv preprint*, abs/2502.01384, 2025.
URL [https://arxiv.org/abs/2502.01384](https://arxiv.org/abs/2502.01384).
- Zeng et al. (2025)
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen.
Acecoder: Acing coder rl via automated test-case synthesis.
*ArXiv preprint*, abs/2502.01718, 2025.
URL [https://arxiv.org/abs/2502.01718](https://arxiv.org/abs/2502.01718).
- Zhang et al. (2025)
Ruixiang Zhang, Shuangfei Zhai, Yizhe Zhang, James Thornton, Zijing Ou, Joshua Susskind, and Navdeep Jaitly.
Target concrete score matching: A holistic framework for discrete diffusion.
*ArXiv preprint*, abs/2504.16431, 2025.
URL [https://arxiv.org/abs/2504.16431](https://arxiv.org/abs/2504.16431).
- Zhang et al. (2023)
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua M. Susskind, and Navdeep Jaitly.
PLANNER: generating diversified paragraph via latent language diffusion model.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), *Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023*, 2023.
URL [http://papers.nips.cc/paper_files/paper/2023/hash/fdba5e0a9b57fce03e89cc0cad0a24e9-Abstract-Conference.html](http://papers.nips.cc/paper_files/paper/2023/hash/fdba5e0a9b57fce03e89cc0cad0a24e9-Abstract-Conference.html).
- Zhao et al. (2025)
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover.
d1: Scaling reasoning in diffusion large language models via reinforcement learning.
*ArXiv preprint*, abs/2504.12216, 2025.
URL [https://arxiv.org/abs/2504.12216](https://arxiv.org/abs/2504.12216).
- Zhao et al. (2023)
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li.
Pytorch fsdp: Experiences on scaling fully sharded data parallel.
*Proc. VLDB Endow.*, 16(12):3848–3860, 2023.
ISSN 2150-8097.
doi: 10.14778/3611540.3611569.
- Zheng et al. (2024)
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong.
A reparameterized discrete diffusion model for text generation.
In *Conferenec on Language Modeling, COLM, October 7-9, 2024, Philadelphia, PA*, 2024.
- Zhou et al. (2024)
Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, and Pengfei Liu.
Programming every example: Lifting pre-training data quality like experts at scale.
*ArXiv preprint*, abs/2409.17115, 2024.
URL [https://arxiv.org/abs/2409.17115](https://arxiv.org/abs/2409.17115).
- Zhu et al. (2025)
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, and Chongxuan Li.
Llada 1.5: Variance-reduced preference optimization for large language diffusion models.
*ArXiv preprint*, abs/2505.19223, 2025.
URL [https://arxiv.org/abs/2505.19223](https://arxiv.org/abs/2505.19223).
- Zhuo et al. (2024)
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al.
Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions.
*ArXiv preprint*, abs/2406.15877, 2024.
URL [https://arxiv.org/abs/2406.15877](https://arxiv.org/abs/2406.15877).

## Appendix A Problem Formulation

### A.1 Mask Diffusion Models

Diffusion models (Ho et al., 2020; Song et al., 2021) contain a forward process that gradually corrupts data $\bm{x}_{0}\sim p_{data}(\bm{x}_{0})$ into noisy variables $\bm{x}_{1:T}$ through $q(\bm{x}_{1:T}|\bm{x}_{0})=\prod_{t=1}^{T}q(\bm{x}_{t}|\bm{x}_{t-1})$, and a backward process that models the joint probability as $p_{\theta}(\bm{x}_{0:T})=p_{\theta}(\bm{x}_{T})\prod_{t=1}^{T}p_{\theta}(\bm{x
}_{t-1}|\bm{x}_{t})$, denoising $\bm{x}_{t}$ to reconstruct $\bm{x}_{0}$. The parameters $\theta$ are learned by minimizing the negative log-likelihood via the evidence lower bound (ELBO):

$$ $\displaystyle-\log p_{\theta}(\bm{x}_{0})$ $\displaystyle\leq\mathbb{E}_{q(\bm{x}_{1}|\bm{x}_{0})}[-\log p_{\theta}(\bm{x} _{0}|\bm{x}_{1})]+D_{\mathrm{KL}}(q(\bm{x}_{T}|\bm{x}_{0})||p_{\theta}(\bm{x}_ {T}))+\mathcal{L}_{T},$ (5) $\displaystyle\text{with }\mathcal{L}_{T}$ $\displaystyle=\sum_{t=2}^{T}\mathbb{E}_{q(\bm{x}_{t}|\bm{x}_{0})}[D_{\mathrm{ KL}}(q(\bm{x}_{t-1}|\bm{x}_{t},\bm{x}_{0})||p_{\theta}(\bm{x}_{t-1}|\bm{x}_{t} ))].$ (6) $$

Hoogeboom et al. (2021); Austin et al. (2021a) first proposed discrete diffusion models, where they define the forward process with a categorical distribution $q(\bm{x}_{t}|\bm{x}_{t-1})=\text{Cat}(\bm{x}_{t};\bm{Q}_{t}^{\top}\bm{x}_{t-1})$, where $\bm{x}_{t}\in\{0,1\}^{K}$ is a one-hot vector of vocabulary size $K$, and $\bm{Q}_{t}\in[0,1]^{K\times K}$ is the transition matrix. A special case is absorbing discrete diffusion, where $\bm{Q}_{t}=(1-\beta_{t})I+\beta_{t}\mathbf{1}\bm{m}^{\top}$, with $\mathbf{1}$ being an all-ones vector of size $K$ and $\bm{m}$ the one-hot encoding of a special [MASK] token.
Starting from $\bm{x}_{0}$, the $t$-step marginal distribution is $q(\bm{x}_{t}|\bm{x}_{0})=\text{Cat}(\bm{x}_{t};\bm{p}=\bm{\overline{Q}}_{t}^{
\top}\bm{x}_{0})$, where the cumulative product is $\bm{\overline{Q}}_{t}=\prod_{i=1}^{t}\bm{Q}_{i}=\alpha_{t}I+(1-\alpha_{t})
\mathbf{1}\bm{m}^{\top}$, and $\alpha_{t}=\prod_{i=1}^{t}(1-\beta_{t})$. We expect $\alpha_{T}$ to approach 0 0 such that the fully noised data $\bm{x}_{T}$ equals $\bm{m}$ with probability $1$.

For any two arbitrary time points $0\leq s<t\leq 1$, the transition distribution between them is $q(\bm{x}_{t}|\bm{x}_{s})=\text{Cat}(\bm{x}_{t};\bm{\overline{Q}}_{s|t}^{\top}
\bm{x}_{s})$, where $\bm{\overline{Q}}_{s|t}=\bm{\overline{Q}}_{s}^{-1}\bm{\overline{Q}}_{t}=\frac{
\alpha_{t}}{\alpha_{s}}I+(1-\frac{\alpha_{t}}{\alpha_{s}})\mathbf{1}\bm{m}^{\top}$. This allows us to compute the transition probability between any two timesteps using the ratio of their alpha values:

$$ $q(\bm{x}_{s}|\bm{x}_{t},\bm{x}_{0})=\frac{q(\bm{x}_{t}|\bm{x}_{s})q(\bm{x}_{s} |\bm{x}_{0})}{q(\bm{x}_{t}|\bm{x}_{0})}=\begin{cases}\frac{1\cdot(1-\alpha_{s} )}{1-\alpha_{t}}=\frac{1-\alpha_{s}}{1-\alpha_{t}}=1-\frac{\alpha_{s}-\alpha_{ t}}{1-\alpha_{t}}&\text{if }\bm{x}_{t}=\bm{x}_{s}=\bm{m},\\ \frac{(1-\frac{\alpha_{t}}{\alpha_{s}})\cdot\alpha_{s}}{1-\alpha_{t}}=\frac{ \alpha_{s}-\alpha_{t}}{1-\alpha_{t}}&\text{if }\bm{x}_{t}=\bm{m}\neq\bm{x}_{s} .\\ \end{cases}$ (7) $$

$$ $p_{\theta}(\bm{x_{s}}|\bm{x}_{t})=\frac{\alpha_{s}-\alpha_{t}}{1-\alpha_{t}}f_ {\theta}(\bm{x}_{t})+\frac{1-\alpha_{s}}{1-\alpha_{t}}\bm{m}.$ (8) $$

$$ $D_{\mathrm{KL}}(q(\bm{x}_{s}|\bm{x}_{t},\bm{x}_{0})||p_{\theta}(\bm{x}_{s}|| \bm{x}_{t}))=\begin{cases}\frac{\alpha_{s}-\alpha_{t}}{1-\alpha_{t}}D_{\mathrm {KL}}(\bm{x}_{0}||f_{\theta}(\bm{x}_{t})),&\text{for }\bm{x}_{t}=\bm{m};\\ 0,&\text{for }\bm{x}_{t}\neq\bm{m}.\\ \end{cases}$ (9) $$

Thus,

$$ $\mathcal{L}_{T}=\sum_{t=2}^{T}[-\frac{\alpha_{s}-\alpha_{t}}{(t-s)(1-\alpha_{t })}\delta_{\bm{x}_{t},\bm{m}}\bm{x}_{0}^{\top}\log f_{\theta}(\bm{x}_{t}) \Delta_{t}].$ (10) $$

where $\delta_{\bm{x}_{t},\bm{m}}$ is the indicator function, and $f_{\theta}(\bm{x}_{t})$ represents the logits of the tokens.
In the continuous time limit where $T\rightarrow\infty$, we set a small timestep $\Delta_{t}=t-s=\frac{1}{T}\in(0,1)$. The sum over timesteps becomes an integral, and we have $\alpha_{t}^{\prime}=\frac{\alpha_{t}-\alpha_{s}}{t-s}$. Following the noise schedule $\alpha_{t}=1-t$, which is widely adopted by Shi et al. (2024); Sahoo et al. (2024); Lou et al. (2024); Ou et al. (2024), we get $\frac{-\alpha_{t}^{\prime}}{1-\alpha_{t}}=\frac{1}{t}$.
This can be substituted into Eq. ([6](https://arxiv.org/html/2506.20639v2#A1.E6)), yielding the final ELBO at a sampled time $t$ as a weighted cross-entropy loss:

$$ $\mathcal{L}_{t}=\frac{1}{t}\mathbb{E}_{q(\bm{x}_{t}|\bm{x}_{0})}\left[-\sum_{n =1}^{N}\delta_{\bm{x}_{t}^{n},\bm{m}}(\bm{x}_{0}^{n})^{\top}\log f_{\theta}( \bm{x}_{t})_{n}\right].$ (11) $$

### A.2 Generation Autoregressive-ness

We formally define local and global autoregressiveness (AR-ness) here.

##### Problem Setup

Inference for dLLMs often utilizes low-confidence remasking with the number of diffusion timesteps set equal to the sequence length to ensure high performance.
In this setting, let the target sequence length be $L$, and at each diffusion decoding iteration $t=1,\dots,T$, the set of still-masked positions just before step $t$ is
$M_{t-1}\subseteq\{1,2,\dots,L\}$.
We denote by $p_{t}\in M_{t-1}$ the single position unmasked at step $t$, thereby producing the full decoding order $\{p_{1},\dots,p_{T}\}$.

##### Local Autoregressive-ness: Contiguous Next-Token Prediction

For any integer $k\geq 1$, define

$$ $\mathbb{I}_{\mathrm{local}}(t,k)=\begin{cases}1,&\text{if }\{p_{t-i}\}_{i=1}^{ k}=\{\,p_{t}-i:i=1,\dots,k\},\\ 0,&\text{otherwise.}\end{cases}$ $$

The *Local AR-ness@$k$* is then computed as
$\mathrm{NTP@}k=\frac{1}{T}\sum_{t=1}^{T}\mathbb{I}_{\mathrm{local}}(t,k)$. Local AR-ness measures the tendency to decode the immediate successor of a previously generated token, capturing sequential continuity. It is non-increasing with $k$, as it becomes harder to maintain longer consecutive spans.

##### Global Autoregressive-ness: Earliest-First Mask Selection

At step $t$, sort the masked positions $m_{t-1}^{(1)}<m_{t-1}^{(2)}<\cdots<m_{t-1}^{(|M_{t-1}|)}$.
Then

$$ $\mathbb{I}_{\mathrm{global}}(t,k)=\begin{cases}1,&\text{if }p_{t}\in\{m_{t-1}^ {(1)},\dots,m_{t-1}^{(k)}\},\\ 0,&\text{otherwise.}\end{cases}$ $$

The *Global FMS-ratio@$k$* is
$\mathrm{FMS@}k=\frac{1}{T}\sum_{t=1}^{T}\mathbb{I}_{\mathrm{global}}(t,k)$. Global AR-ness measures the tendency to always unmask the earliest remaining token, capturing a front-to-back filling strategy. Together, these ratios reveal how the model behaves during generation. The ratio is non-decreasing with $k$, as the criterion becomes easier to satisfy when more early positions are allowed.

### A.3 Coupled GRPO

In this section, we provide a detailed formulation of our Coupled GRPO algorithm. As discussed in §[5](https://arxiv.org/html/2506.20639v2#S5), we improve upon the baseline GRPO by introducing a coupled sampling scheme for more accurate probability estimation. The complete Coupled GRPO algorithm is presented in Algorithm [1](https://arxiv.org/html/2506.20639v2#alg1).

##### Probability Estimation with Coupled Sampling

For a given completion sequence $o$ of length $L$, we select $\lambda$ timestep pairs $(t,\hat{t})$ where $t+\hat{t}=T$. For each pair, we create two complementary masks $M_{t}$ and $M_{\hat{t}}$ defined as binary vectors in $\{0,1\}^{L}$, such that:

$$ $M_{t}\,\lor\,M_{\hat{t}}\;=\;\mathbf{1},\quad M_{t}\,\land\,M_{\hat{t}}\;=\; \mathbf{0},$ (12) $$

where $\lor$ and $\land$ denote element-wise OR and AND, respectively, and $\mathbf{1},\mathbf{0}\in\{0,1\}^{L}$ are the all-ones and all-zeros vectors.
The probability estimation for token $o^{k}$ (also marked as $\bm{x}_{0}$) is then computed as:

$$ $\pi_{\theta}(o^{k}|c,o_{t<T}^{k})=\frac{1}{\lambda+1}\bigg{[}\sum_{t+\hat{t}=T }^{\lambda}[\mathcal{L}_{t}(\bm{x}_{t})+\mathcal{L}_{\hat{t}}(\bm{x}_{\hat{t}} )]+\mathcal{L}_{T}(\bm{x}_{T})\bigg{]}.$ (13) $$

$\mathcal{L}_{t}$ is the loss term from Eq ([11](https://arxiv.org/html/2506.20639v2#A1.E11)) at timestep $t$. In detail, we have $\mathcal{L}_{t}(\bm{x}_{t})=M_{t}\cdot\frac{1}{t}\cdot\text{CE}(\bm{x}_{t},\bm
{x}_{0})$ where CE stands for the cross entropy loss.

Figure: Algorithm 1 Coupled GRPO: Policy Optimization with Coupled Sampling

##### Analysis

The coupled sampling scheme provides several benefits:
(i) Full Coverage: Each token is guaranteed to be evaluated exactly once in each coupled pair, ensuring complete coverage of the sequence.
(ii) Reduced Variance: By evaluating each token under realistic partial-masking contexts, we reduce the variance in probability estimates compared to full masking.
(iii) Computational Efficiency: The coupled sampling requires only two additional forward passes per update compared to the d1 (Zhao et al., 2025) baseline when $\lambda=1$.
The variance reduction can be formally quantified in the next section, §[A.4](https://arxiv.org/html/2506.20639v2#A1.SS4).

### A.4 Theoretical Analysis of Coupled-GRPO

In this section, we provide a formal analysis of the coupled sampling scheme used to estimate the per-token log-probability proxies within our coupled-GRPO framework.
GRPO requires stable estimates of these per-token quantities to compute the importance sampling ratios for the policy gradient update.
We demonstrate that our coupled approach can be viewed as a direct and powerful application of the Antithetic Variates (Hammersley & Morton, 1956; Hammersley & Mauldon, 1956) variance reduction technique.
We prove that it provides an unbiased estimator for the desired per-token quantity and, critically, that it is guaranteed to reduce estimation variance.

The core challenge is to obtain a stable estimate of a score for each token in a generated sequence, where this score serves as a proxy for its log-probability. This score is defined as an expectation over a random process involving a diffusion timestep $t$ and a mask $M$.

Assuming the linear noise schedule, the sampling process is as follows:
A timestep $t$ is drawn from a distribution on $[0,1]$, typically $t\sim U(0,1)$.
Different from §[A.3](https://arxiv.org/html/2506.20639v2#A1.SS3) which formulates the loss computation batch-wise, in this section we examine the token-wise formulation.
Conditional on $t$, a mask for each component in the sequence is sampled independently from a Bernoulli distribution $M_{k}\sim\text{Bernoulli}(t),k=1,\dots,L$.

For each token $k$ in a sequence $o$, we define a per-token scoring function, $g(t,M,k)$, which is non-zero only if token $k$ is masked:

$$ $g(t,M,k)=M_{k}\cdot\frac{1}{t}\cdot\ell(o^{k}|c,o_{\mathbf{1}-M}),$ (14) $$

where $\ell(\cdot)$ is the cross-entropy loss for token $o^{k}$ given the condition $c$ and the unmasked context $o_{\mathbf{1}-M}$. The quantity we wish to estimate for each token $k$ is its expected score:

$$ $v_{k}=\mathbb{E}_{t,M}[g(t,M,k)].$ (15) $$

This estimated $v_{k}$ is then used to compute the policy probability ratio $\pi_{\theta}/\pi_{\text{old}}$ in the GRPO objective.

#### A.4.1 Standard vs. Coupled Monte Carlo Estimators

Standard MC Estimator.
To estimate the vector of scores $(v_{1},\dots,v_{L})$, one can draw $2N$ i.i.d. pairs $\{(t_{i},M_{i})\}_{i=1}^{2N}$. The estimator for each token $k$ is:

$$ $\hat{v}_{k,\text{MC}}=\frac{1}{2N}\sum_{i=1}^{2N}g(t_{i},M_{i},k).$ (16) $$

In any given sample $i$, $g(t_{i},M_{i},k)$ is non-zero only for the subset of tokens $k$ where $M_{i,k}=1$. Many samples are needed to obtain a reliable, non-zero estimate for all tokens.

Coupled (Antithetic) Estimator.
Our coupled sampling method generates antithetic pairs. We draw $N$ pairs $\{(t_{i},M_{i})\}_{i=1}^{N}$ and deterministically create their counterparts $(\hat{t}_{i},\hat{M}_{i})=(1-t_{i},\mathbf{1}-M_{i})$. The antithetic variates (AV) estimator for $v_{k}$ is:

$$ $\hat{v}_{k,\text{AV}}=\frac{1}{N}\sum_{i=1}^{N}\frac{g(t_{i},M_{i},k)+g(\hat{t }_{i},\hat{M}_{i},k)}{2}.$ (17) $$

A key structural property emerges here: for any given token $k$ and sample $i$, exactly one of the two terms in the inner sum is non-zero. This is because $M_{i,k}$ and $\hat{M}_{i,k}=1-M_{i,k}$ are binary complements. This guarantees that every token receives a non-zero score contribution from every coupled pair, ensuring full coverage and making the estimation process vastly more efficient.

#### A.4.2 Proofs

Unbiasedness.
We show that $\hat{v}_{k,\text{AV}}$ is an unbiased estimator of $v_{k}$.

###### Proof.

The proof relies on showing that the joint probability distribution of $(t,M)$ is identical to that of its antithetic counterpart $(\hat{t},\hat{M})$.
As proven in the previous version, under the symmetric sampling scheme for $t$ and the dependent Bernoulli sampling for $M$, we have $p(t,M)=p(\hat{t},\hat{M})$.

Since the random variables $(t,M)$ and $(\hat{t},\hat{M})$ are identically distributed, the expectation of any function of these variables is the same:

$$ $\mathbb{E}[g(t,M,k)]=\mathbb{E}[g(\hat{t},\hat{M},k)]=v_{k}.$ (18) $$

By linearity of expectation, the expectation of the AV estimator is:

$$ $\mathbb{E}[\hat{v}_{k,\text{AV}}]=\frac{1}{2N}\sum_{i=1}^{N}(\mathbb{E}[g(t_{i },M_{i},k)]+\mathbb{E}[g(\hat{t}_{i},\hat{M}_{i},k)])=\frac{1}{2N}\sum_{i=1}^{ N}(v_{k}+v_{k})=v_{k}.$ (19) $$

Thus, the coupled estimator is unbiased.
∎

###### Proof.

##### Variance Reduction.

We now provide a direct and rigorous proof that the coupled estimator has lower variance than the standard MC estimator.

###### Proof.

The variance of the AV estimator for token $k$ is given by:

$$ $\text{Var}(\hat{v}_{k,\text{AV}})=\frac{\text{Var}(g(t,M,k))+\text{Cov}(g(t,M, k),g(\hat{t},\hat{M},k))}{2N}.$ (20) $$

Variance is reduced if and only if the covariance term is negative.
Let us analyze the covariance:

$$ $\text{Cov}(g(t,M,k),g(\hat{t},\hat{M},k))=\mathbb{E}[g(t,M,k)\cdot g(\hat{t}, \hat{M},k)]-\mathbb{E}[g(t,M,k)]\cdot\mathbb{E}[g(\hat{t},\hat{M},k)].$ (21) $$

Consider the product term inside the first expectation:

$$ $g(t,M,k)\cdot g(\hat{t},\hat{M},k)=\left(M_{k}\cdot\frac{1}{t}\ell(\dots) \right)\cdot\left(\hat{M}_{k}\cdot\frac{1}{\hat{t}}\ell(\dots)\right).$ (22) $$

The crucial insight is that the product of the mask indicators $M_{k}\cdot\hat{M}_{k}$ is always zero, since $\hat{M}_{k}=1-M_{k}$ and $M_{k}$ is either 0 or 1.
Therefore, the product of the scoring functions is identically zero for all possible values of $t$ and $M$. This means its expectation is also zero:

$$ $\mathbb{E}[g(t,M,k)\cdot g(\hat{t},\hat{M},k)]=0.$ (23) $$

Substituting this back into the covariance formula:

$$ $\displaystyle\text{Cov}(g(t,M,k),g(\hat{t},\hat{M},k))$ $\displaystyle=0-(\mathbb{E}[g(t,M,k)])\cdot(\mathbb{E}[g(\hat{t},\hat{M},k)])$ (24) $\displaystyle=-v_{k}\cdot v_{k}$ (25) $\displaystyle=-v_{k}^{2}.$ (26) $$

Since the loss $\ell(\cdot)$ is non-negative, the scoring function $g$ is non-negative. Its expectation, $v_{k}$, must therefore be non-negative. Assuming there is some configuration where a loss is incurred (i.e., $v_{k}>0$), we have:

$$ $\text{Cov}(g(t,M,k),g(\hat{t},\hat{M},k))=-v_{k}^{2}<0.$ (27) $$

The covariance is guaranteed to be negative. The variance reduction is therefore not just plausible but a mathematical certainty of this estimation scheme. The amount of reduction is:

$$ $\text{Var}(\hat{v}_{k,\text{MC}})-\text{Var}(\hat{v}_{k,\text{AV}})=\frac{ \sigma^{2}_{g}}{2N}-\frac{\sigma^{2}_{g}-v_{k}^{2}}{2N}=\frac{v_{k}^{2}}{2N}>0.$ (28) $$

∎

This result stems directly from the mutually exclusive nature of the estimators for a given token within a coupled pair, a direct consequence of the complementary masks.

###### Proof.

## Appendix B Implementation Details

### B.1 Training Details

##### Adaptation pretraining

During pre-training, we filter the code pre-training corpus from RefineCode (Huang et al., 2024) and Stackv2 (Lozhkov et al., 2024). Since RefineCode only provides each item’s index in Stackv2, we built a local index engine to download the raw data from Stackv2.
We also used text and math data from Fineweb (Penedo et al., 2024) and DCLM-pro (Zhou et al., 2024).
The final processed dataset contains around 400B tokens, as shown in Table [3](https://arxiv.org/html/2506.20639v2#A2.T3). We adopted the code-to-text ratio suggested in Qwen-2.5-Coder (Hui et al., 2024) and OpenCoder (Huang et al., 2024).
The training was conducted on 10 nodes of 8 H100 GPUs each, using BF16 and full-shard FSDP (Zhao et al., 2023). The total wall-clock time for training on 65B tokens (100,000 global steps) was approximately 40 hours.
The single-GPU batch size was $2$, and the context window was $4096$. Following LLaDA (Nie et al., 2024), we truncated 1% of the data to a random length to improve handling of variable-length inputs.
Additionally, for another 1% of the data, we used a random-length prefix as the condition during the diffusion process, which was kept unnoised.
We used the Adam optimizer with a maximum learning rate of 2e-5, with a linear warmup of 20,000 steps followed by a cosine decay schedule to 10% of its peak value at the end of training. The attention mask annealing was performed over 10,000 steps, following DiffuLLaMA (Gong et al., 2025).
In our experiments, we observed that training on more tokens in Stage 1 did not improve performance on downstream tasks (Table [4](https://arxiv.org/html/2506.20639v2#A2.T4)), nor did it lead to further improvements in Stage 2.
Therefore, we used the model trained with 65B tokens as our Stage 1 model. With a higher-quality pre-training corpus, we might have reached different conclusions.

**Table 3: Adaptation Training Data Recipes on Stage 1**
| Source | # tokens | Sample weight | Percentage |
| --- | --- | --- | --- |
| RefineCode from Stackv2 (code) | 330B | 1 | 78% |
| Fineweb code page (text) | 55B | 1 | 20% |
| DCLM subset (text) | 33B | 1 |  |
| Fineweb math page (math) | 3B | 3 | 2% |

**Table 4: Raw results on coding tasks for LLMs and dLLMs in 7/8B scale. ^∗ denotes that the results are collocated from public reports instead of evaluating by ourselves. BigCodeBench has completion (C) and instruction (I) version of template during evaluation. +SFT means we conduct the same instruction tuning with DiffuCoder-Instruct.**
| Model | HumanEval | MBPP | BigCodeBench (C) | BigCodeBench (I) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| - | Plus | - | Plus | Full | Hard | Full | Hard |  |
| Base Models |  |  |  |  |  |  |  |  |
| Qwen2.5-Coder | 61.6 | 51.8 | 75.9 | 61.4 | 46.1 | 16.2 | 40.2 | 14.2 |
| OpenCoder^∗ | 66.5 | 63.4 | 79.9 | 70.4 | 40.5 | 9.5 | – | – |
| LLaDA | 35.4 | 30.5 | 50.1 | 42.1 | 18.9 | 4.1 | – | – |
| Dream | 56.7 | 50.0 | 68.7 | 57.4 | 23.6 | 4.1 | – | – |
| DiffuCoder (Stage 2 65B) | 67.1 | 60.4 | 74.2 | 60.9 | 40.2 | 12.8 | 1.8 | 0.7 |
| - Stage 1 65B | 39.0 | 31.7 | 48.4 | 38.3 | – | – | – | – |
| - Stage 1 720B | 31.1 | 23.3 | 38.8 | 31.3 | – | – | – | – |
| - Stage 2 16B | 66.5 | 61.0 | 71.9 | 57.1 | 36.7 | 8.8 | – | – |
| Instruct Models |  |  |  |  |  |  |  |  |
| Qwen2.5-Coder-Instruct | 90.2 | 85.4 | 83.9 | 72.0 | 50.7 | 21.6 | 42.2 | 18.2 |
| Qwen2.5-Coder+SFT | 82.9 | 75.6 | 80.1 | 66.1 | 46.9 | 16.2 | 39.5 | 14.9 |
| OpenCoder-Instruct^∗ | 83.5 | 78.7 | 79.1 | 69.0 | 40.3 | 16.9 | – | – |
| LLaDA-Instruct | 35.4 | 31.7 | 31.5 | 28.6 | 16.5 | 2.7 | 14.1 | 1.4 |
| Dream-Instruct | 57.9 | 53.7 | 68.3 | 56.1 | 10.6 | 0.7 | 11.4 | 2.7 |
| Dream+SFT | 56.7 | 50.6 | 71.7 | 58.7 | 27.4 | 6.1 | 25.9 | 5.4 |
| DiffuCoder-Instruct (5 ep) | 72.0 | 65.2 | 75.1 | 61.9 | 35.7 | 12.2 | 34.0 | 8.8 |
| - 1 epoch | 67.1 | 60.4 | 75.7 | 61.9 | 31.1 | 8.1 | 32.0 | 6.1 |
| - 1 ep flexible padding | 65.2 | 58.5 | 71.2 | 59.5 | 35.4 | 11.5 | 32.4 | 10.1 |
| + coupledGRPO (1ep) | 73.2 | 68.3 | 78.6 | 67.5 | 40.4 | 10.8 | 37.5 | 10.8 |
| + coupledGRPO (2ep) | 70.1 | 65.2 | 79.1 | 65.3 | 42.8 | 14.9 | 39.4 | 13.5 |
| Commercial Models |  |  |  |  |  |  |  |  |
| GPT 4o^∗ | 90.2 | – | 82.2 | – | 49.9 | – | – | – |
| Mercury^∗ | 90.0 | – | 77.1 | – | 45.5 | – | – | – |
| Gemini Diffusion^∗ | 89.6 | – | 76.0 | – | 45.4 | – | – | – |

##### Mid-training

We used around 16B tokens of annealing code data (Huang et al., 2024) during mid-training. This dataset contains an algorithmic corpus and synthetic data, such as high-quality code snippets and code textbooks.
This high-quality mid-training data significantly improves the capacity of the base model, and we chose the model trained on 65B tokens (roughly 4 epochs over the training data) as the final version of DiffuCoder Base.
The training was carried out on 8 nodes with 8 A100 GPUs each, using BF16 and full-shard FSDP (Zhao et al., 2023), which took 90 hours of wall-clock time. We used the Adam optimizer with a maximum learning rate of 1e-5 and a linear warmup of 2,000 steps.
Other settings were the same as in Stage 1.

##### Instruction tuning

We conducted classifier-free guidance SFT (Gong et al., 2023a; Nie et al., 2024; Ye et al., 2024b) for DiffuCoder using 436K instruction tuning samples from OpenCoder (Huang et al., 2024).
The classifier-free guidance approach uses a conditional mask to prevent the diffusion process from adding noise to the condition prefix. We used the same chat template as Qwen2.5-Coder (Hui et al., 2024).
We trained a new padding token that was used to pack each sample to a fixed length to control the generation length. We compared different SFT strategies for DiffuCoder instruction tuning, including: (i) using a fixed sequence length of 2048 for each sample with a conditional mask; (ii) using a fixed 2048 length but mixing conditional and unconditional training; and (iii) *flexible padding*, where we padded to the maximum sequence length in the current batch instead of a fixed 2048. Finally, we progressively added a conditional mask for the first epoch, considering that Stages 1 and 2 were trained unconditionally, and then trained the remaining 4 epochs using a fixed sequence length of 2048. The SFT code is based on LLaMA-Factory(^5^55[https://github.com/hiyouga/LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)). The training was performed on 8 nodes of 8 H100 GPUs each, using BF16 and ZeRO2 (Rajbhandari et al., 2020), taking around 24 hours.
We used the Adam optimizer with a maximum learning rate of 1e-5 and a linear warmup ratio of 0.1, followed by a cosine decay schedule to 10% of its peak value at the end of training.

##### Coupled GRPO

For RL training, for both our model and the ablation baselines, we filtered 21K hard samples from Acecoder-87k (Zeng et al., 2025) with verifiable test cases and trained for one epoch.
We used the pass rate of reference solutions in this dataset to filter for questions with a low average (bottom 20%) and high variance (top 40%) pass rate. These were marked as hard samples, yielding the final 21K samples used for GRPO training.
We found that filtering the training samples by difficulty into a proper range is important. We used the online sandbox E2B(^6^66[https://e2b.dev/](https://e2b.dev/)) for code execution and reward verification.
We trained the models on a single node with 8 H100 GPUs for a wall-clock time of 40 hours.
The default GRPO training parameters were: reference model sync steps $64$, number of iterations $\mu=2$, $\beta=0.01$, $\varepsilon=0.5$, learning rate 1e-6, and maximum completion length $256$. The rollout parameters were: diffusion timesteps $256$, rollout samples $G=10$, and sampling temperature $1.2$.
When sampling coupled $t$, we empirically chose a range of $[0.2,0.8]$ instead of $[0,1.0]$ to avoid extreme loss values (Figure [8](https://arxiv.org/html/2506.20639v2#A2.F8)).
Despite training for only one epoch with coupled GRPO, we observed that DiffuCoder-Instruct’s performance continued to rise as we added more training steps.

Figure: Figure 8: Validation loss distribution of DiffuLLaMA (Gong et al., 2025) for different timesteps. The weighted loss refers to $\mathcal{L}_{t}$ in Eq. ([11](https://arxiv.org/html/2506.20639v2#A1.E11)), while the unweighted loss refers to the cross-entropy term in the masked diffusion loss without the $1/t$ weighting. Timesteps that are too large or too small will lead to extreme values. The sweet spot is in the span of $[0.2,0.8]$, which is also consistent for DiffuCoder.
Refer to caption: x13.png

We designed a weighted reward for each completion $o_{i}$ as $r(o_{i})=2.0\times r_{\text{code}}(o_{i})+0.5\times r_{\text{format}}(o_{i})$.
$r_{\text{code}}(o_{i})$ is the pass rate on the test cases, evaluated only if $r_{\text{format}}(o_{i})=1$.

- •
0.5 if $o_{i}$ contains a valid Markdown code block and passes a Python syntax check;
- •
0.25 if the code block format is correct but the code has a syntax error;
- •
0 if the Markdown block format is not matched.

The raw results are in Table [4](https://arxiv.org/html/2506.20639v2#A2.T4). Our design choices in §[B.1](https://arxiv.org/html/2506.20639v2#A2.SS1) rely on these results. DiffuCoder Base underperforms on the instruction-query BigCodeBench (I), likely because the pre-training corpus teaches the base model to focus on completion rather than instruction following.
Consequently, it fares better on the subset BigCodeBench (C) with completion query. After instruction tuning, DiffuCoder-Instruct achieves reasonable performance on BigCodeBench (I), indicating that instruction tuning teaches the model to follow instructions.

Table: Table 5: Chat template during the evaluation for HumanEval and BigCodeBench (C).

### B.2 Evaluation Details

By default, LLaDA (Nie et al., 2024) employs low confidence remasking with temperature 0 0, while Dream (Ye et al., 2025) uses top negative entropy remasking with temperature $0.2$.
Both models use a maximum sequence length and diffusion timesteps of $512$. GSM8K tests are conducted using lm-harness(^7^77[https://github.com/EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness)), and code benchmarks are based on Qwen2.5-Coder’s evaluation suitcase(^8^88[https://github.com/QwenLM/Qwen2.5-Coder/tree/main/qwencoder-eval](https://github.com/QwenLM/Qwen2.5-Coder/tree/main/qwencoder-eval)).
DiffuCoder shares the architecture (Qwen2.5-7B), tokenizer, and inference implementation of Dream-7B. The chat templates we used during the inference are listed below.

## Appendix C Additional Results

### C.1 Entropy Pattern

In §[4.2](https://arxiv.org/html/2506.20639v2#S4.SS2), we present the phenomenon *entropy sink* and illustrate the L-shaped distribution of the confidence score for a single time step (the first forward). In Figure [9](https://arxiv.org/html/2506.20639v2#A3.F9), we extend the visualization to 2D heatmaps. We can still observe a casual bias in these examples, but more token information and flexibility are involved compared to the strict causal mask.

Figure: Figure 9: Visualization of the decoding entropy for random samples. The x-axis is the index of generated token, while y-axis refers to decoding steps. Here we set the diffusion timestep and generation length to be equal.
Refer to caption: x14.png

### C.2 Decoding Samples

Figure [10](https://arxiv.org/html/2506.20639v2#A3.F10) demonstrates the generation order for different temperatures. The background color runs monotonically from red (earliest) through the spectrum to purple (latest).
As we can see, the higher temperature leads to less AR sequence generation. The model tends to determine the right-hand side first, including pad tokens which decide the generation length, and the key parts of this code snippet are generated near the end.

Figure: Figure 10: Visualization of the decoding trajectory of DiffuCoder-Instruct under different sampling temperatures. Each character’s background is colored from red to purple according to the recover order of the [MASK]. Left: temperature is $0.2$; Right: temperature is $1.2$.
Refer to caption: extracted/6574020/fig/case1.png

### C.3 Decoding Timesteps

Figure: Figure 11: Different model variants act differently when changing decoding timesteps to 1/2 of the sequence length. 1x means the default setting where decoding timesteps are equal to the sequence length while 2x means 1/2 fewer steps which will result in 2x speedup.
Refer to caption: x17.png

Another key motivation for measuring AR-ness is its relationship with generation parallelism.
A model with very high AR-ness implies strong left-to-right token dependencies, which limits opportunities for parallel decoding.
Conversely, lower AR-ness suggests that the model can generate multiple tokens more independently, enabling fewer diffusion steps and thus faster generation.
In other words, by monitoring AR-ness, we also uncover how much headroom remains for accelerated, parallel decoding schedules. In Figure [11](https://arxiv.org/html/2506.20639v2#A3.F11), if we correlate the performance drop from 1x to 2x with the non-AR (parallelism) decoding pattern, where a higher drop indicates higher AR-ness, then we can draw the following conclusions. (1) Compared with the Instruct model (the starting point of RL training), GRPO training improves DiffuCoder-Instruct’s parallelism. (2) Compared with one epoch of instruction tuning, training for more epochs (5ep here) can reduce the AR-ness of the model. (3) For different sampling temperatures in the GRPO-trained DiffuCoder, the higher temperature (0.4) brings less AR-ness and thus a smaller performance drop at 2x speed.

### C.4 Coupled GRPO Training

We monitored the completion length during GRPO training but did not observe a consistent increase in length as seen in AR GRPO training (Shao et al., 2024; OpenR1, 2025). A possible reason is that we do not encourage long-chain reasoning generation (Liu et al., 2025c), which could be a future research direction.

In our experimental environment, the end-to-end GRPO training time for DiffuCoder is twice that of the AR model Qwen2.5-Coder.

Figure: Figure 12: Reward curves during GRPO training. Left: Comparison between coupled GRPO and d1 baselines (based on an early version of DiffuCoder-Instruct). Middle: Decoupled GRPO uses the same number of sampling times but with randomly sampled masks (based on an early version of DiffuCoder-Instruct). Right: Coupled-GRPO on DiffuCoder is compared with regular GRPO for the AR model Qwen2.5Coder+SFT.
Refer to caption: x18.png

## Appendix D Discussions

##### Data Quality

It is well-known that data quality plays a crucial role in LLM training (Hui et al., 2024). Our current experiments rely entirely on open-source community datasets (Huang et al., 2024). Since our focus is on experimental validation rather than achieving absolute performance, the data used may not be of the highest quality, and the amount of high-quality data is also limited. Future scaling, either in pre-training or RL, could benefit from better-curated datasets.

##### Template Sensitivity

The instruction templates used in our GRPO training data (Zeng et al., 2025) are relatively fixed and lack diversity. This may limit the model’s generalization ability, as it could become overly reliant on specific prompt formats. Incorporating more diverse, high-quality instructions could help improve robustness during evaluation.

##### Python Code

Our current training and evaluation primarily focus on Python. Extending these methods to multiple programming languages and exploring mixed text-code agent settings (Xu et al., 2024) are promising future directions.

##### Long Reasoning

DiffuCoder has not been trained on tasks involving long reasoning chains (An et al., 2025; Liu et al., 2025c). This is mainly due to the limited sequence length and slower inference speed of current dLLMs. Supporting longer reasoning chain remains a challenge for future work.

##### Entropy Analysis

Recent studies on AR LLMs have examined token-level entropy changes during RL training (Cui et al., 2025; Liu et al., 2025a; Agarwal et al., 2025). A promising direction would be to further analyze token entropy in our setting to better understand its dynamics and potential impact on reward optimization.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1.	Please evaluate λ>1 (multi-pair coupling) and alternative t-samplings (e.g., Beta/log-uniform), and report the variance–accuracy–compute trade-offs.
2.	Please add a compute-parity comparison against semi-autoregressive RL (block diffusion / re-masking) with matched rollouts and wall-clock time to quantify the benefit of avoiding AR bias.
3.	Please demonstrate robustness by sweeping syntax/format/test weights, adding static analysis and richer tests, and reporting mean ± σ over 3 seeds.
4.	Please report practicality: end-to-end throughput (tokens/s) and training/inference cost, and comment on whether your method stacks with existing dLLM accelerations.
soundness: 3
strengths: 1.	Proposes local/global AR-ness@k to compare dLLMs vs. AR LLMs across stages and modalities. 
2.	Higher temperature makes decoding less AR-like and raises pass@k; real sample trajectories are visualized. 
3.	Coupled-GRPO lowers variance via complementary masks and avoids semi-AR bias, yielding stable post-training gains. 
4.	From adaptation → mid-training → SFT → RL, with compute setups that aid reproducibility. 
5.	When halving steps (2× speed), performance drops less after GRPO.
summary: This paper is a systematic study and methodological work on applying diffusion language models (dLLMs) to code generation; it introduces AR-ness metrics to quantify decoding order and a diffusion-native RL approach (coupled-GRPO), delivers consistent gains across multiple code benchmarks, and reveals the key phenomenon that temperature affects both token selection and generation order.
weaknesses: 1.	Generalization to multi-language, multi-file, or agentic tasks is unclear. 
2.	Multi-stage large-scale training plus RL rollouts; dLLM GRPO takes ~2× AR’s wall time. 
3.	Even with variance reduction, the training still approximates token log-likelihoods.
4.	Stage-1 at ~700B tokens hurts downstream performance, implying sensitivity to data quality and early stopping. 
5.	No λ>1 (multi-pair coupling) or alternative t-distributions; limited analysis of reward weighting and verifiers.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Have you considered applying coupled-GRPO to existing diffusion backbones like LLaDA or Dream to assess its modularity and transferability?
2. Given that the proposed method enhances performance with only two diffusion blocks, have you observed any performance degradation or instability with longer generation lengths or more complex prompts?
3. Do you expect the entropy-sink decoding behavior to generalize to non-code domains, or is it specific to code and math where token dependencies are more structured?
soundness: 4
strengths: * Proposes the first full pipeline for large-scale masked diffusion models in code generation, demonstrating competitive performance with AR models.
* Introduces coupled-GRPO, a principled, theoretically justified improvement over prior RL-based DLM training.
* Deep behavioral analysis offers valuable insight into how diffusion decoding diverges from autoregressive patterns.
* Experiments are comprehensive, including comparisons with prior DLMs, autoregressive baselines, and multiple ablations.
* The paper is clearly written and well-structured, making technical innovations accessible.
summary: This paper presents *DiffuCoder*, a large-scale diffusion language model (DLM) designed for code generation. Unlike traditional DLMs that struggle with fixed decoding order and limited self-correction capability, DiffuCoder introduces a complete four-stage training framework including pretraining, mid-tuning, instruction fine-tuning, and reinforcement learning. A major contribution lies in the proposed coupled-GRPO, a variance-reduced reinforcement learning algorithm that enables effective self-reflective sampling under high-temperature decoding. The authors also conduct detailed behavioral analysis, demonstrating DiffuCoder's flexible, non-autoregressive decoding strategy and strong performance across major code and math benchmarks.
weaknesses: This work is strong across the board — in terms of methodological innovation, clarity of explanation, and thorough ablation studies. I did not find any substantive weaknesses, and I believe the paper easily meets the bar for poster acceptance at ICLR. However, given that the core method could likely generalize to domains beyond code (e.g., open-ended language modeling, multimodal reasoning), it would have been valuable to see some discussion of this potential in the main paper. Including such analysis or preliminary results could elevate the work to oral-level significance.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to **Weaknesses**.
soundness: 3
strengths: 1- Provides the **first systematic measurement of AR-ness** in diffusion LLMs, with clear metrics and visualizations.

2- **Coupled-GRPO** is theoretically motivated (antithetic variates) and supported by formal variance-reduction proof.

3- Offers a **complete open-source training recipe** for a diffusion-based code LLM, potentially useful for the community.

4- Connects decoding order, sampling temperature, and parallelism, yielding practical insights for efficient generation.

5- Writing and figures are generally clear and reproducible; appendices are thorough.
summary: This paper introduces DiffuCoder, a 7B-parameter diffusion LLM (dLLM) for code, and uses it to investigate the decoding behavior of dLLMs. The authors propose "autoregressive-ness" (AR-ness) metrics to quantify how left-to-right dLLM generation is and analyze the effect of sampling temperature on diversity. Based on these insights, they introduce Coupled-GRPO, a reinforcement learning method that uses complementary mask pairs to reduce variance in policy gradient estimation. Experiments show DiffuCoder is competitive with AR baselines and that Coupled-GRPO provides modest performance gains (+4.4% on EvalPlus) and improved decoding parallelism.
weaknesses: ### **Incremental Analytical Novelty** 
The key observations from the AR-ness analysis (e.g., the effect of temperature, the difference between adapted and from-scratch models) confirm properties that are largely expected from the principles of masked diffusion. The contribution is in the measurement, not the discovery of new phenomena. I would suggest that the authors tone the it down in the contribution sections.

### **Modest and Inconsistent Algorithmic Gains**
The improvements from Coupled-GRPO are relatively small and not uniform across all evaluated benchmarks (Table 2), raising questions about its overall impact.

### **Insufficient Ablations**
The paper lacks critical ablations for its core method, including:
- A sweep of the coupling parameter λ to test if the benefits of "coupled" sampling scale.
- A direct comparison against a "2x decoupled" baseline to isolate the effect of complementary sampling from simply doubling the number of forward passes.
- A clear analysis of the computational cost/overhead of Coupled-GRPO.

### **Missing Baseline**
I understand that **Dream-Coder-7B** is relatively a new baseline, but the absence of a comparison with that model gives an unjust favor to DiffuCoder over other open-sourced dLLMs. I would love to see that comparison added to the tables in the rebuttal period, as the model was released almost two months prior to the submission deadline.

### **Overstated Claims** 
Several passages (e.g., on temperature-driven diversity) could be more accurately framed as measurements of known model properties rather than novel discoveries.

### **Missing Citations**
The paper does not fully contextualize its work within the broader literature. Specifically:
- When discussing early masked diffusion formulations (e.g., around line 52), the paper should cite "Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data" (Ou et al., 2024)[1] as a concurrent and relevant work in this space.
- The discussion of semi-AR decoding and Block Diffusion (Arriola et al., 2025)[2] would be strengthened by also citing the concurrent work "Unifying Autoregressive and Diffusion-Based Sequence Generation" (Fathi et al., 2025)[3], which explores similar hybrid modeling ideas. Properly crediting these concurrent works is important for a complete scholarly narrative.

### **Practical Utility of AR-ness** 
Beyond interpretability, what are the actionable insights or design recommendations from the AR-ness analysis? For example, could a model's AR-ness score be used to dynamically adjust the number of diffusion steps during inference for speedup without a major performance drop?

### Minor Typographical or formatting issues
1- Page 4, Section 3: "We observed that training with 700B tokens in Stage 1 led to worse performance than using only 65B tokens" -> This is a key observation. The reasoning is plausible, but could be slightly expanded.

2-Page 5, Table 1 Footnote: "Scores are bolded when our model outperforms LLMs specialized for code (excluding LLaDA or Dream)." -> This is an unusual and potentially confusing bolding criterion. It would be clearer to bold the best score in each column or simply state the comparison in the text.

3- Page 8, Section 5: "w. full mask completion" -> Should be "w/ full mask completion" or "with full mask completion" for formal writing.

### **References**
[1]  Ou et al., "Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data"

[2] Arriola et al., "Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models"

[3] Fathi et al., "Unifying Autoregressive and Diffusion-Based Sequence Generation"

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses for major concerns.

Miscellaneous:
The figure legends overlap with the plots, which makes them difficult to read. Please consider reformatting the figures to improve clarity and presentation.
soundness: 3
strengths: 1. The proposed AR-ness metrics are intriguing and provide insightful analysis of how and when discrete diffusion models exhibit autoregressive-like behavior. While some obervations may seem intuitive, concluding them in a formal way is a meaningful contribution.
2. The introduction of Coupled-GRPO is a simple yet effective technique for adapting GRPO to discrete diffusion models. Table 1 presents a comprehensive benchmark across various LLMs and dLLMs, showing that DiffuCoder clearly outperforms other diffusion-based counterparts, and that DiffuCoder + Coupled-GRPO further improves upon the base model. I also agree that approaches which make dLLMs overly autoregressive undermine their purpose, so it is encouraging to see these results.
summary: The authors trained a 7B diffusion large language model (dLLM), DiffuCoder, on 130B tokens of code and analyzed its decoding behavior on the “AR-ness” of the its sampling process. The authors’ contributions are two-fold: 1. proposed a global and a local AR-ness metric for dLLMs and used the metrics to analyze the sampling process; 2. proposed coupled-GRPO, a RL technique that adapts GRPO of autoregressive LLMs to dLLMs.
weaknesses: The two main contributions of the paper are valuable in their own right, but I find it difficult to fully understand how they connect as part of a unified method. While the AR-ness metrics are conceptually interesting and insightful, it remains unclear how they relate to or motivate the proposed Coupled-GRPO. Specifically, how does Coupled-GRPO address or influence the behavior measured by the AR-ness metrics? There is only a brief discussion around line 468 and a few related observations in Appendix D.4, but it would be helpful if the authors could provide a clearer explanation or reference that explicitly connects how Coupled-GRPO contributes to improving non-AR behavior in diffusion models.

For Coupled-GRPO, the method pairs timesteps whose indices sum to T, the total number of diffusion steps. This design implicitly assumes a symmetric noising schedule, where the masking ratios at t and T−t are approximately complementary. It would be helpful for the authors to clarify whether this pairing choice is arbitrary or theoretically justified, and whether they have explored or evaluated alternative coupling strategies under non-symmetric schedules.


Decision: Accept (Poster)