Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models
Authors: Yinjie Wang
Ling Yang
Bowen Li
Ye Tian
Ke Shen
Mengdi Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Language Models
Reinforcement Learning
Primary area: foundation or frontier models, including LLMs

Abstract: The extension of diffusion models to language tasks has shown promising results, but their post-training methods remain largely unexplored. We highlight the importance of aligning a diffusion language model’s preference-inference trajectory with its post-training objective. To this end, we propose TraceRL, a trajectory-aware reinforcement learning framework for DLMs that incorporates information from inference trajectories into post-training and is applicable to both full-attention and block-attention diffusion models. We also introduce a diffusion-based value model that enhances training stability and naturally accommodates process rewards. We demonstrate TraceRL’s superiority in enhancing a model’s reasoning ability on complex math and coding tasks, as well as its applicability in scaling block diffusion models to larger block sizes. Employing TraceRL, we derive a series of state-of-the-art diffusion language models, namely TraDo. Although smaller than Qwen2.5-7B-Instruct, TraDo-4B-Instruct consistently outperforms it on complex math reasoning tasks. TraDo-8B-Instruct achieves 4.5% higher accuracy on MATH500 than Qwen2.5-7B-Instruct and 6.6% higher accuracy on LiveCodeBench-V2 than Llama3.1-8B-Instruct. Through curriculum learning, we also develop the first 8B-scale long-CoT diffusion language model. We open-source our code at [https://github.com/Gen-Verse/dLLM-RL](https://github.com/Gen-Verse/dLLM-RL).

Published as a conference paper at ICLR 2026
REVOLUTIONIZING REINFORCEMENT LEARNING FRAMEWORK FOR DIFFUSION LARGE LANGUAGE MODELS
Yinjie Wangl.2,, Ling Yang'*; Bowen Li", Princeton University. University of Chicago, Project: https//gituub.com/innnneeeeee -er.-Genn-eeeee..LLM---
Ye Tian", Ke Shen, Mengdi Wang* Peking University
ABSTRACT
The extension of diffusion models language tasks has shown promising sults, but their post-training methods remain largely unexplored. We highlight the importance of aligning diffusion language model's preference-inference jectory with its post-training objective. To this end, we propose TraceRL, a traeectory-aware reinforcement learning framework for DLMs that incorporates information from inference trajectories into post-training and is applicable to both full-atten alll-attettonn and attentono diffusion models. We also introduce a diffusion- based value model that enhances training stability and naturally accommodates process rewards. We demonstrate TraceRL's superiority enhancing a model's reasoning ability on complex math and coding tasks, well as its applicabil- ity in scaling block diffusion models to larger block sizes. Employing TraceRL, we derive series of oftte-of-theart diffusion language models, namely TraDo. Although smaller than Qwen2.5-775 Instruct, TraDo-4B-Instruct consistently out- performs it on complex math reasoning tasks. TraDo-8B Instrucct achieves 4.5% higher accuracy on MATH500 than @0wen..5-7B-Instreee and 6.6% higher ac- curacy on LiveCodeBench-V? than Llama3.1 1.aaa...11.00 1-S--tntt Through curriculum learning, we also develop the first 8B-scale long-CoT diffusion language model.
INTRODUCTION
To apply diffusion models (Ho et al., 2020; Song & Ermon, 2019) on language tasks, approaches such as projecting discrete tokens into continuous space (Graves et al., 2023; Dieleman et al., 2022: Gulrajani & Hashimoto, 2023), and employing state transition matrices to derive the diffu- sion process (Austin al., 2021; Sahoo al., 2024; Shi et al., 2024; Ou al., 2024) have been explored. Masked diffusion models (MDMs) have emerged as promising and scalable architecture for diffusion language models (DLMs) recently (Nie et al., 2025; Ye et al., 2025). DLMS enable sub- stantial inference acceleration by parallel generation (Labs et al., 2025; Google DeepMind, 2025; Song et al., 2025), and improve consistency through bidirectional attention (Zhang et al., 2023).
Diffusion language models with full attention have been explored in coding tasks (Labs et al., 2025; Google DeepMind, 2025; Gong et al., 2025; Xie et al., 2025) and in fixed-format reasoning tasks (Ye et al., 2024; 2025). Beyond the full attention mechanism, block-attention diffusion models (Ariola al., 2025) scaled up to large language models that excel at complex reasoning tasks (Cheng al., 2025). However, a unified and effective reinforcement learning (RI.) framework applicable to both full-attention and block-attention DLMS remains largely unexplored.
Existing post-training frameworks focus on full-attention DLMs by optimizing the entire sampled sequence with randomly added masks. This approach leads to mismatch with the optimal inference process (Ye et al., 2025; Nie et al., 2025; Gong et al., 2025; Yang et al., 2025b), since the sequen- and logical nature of language is not purely random (Arriola et al., 2025; Gong et al., 2025). We first demonstrate the importance of aligning the inference trajectory with the training objective
"quual Contribution. Contact: yangling08 18@ 163.com "orresponning Author

---

Published as a conference paper at ICLR 2026
through empirical studies, and then propose trajectory-awart reinforcement learning method, Trac- eRL, which is applicable on both full-attention and block-attention DLMs. TraceRL demonstrates stronger optimization performance than baseline methods across diverse DLMs and tasks. Notably, optimizing the math reasoning ability of a 4B model yields a 5.4% improvement in accuracy on MATH500, outperforming Qwen2.5-7BBInsttee that reduces variance and enhances training stability. Furthermore, we show that this value model naturally accommodates the process rewards, providing stronger reward supervision.
We also introduce diffusion-based value model
Trained solely with TraceRL, we obtain 4B and 8B state-of-the-art diffusion language instruction models on reasoning tasks, both surpass strong autoregressive models on math reasoning bench- marks. Furthermore, we develop the first 8B-scale long-CoT diffusion language model, TraDo-8P --BB- Thinking, by combining TraceRI. with long-CoT SFT.
We summarize our main contributions as follows:
We demonstrate the importance of aligning the training objective with the inference tra- jectory through empirical experiments. Building this insight, propose TraceRL, araeectory-aware RL method applicable to both full-attention and block-attention DLMs.
We introduce diffusion- stability via token-wise baseline and GAE-based targets optimized using a clipped regres- sion objective. This value model also naturally accommodates process rewards, providing more nuanced reward supervision.
value model that reduces variance and enhances training
The superiority of TraceRL is demonstrated across multiple architectures and tasks. Opti- mizing 4B model on math tasks yields 5.4% higher accuracy on MATH500, surpassing Owen2.5--BBInntreue while optimizing 8B model achieves 7.4% higher accuracy on LiveCodeBench--22 TraceRI. also scales block diffusion models to larger block sizes.
Using TraceRL alone, we obtain state-of-the-are diffusion language models, the TraDo series. TraDo-81 BulDo-8B-Iie Instruct achieves 6.1% % higher accuracy than Qwen2.5-7B-IInsee IInsructt on MATH500 and 5. 5.9% higher accuracy than ---ma-..1-8B-Inseeee on LiveCodeBench- V2. Furthermore, through curriculum learning, we develop the first 8B-scale long-CoT DLM.
2 RELATED WORK
2.1 MASKED DIFFUSION LANGUAGE MODELS WITH FULL ATTENTION
Masked diffusion language models (Nie et al., 2025; Yang et al., 2025b) randomly replace a subset of tokens in the raw sample Zo with mask tokens [MASK], resulting in where t E [0, 1]. This process can be formulated as Equation generation, where multiple masked tokens are approximately recovered in one step when transition- ing from noise level to a lower level s < t.
Its reverse process can be accelerated through iterative
g(te mo) = II Cat (r); (1 1)).2 + (θ[MASK]) [=l
has been demonstrated that the training objective of DLMS can be derived from data likelihood (Shi et al., 2024; Sahoo al., 2024; Zheng et al., 2024; Ou al., 2024). Specifically, the optimiza- tion objective is the evidence lower bound of log (x)):
Truu(TO, Q,0)
E [x0)
Σ [MASK]
log Pogh) Izs Q) dt,
(2)
t t|crol
where Q denotes the prompt part, and |mol denotes the number of tokens in zig.
2.2 ADAPTED MODELS AND BLOCK DIFFUSION MODELS
Adapting from autoregressive model has been shown to be a promising approach for preserving the AR model's capabilities (Ye et al., 2025). This approach is efficient and has led to powerful DLMs (Ye et al., 2025; Gong al., 2025; Xie al., 2025; Cheng et al., 2025).
2

---

Published as conference paper at ICLR 2026
Trajectory example
reward 7
slice data x (L/s)
Jpolicy (8p)
Sampling steps
Policy Bp
likelihood
full?
π
π +r (+V)
only slice within block
Step 3
block?
(8/s)
token-wiss value V
Task
Step 4
Sample & Shrink trajectory
3
step-wise sample
π
Trajectory information
Decoded
Value
Fvalue (9p)
Target
Masked
(Optional)
Trajectory illustration (left) and TraceRL. overview (right, see details in Section 4). Right panel is an example of TraceRI. with shrinkage parameter 2, sequence length I, = 6, and block size (when using block diffusion). We aggregate consecutive steps to perform trajectory- aware reinforcement learning. Integers inside the squares indicate trajectory information.
Figure I:
Block diffusion models (Arriola et al., 2025; Cheng et al., 2025) employ block-diffusion atten- tion mechanism that combines the training efficiency of autoregressive models with the sampling efficiency of diffusion models. Their block attention naturally supports K KV-cache, mitigating the performance degradation often observed with KV-cache in full-attention DLMs (Hu al., 2025b; Liu et al., 2025; Ma et al., 2025; Arriola et al., 2025; Yu al., 2025b; Wu et al., 2025).
2.3 RL FOR DIFFUSION LANGUAGE MODEL
Reinforcement learning methods for full-attention models have been explored in prior work (Yang al., 2025;; Gong al., 2025; Zhao et al., 2025). In MMaDA, Yang al. (2025b) introduce random masking for each rollout sample and optimize a PPO objective for training. Huang et al. (2025) treats each intermediate step the reverse diffusion process latent thinking action and optimizes the entire reasoning trajectory. Gong al. (2025) propose augmenting each rollout with both single noise sample 3y and its complementary counterpart Tt, thereby doubling the effective training data. They provide theoretical analysis showing that this strategy reduces variance, yielding faster and more stable optimization than random- aandom-masking RL.
3 MISMATCH BETWEEN TRAJECTORY
OBJECTIVE AND INFERENCE
Pretraining under the fully random masking objective (Equation 2) enables parallel decoding. How- ever, natural language inherently depends on context. The widely adopted decoding strategy and KV-cache techniques create mismatch between the post-training objective and the model's infer- ence behavior. We present a simple experiment to illustrate this mismatch in this section.
3.1 INFERENCE TRAJECTORY
Strategies have been proposed to improve sampling quality. Given a masked sequence 2e, we select [MASK] tokens unmask based on confidence at each position i, defined as the maximum predicted probability max,; Po (xi Fz, Q). A common strategy is to unmask either a fixed number of tokens with the highest confidence (Chang et al., 2022; Kim al., 2025), referred as static sampling or all positions whose confidence exceeds a threshold T (Yu et al., 2025b; Wu et al., 2025), referred as nymimic sampling. with the latter often faster. By iteratively applying this procedure starting from fully masked sequence, we eventually recover fully unmasked sequence.
3.2 SEMI-AUTOREGRESSSIIE FINE-TUNING
Block diffusion models naturally employ semi semi-autoregressivee fine-tuning with T (z, Q, 0) (Equa- tion 7), which trains the model generate later tokens conditioned earlier context, while preserv-

---

Published as a conference paper at ICLR 2026
Table I: We explore how effectively different methods tune the model to learn CoT reasoning and thereby improve reasoning accuracy under non-CoT prompts. 2000 datapoints were generated using Qwen2.5 models and filtered for quality. 1 denotes the length of each step in the complete trace. "xm" indicates that we apply independent random maskings to augment the dataset for fair comparison. "Token forward" denotes the number of tokens processed by the model, representing computational load/time. "Token trained" refers to the number of tokens directly contributing the optimization objectíve. We report accuracy on MATH500. The block-attention model used here is SDAR-4B-Chal (block size of 4), and the full-attention model is Tream-7B-Instruet.
Full Attention
Block Attention
Token Utilization Efficiency
trace
semi-ar (B 16)
fully random
trace
semi-ar (B
1 = 16
x2
x1
x35
x1
1=
2
x2
x1
accuracy (%)
54.4
53,4
52.6
45.1
39.6
71.3
70.4
70.0
# token forward (M)
39.2
78.4
39.2
39.2
1.1
4.5
4.5
2.2
# token trained (M)
1.1
1.1
0.6
20.0
0.6
1.1
ing the sampling efficiency of DLMs. For full-attention DLMs, applying the semi objective inevitably increases forward passes by about [T/B]] B] (see Figure 6). Nevertheless, the optimization performance of Tsem (I, 0) is substantially better than that of Jun((r, (z. Q, 0) under [L/B] B3 independent repetitions with the same computational load (see Table 1). This demonstrates the importance of aligning the post-training objective with the general left-to-right inference pattern.
3.3 ALIGNING POST-TRAINING WITH PREFERRED INFERENCE TRACES
To further investigate, we collect the pre-trained model (DLM))s own preferred inference traces and use them for fine-tuning. Specifically, we apply static sampling (see definition in Section 3.1) 1) to obtain the optimal trace for each data point, which then used in our fine-tuning (see Appendix D.1 for details). Table shows that using the model's own preferred inference traces achieves superior performance compared to the baselines, even when the computational load equal to or lower than that of the baselines, for both block-attention and full-attention models. A major limitation of this fine-tuning approach is that collecting inference traces is computationally and practically costly. In contrast, reinforcement learning naturally produces these traces during rollouts, making practical and effective strategy for post-training. Accordingly, we propose TraceRL. in the following section.
RL TRAINING WITH TRAJECTORY
Current RL methods for DLMs focus on full-attention models, reward or penalize rollout responses based on the overall sequence generated (Zhao al., 2025; Yang et al., 2025b; Gong et al., 2025) using the random masking objective Julll (Equation 2). We propose TraceRL (Figure 1, detailed pipeline in Appendix D.7), which instead focuses on the intermediate traces generated by the DLM and can be applied across different architectures. We also introduce diffusion-based value model, which improves training stability and naturally accommodates process rewards.
For each generated response t; given the task Q, we present it in the trajectory form T, (1) U ... U T: (IT)))) where |T&l is the number of decoding steps, and (t()) is the set of tokens decoded during the t-th step. TraceRL rewards or penalizes the sampling trajectory under policy ge. based on the verifiable reward r; assigned to T,. Process rewards can be incorporated via value model.
4.1 ACCELERATED TRAINING WITH SHRINKAGE PARAMETER
For full-attention models, decomposing the sequences based on each sampling step leads large number of forward passes during training. To address this, we introduce a shrinkage parameter 8, which aggregates every consecutive steps to improve training efficiency. Formally, compress
the trajectory T; into T - (1) U U (1TT where (k) e (min(sh//t)))) (T( and |T,"
4

---

Published as a conference paper at ICLR 2026
[|r///s], then optimize the following objective:
In!1 [ΣΣΣ]
THe (ok T. :1 ((-1)))
J Jpalicy (0p) E
Dask
Col
Fold Ok T/² (t- 1))) A)/
-KLL,
(r 1 (Q)
i=1 t=1 ONE
(3)
where Cr, A) - (in(rr,,llpprr (,, (f1:: - U}=1;((), Toid is the old policy, A is the standardized reward, and KIL = [Tall
By introducing the shrinkage parameter 8, we reduce training complexity by a factor of This allows Fpolicy (8p) to be used for efficient RLVR (reinforcement learning with verifiable rewards).
4.2 VALUE MODEL WITH DIFFUSION MODELING
Rather than assigning a single sequence-leve advantage to all tokens in a completion, value func- tion enables token-wise advantages conditioned the prefix (e.g., GAE-based estimates), providing a variance-reducing baseline that typically stabilizes policy optimization (Schulman et al., 2017; Hu et al., 2025a). We adopt diffusion-basee value model to estimate step-wise values. For each rolled-out trajectory т (possibly shrunken by a shrinkage factor 8), we retain the notation 5 for brevity. We evaluate frozen value network Vala obtain token-wise values along the trajectory: yoldd (Vai(T)) J E Concretely, for each trace step condition the prefix 7(1:t 1) and predict the values for all tokens generated at that step, i.e., Vy :j E (t) }. We then define the step-wise value at step as V ,old Ejer(o) Vi /lr(t)) Vyald is treated as a stop-gradiens baseline when constructing returns and advantages; the learnable value network Vo, updated only via its own regression objective.
We now derive the training objective. Given token-wist rewards Tj, we first form the step-wiss wise reward rī Ejer(o r//]=(()]. We compute step-wise returns recursively as R2 = rt RRi4+ for t < |T| with terminal condition Ri = 0. Using the frozen baseline, define V.r.old Ejer()) //00///(()))) the TD residual is 87 r V, + VV and the step-wise GAE is A; 8 7 We then map step-level quantities back to tokens as R3 ry + R and A, (r, Vy + VV + >AA4 where is such that E T(z)). The value network is trained using clipped regression loss:
Jealue E,
Σ
max (T) R2,2, (V R.)3)]
[
jET
where yelip- = +cipp (T) update the policy. The explicit forms of A, and the theoretical analysis are provided in Appendix A.
(T) -e, e). The token advantages A, are used in Jpolicy (0p,) to
4.3 SLICED TRAINING IN BLOCK DIFFUSION
Block diffusion uses block attention for efficient supervised fine-tuning, and we extend this to re- inforcement learning. For each processed trace generated by biock-wise inference, we write
(......,(1+1//5)) where each block is by (Tr.1...., Tk((bu)) with [by] steps, and B is block size. The training objective can then be sliced from Σ f(r(i)) into a B' training slice, B¹ [B/s]. Each slice requires one forward pass with block attention (Figure 1). This formulation max- mechanism and enables highly parallel and efficient training, for both policy and value models, significantly more efficient than full-attention training.
[r///B1 f(TK.)) ((Tma))))) where f is task-specifie function and B' max { [bx] } <
imizes the utility of the
5 EXPERIMENTS
In this section, we demonstrate the superiority of TraceRL across diverse tasks and models, as well as the advantages of incorporating value model, including enhanced training stability and the natural
5

---

Published as a conference paper at ICLR 2026
Table The main benchmark results across different math and coding tasks. "Static" denotes static sampling, and "Dynamic" denotes dynamic sampling. The long-CoT model Me-DD--B-Inntte is evaluated using dynamic sampling with threshold 0.9.
Model
MATH500
A1ME2024
GSMSK
LiveBench
Autoregressive Models
Dlamad 1-8B-Iasiract
S1.9
6.7
8A.5
20.0
19.7
(weon2.5-78-lnssmee
74.0
8.2
99.9
26.9
31,1
Diffusion Language Models
Statie
Dynanic
State
Dynamic
Statie
Dynanic
Statie
Dynamic
Statie
Dynonic
ULaDA-BB 1DDA-88--aa Innssrrcs
17.3
18.3
033
1.7
82.3
82.3
53
5.3
4.3
6.0
Dream-7B-lnstrue
10.7
SDAR-BB-CCA
702
67.4
8.2
902
88.9
15.6
11.2
14:0
7.6
IIInstructt
75.6
71.8
8,3
10.3
91.2
90.3
18.7
15.1
12.9
10.4
SDAR-SB-Char
74.3
70.7
11.8
8.3
90.4
18.5
15.3
11.2
TraDo-SB-Instruuc
14.3
75.5
44.
13.
13.3
+2.7
13.3
40.8
67.4
47.
22.7,11.2
49.4
TraDo-aB-Thhille Thinking
87.4,133
-5.5
94.2
-3.1
34.6.161
36.0.33.
ability to accommodate process rewards for trajectory-wise supervision. We present comprehensive evaluation results for our ofate-of-the-ar TraDo models and highlight interesting applications such block size enlargement. More ablation studies are included in Appendix C.
5.1
EXPERIMENTAL SETUPS
5.1.1 DATA
We use different data sources for reinforcement learning. For the math tasks, we choose the MATH training dataset (Hendrycks et al., 2021) and retain only level 3-5 tasks (Hu et al., 2025a), result- ing in 8K hard tasks. For the coding RL setting, use 6K verified problems from PrimeIntellect (Jaghouar et al., 2024), validated by DeepCoder (Luo et al., 2025), as coding dataset in stdio format. In addition, we randomly sample 5K problems in the functional-test format from the Open- Coder training set (Huang et al., 2024).
Our evaluation focuses on reasoning tasks in mathematics and coding, For mathematics. we use GSM8K (Cobbe et al., 2021), MATH500 (Hendrycks et al., 2021), and AIME2024 (Mathematical Association of America, 2024). For coding, we use LiveCodeBench--V V2 LiveBench (White et al., 2024).
(Jain al., 2024) and
5.1.2 MODELS AND OPTIMIZATION
Our experiments include both full-attention and block-attention models. For full attention, we use Dream-7B Instruct (Ye et al., 2025) the base model, and for block attention, use the SDAR (Cheng et al., 2025) series of models, trained with a block size of 4.
We now describe our reinforcement learning settings. For the block-diffusion model, at each sam- pling step we sample 128 tasks and 32 responses per task, using a dynamic sampling strategy (T = 0.9) and temperature 1.0. For the full-attention model, during each RL sampling step sample 56 tasks and generate 8 responses per task from the policy лo with KV-Cache, with a tem- perature of 1.0. During training, we set the learning rate of the policy
0.2 and 3 = 0.01. When using the value model, we set the RL learning rate to 5 x 10-6, and use Y = 1.0 as default values. Our 4B and 8B instruction models, TraDo, based on SDAR-4B. Chat and SDAR -B 3B-Chat, are first trained on math tasks to convergence, then on coding tasks to convergence, and finally retrained on math tasks. Additional optimization details are provided the Appendix D.4 and D.5.
10-6 6 with
5.1.3 EVALUATION
For the block-attention model, we use temperature f = 1.0 with dynamic sampling and greedy decoding (top-k 1 for static sampling), with default block size of 4. For the full-attention model, we use temperature = 0.1 and block size 32 for static sampling. For the LLaDA model, adopt a block size of 32 for dynamic sampling, whereas for the Dream model, use block size of for dynamic sampling. Additional evaluation details and prompt templates are provided in Appendix D.3 and D.2.

---

Published as a conference paper at ICLR 2026
48 model
model
0.76
0.78
0.77
more accurate
0.74
0.76
MATH500
0.75
0.74
faster
Qwen2.5-1 5-78-Instruct
Dynamic Sampling
Dynamic Sampling Static Sampling
Static Sampling
step
step
Figure 2: The TraceRL training curves for the 4B and 8B models the math task. The red curve denotes the dynamic sampling accuracy, which yields faster sampling speed, while the blue curve denotes the static sampling accuracy, which yields higher accuracy. The 4B model is trained with value model, whereas the 8B model is trained directly using Jpolicy.
Dynamic Sampling Acc
Static Sampling Acc
0.76
0.75
0.72
MATH500
0.70
0.69
TaceRL
0.68
0.70
coupled within block "endom wkthin block
0.67
0.69
60
120
1400
160
60
120
140
160
steps
steps
Figure 3: RL method ablations block diffusion models for math RL tasks. The red and yellow curves represent TraceRL. with and without a value model, respectively. The blue curve corresponds training with random masking objective in each block, similar the semi-autoregressive ap- proach. The green curve represents training with an additional complementary mask within block.
5.2 INSTRUCTION MODELS TRAINED WITH TRACERL
We obtain TraDo-4B-Ima In-Itrut and TraDo-SB-Insttrrn by applying TraceRL to math and coding tasks, starting from the SDAR base model. We evaluate these models math and coding tasks, well as five reasoning datasets, and compare them against both strong diffusion language models (Cheng et al., 2025; Ye et al., 2025; Nie et al., 2025) and autoregressive models (Yang al., 2024a; Dubey et al., 2024) (see Table 2). Our instruction models achieve state-of-the-art performance on reason- ing tasks among open-source diffusion language models under both dynamic and static sampling. highlighting the effectiveness of TraceRL. Notably, the 4B model yields a 5,4% accuracy gain on MATH500, surpassing Owen2.5-7B-Instrree across all math tasks. 8B model achieves 7.4% gain on LiveCoddeBench-V2 scoring 5.9% higher than Llama-3.1 1-8B-IInstrul the same benchmark.
5.3 TRAINING DYNAMICS OF TRACERL
We present the training dynamics on math tasks for our 4B and 8B instruction models (see Figure 2). Although dynamic sampling is adopted during RL training, both dynamic and static accuracy im- prove steadily, suggesting further potential for scaling. This RL training substantially enhances the models' math reasoning ability: MATH500, MaaDo-4--Instrus improves by 5,4% (static) and 4.2% (dynamic), surpassing Qwen2.5-7B. 7B- after optimization, while TraDo-8B. 80---BtBInttt improves by 4.2% (static) and 4.8% (dynamic) (see Figure 2).

---

Published as a conference paper at ICLR 2026
Full Attention (Dream-Coder)
Full Attention (LLaDA)
0.2525
IveCDdeBenchNr
0.2500
0.2475
MBPP
0.35
0,2400
0.2375
coupled masking
0.34
fully random
0.2350
10
40
50
steps
steps
Figure 4: RL training ablations for the full-attention model on coding tasks. (a). Dream-7B-Coded Instruct. (b) LLaDA-SB-Insteee
Training Variance
Advantage of Process Reward
0.750
0.725
0.630
reward
0.675
MATH500
0.850
0.620
0.500
0.615
without
TraceRL with process reward
0.575
with value model
TraceRL process reward
40
60
80
100
120
140
160
step
step
Figure 5: Comparison training with and without a value model. (a) Incorporating a value model reduces training fluctuations during training. This experiment is conducted on SDAR-4B-Chat. (b) value model enables integration of trajectory-level process rewards, yielding faster optimization than relying solely outcome rewards. This conducted on SDAR-1.7B-COE
5.4 STRONGER OPTIMIZATION WITH TRACERL
We compare TraceRL with existing RL methods, focusing first on block diffusion models. Although current RL methods primarily designed for full attention models, adapt them directly to the block setting. For the random masking approach (Yang et al., 2025b), we restrict sampling within each block, making resemble eeorreevoreeressvvve method. For coupled RL. (Gong al., 2025), we introduce a complementary objective within each block, which yields more stable and effective training. We evaluate these methods on math tasks, with results shown in Figure 3. The experiments demonstrate that TraceRL achieves the best optimization performance, both with and without value model. All RL methods use dynamic sampling during rollout, and TraceRL delivers superior optimization under both dynamic and static evaluation. These results highlight the importance of optimizing over the preferred trace, even within small block.
5.5
ALSO FIT FOR FULL ATTENTION MODELS AND CODING TASKS
To demonstrate the broad applicability TraceRL, we conduct experiments with full- atll-attention mod- els on coding RL tasks. For the adapted model, we employ Dream-78B Cerr----roorr-sttueeee first fine- tuning on distillation data to provide cold start and then performing RI. training on stdio-format coding tasks (see Appendix D.5 for details). We use static sampling and set the shrinkage param- eter to 8. For the pretrained DLM, we optimize LLaDA-8B-Instre tasks, applying dynamic sampling with s 1. As in Table two baseline methods (Appendix D.5). We find that TraceRL converges faster and achieves the best performance compared with the baselines (see Figure 4). Notably, achieve 25.0% accuracy LiveCodeBench- V2, establishing a new state the art among open-source full-attention DLMs.
on functional-test coding augment the training data for
5.6 DIFFUSION VALUE MODEL ENHANCES TRAINING STABILITY
Training a policy model coupled with value model provides a variance-reduciio baseline LLM RL training (Schulman et al., 2017; Hu et al., 2025a). We extend this idea to diffusion language models. In our diffusion- based value model, the verifiable reward the entire trajectory is assigned to the tokens in the final sampling step, while the reward for all preceding tokens is set zero. We
8

---

Published as a conference paper at ICLR 2026
Table 3: Adapting block size from B to 8 on reasoning tasks with TraceRL. Reported values accuracies of these baselines under dynamic sampling with threshold 0.9.
Model
inference block size
MATH500
GSM8K
LiveCodeBench
4
67.4
88.9
11.2
Base model with B = 4
60.2
83.0
9.8
Enlarge B to 8 by TraceRI.
67.7
88.7
10.8
then use our proposed modeling to estimate values and advantages, jointly optimizing the policy and value models. This approach reduces variance and stabilizes the training process (see Figure (a)) for the 4B model on math tasks. Specifically, the variance of the reward during training drops from 6.6 x 10- to 3.6 x 10-4 (a reduction about 45.5%). We also apply the same setting to SDAR-1.7B and find that the variance drops from 8.2 x 10 to 5.3 10 35.4%) when using the value model.
(a reduction of about
5.7 VALUE MODEL ACCOMMODATES PROCESS REWARD
Instead of assigning a single reward to the entire sequence, trajectory-level rewards enable more fine-grained supervision by incorporating process rewards for intermediate steps (Lightman et al., 2023). TraceRL leverages iiffusion-based value model to integrate such process rewards. In this ex- periment, use SDAR-1.7 1...--.BBhha the base model for both the policy and the value networks, and employ Qwen3-4B as the process reward model, which provides rewards for intermediate segments of 200 tokens in addition in this experiment. Compared with using only the verifiable reward without a value model, incorpo- rating process rewards yields more effective optimization (see Table (b)).
overall verifiable reward (Appendix D.6). We (7, A) (0.99. 1)
5.8 SCALING BLOCK SIZE WITH TRACERL
A block size of limit inference flexibility, and some acceleration methods rely larger block sizes (Hong et al., 2025; Song et al., 2025; Wang et al., 2025a). Therefore, we therefore explore using TraceRL increase the block size. We first perform rollout with block size of B 4. then apply TraceRL with B = 8. After 60 steps, we switch the rollout to B = 8 and continue optimizing with for 40 steps, yielding model better adapted to larger block sizes. As shown in Table 3, this approach proves effective across different tasks, despite being trained only on math tasks.
5.9 LONG-CoT DIFFUSION LANGUAGE MODEL
The long-CoT diffusion language model TraDo-8B- -hinnking is derived from TraDo 8B-Instruce After obtaining TraDo-8B Ba--sB---Innttttee through RL training. we further apply scooeeuttoeeressivve fine- tuning on 75K samples of OpenThoughts-3 dataset (Guha et al., 2025). As the first long-CoT dif- fusion language model, benchmarks (Table 7), achieving 85.8% accuracy MATH500 (Table 2) and demonstrating that diffusion language models can also excel at complex reasoning tasks.
exhibits strong long-CoT reasoning capabilities across
5.10 ANALYSIS ON ACCELERATION RATIO AND RESPONSE LENGTH
We observe that the acceleration ratio (Table 4) increases after TraceRL. optimization. In particular, the 4B model achieves 15.4% speedup MATH500. This can be explained by the model becomes more confident when encountering problems in the domain it has been optimized for, allowing each likely unmask more tokens under the same threshold. We also
step of dynamic sampling observe that the average response length increases on most complex reasoning tasks.
5.11 ALTERNATIVE PARAMETERIZATTION FOR VALUE MODEL TRAINING
We conduct ablation studies on the choices (y, A). The configuration (y, A) = (1, 1) serves as the standard setting for value model training (Hu et al., 2025a). We additionally evaluate () (1, 0) and (y, A) = (0.99, 1) using the SDAR-L.7B 7-Chht base model on math tasks with verifiable rewards under T.aceRI. As shown Table the results remain robust across these parameterizations.

---

Published as a conference paper at ICLR 2026
Table 4: Acceleration ratio and response length analysis. "Acceleration" defined the ratio of response length to the total sampling steps for a task under dynamic sampling, and we report the average ratio across the evaluation dataset. We use dynamic sampling with a threshold of 0.9 here.
MATH500
LiveBench
Model
accelerated
total step
avg len.
accelerated
total step
avg len.
SDAR-4B-Chat
2.28
240
548 tok.
1.56
119
181 tok.
TraDo-4B-Instun
2.63
229
595 tok.
1.61
154
238 tok.
ADAR-8B-CHI
2.42
228
557 tok.
1.56
161
256 tok.
Innstruutt
2.50
240
625 tok.
1.60
151
233 tok.
Table 5: Accuracy on MATH500 for different (7, A) after 60 training steps with the SDAR-1.7B
Base Model (0.99, 1.0)
(1.0, 1.0)
(1.0, 0.0)
MATH500 Accuracy
61.1
63.2
63.0
63.3
5.12 ABLATION STUDY SHRINKAGE PARAMETER
We evaluate different choices of the shrinkage parameter 8. The RI. experiments are conducted with the LLaDA-8B-Instrus base model on functional-test coding data. As shown in Table smaller values of yield better performance, consistent with our finding that closely following the inference trajectory during optimization improves results. Nevertheless, larger values of still achieve strong performance relative to baseline methods. (Huang et al., 2025), as a special case with 1, can be seen as spending more training time to avoid any bias in training.
Table 6: Training time and accuracy for different after 60 training steps.
Base Model
1
2
4
MBPP Accuracy A 100 GPU hours / step
33.2
37.0
36.7
35.6
3.7
2.6
2.1
5.13 AVERAGE RESPONSE LENGTH OF
We evaluate the average response length of MoaDo-SB-Thinkiee across different evaluation datasets (Table 7). The results show that more challenging tasks, the model produces longer responses with more reasoning steps and tokens. This observation is consistent with prior findings on long- CoT large language models (Guo al., 2025; Hu et al., 2025a; Yang et al., 2025a).
Table 7: Average response length of TraDo-8B-Thinkinn on different benchmarks.
MATH500
AIME2024
GSM8K
LiveCodeBench
LiveBench
Avg. length
5872
19397
1458
16291
16233
CONCLUSION
We present a new reinforcement learning method for diffusion language models with diverse archi- tectures. Extensive experiments demonstrate the effectiveness of this method across multiple RL tasks, resulting in three state-of- of the-art diffusion language models. We also highlight its benefits for accelerating inference and scaling block size, pointing promising directions for future research.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
Our research relies solely on publicly available benchmarks for math and coding tasks, and does not involve human subjects, private data, applications with direct ethical risks.
REPRODUCIBILITY STATEMENT
The code is included in the supplementary material. We also describe the experimental details in Section 5.1 and Appendix D.
REFERENCES
Marianne Arriola, Aaron Gokaslan, Justin Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Sub- ham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autore- gressive and diffusion language models. arXiv preprint arXiv:2503.04:: 09573, 2025.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 34: 34:1790 0777719899999 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11315-11325, 2022.
Shuang Cheng. Yihan Bian, Dawei Liu, Yuhua Jiang, Yihao Liu, Linfeng Zhang, Wenghai Wang, Qipeng Guo, Kai Chen, Biqing Qi and Bowen Zhou. Sdar: A synergistic diffu- sion-autoregression paradigm for scalable sequence generation, 2025. URL https://gith www.co///eAstttr///000
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10..16681 2021.
Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer. Conor Durkan, et al. Continuous diffu- sion for categorical data. arXiv preprint arXiv:221 15009, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
Shansan Gong, Ruixiang Zhang. Huangjie Zheng, Jiatao Gu, Navdeep Jaitly, Lingpeng Kong, and Yizhe Zhang. Diffucoder: Understanding and improving masked diffusion models for code gen- eration. arXiv preprint arXiv:2506.20665 2025.
Google DeepMind. Gemini diffusion. https: http://l.../////.clo////t/oooott wwwglooooggooooooooooooooooooo.o httgg///eeeeooooggg/oooeeeeeeeee sepmind/ sepmind/gemini gemini di difffusin////om 2025. Accessed: 2024-07-24.
Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez, Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023.
Etash Guha. Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, Ashima Suvarna, Benjamin Feuer, Liangyu Chen, Zaid Khan, Eric Frankel, Sachin Grover, Caroline Choi, Niklas Muennighoff, Shiye Su, Wanjia Zhao, John Yang. Shreyas Pimpalgaonkar, Kartik Sharma, Charlie Cheng-Jie Ji, Yichuan Deng, Sarah Pratt, Vivek Ramanujan, Jon Saad-Falcon, Jeffrey Li, Achal Dave, Alon Albalak, Kushal Arora, Blake Wulfe, Chinmay Hegde, Greg Durrett, Sewoong Oh, Mohit Bansal, Saadia Gabriel, Aditya Grover, Kai-Wei Chang, Vaishaal Shankar, Aaron Gokaslan, Mike A. Merrill, Tatsunori Hashimoto, Yejin Choi, Jenia Jitsev, Reinhard Heckel, Maheswaran Sathiamoorthy, Alexandros G. Dimakis, and Ludwig Schmidt. Openthoughts: Data recipes for reasoning models, 2025. URL https: / rxiv. org/ abs 6 04178.

---

Published as a conference paper at ICLR 2026
Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood diffusion language models. Ad- vances Neural Information Processing Systems, 16::16633-16715, 2023.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint Xrv: 103. 03874, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Feng Hong, Geng Yu, Yushi Ye, Haicheng Huang, Huangjie Zheng, Ya Zhang, Yanfeng Wang, and Jiangchao Yao. Wide-in, nattow-out: Revokable decoding for efficient and effective dllms. arXiv preprint wrxXiv25077.88777 2025.
Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint vrXiv:2402.06455 2024.
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. gpen-reasoner-zerg, An open source approach to scaling up reinforcement learning on the base model. arXiv preprint (tx.i::5003.2299000 2025a.
Zhanqiu Hu, Jian Meng, Yash Akhauri, Mohamed S Abdelfattah, Jae-sun Seo, Zhiru Zhang, and Udit Gupta. Accelerating diffusion language model inference via efficient kv caching and guided diffusion, arXiv preprint arXiv:2505.2 2025b.
Siming Huang. Tianhao Cheng. Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, J. H. Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models. 2024. URL https: arxivv
/2411. 04905.
Zemin Huang, Zhiyang Chen, Zijun Wang. Tiancheng Li, and Guo-Jun Qi. Reinforcing the diffusion chain of lateral thought with diffusion language models. arXiv preprint arXiv:2505. 10446, 2025.
Hugging Face. Open rl: A fully open reproduction of deepseck-rl, January 2025. URL https:
//github. cww huggingggggg wegggingfacce.. waceeacca//ooocoooooooo open-rl.
Sami Jaghouar, Jack Min Ong, Manveer Basra, Fares Obeid, Jannik Straube, Michael Keiblinger, Elie Bakouch, Lucas Atkins, Maziyar Panahi, Charles Goddard, et al. Intellect-1 technical report. arXiv preprint arXiv:2412.0111
2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07977 2024.
Pengcheng Jiang, Jiacheng Lin, Lang Cao, Runchu Tian, SeongKu Kang, Zifeng Wang, Jimeng Sun, and Jiawei Han. Deepretrieval: Hacking real search engines and retrievers with large language models via reinforcement learning. arXiv preprint arXiv:2503.0020 2025.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training 11ms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095/6 2025.
Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham Kakade, and Sitan Chen. Train for the worst, plan for the best: Understanding token ordering in masked diffusions. arXiv preprint arXiv:2502.06768, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
12

---

Published as a conference paper at ICLR 2026
Inception Labs, Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang. Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, et al. Mereury: Ultra-fast language models based diffusion. arXiv preprint 2ttt::2550.172995. 2025.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Push- meet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- code generation with alphacode. Science, 378(6664)
2022. doi: 10. 10.1126/science.abg 1558. URL https::/ / /www. science. org btrgg//oi. htt///ooo1110111 11t./ science..c......ee abq1158.
Hunter Lightman, Vincet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Zhiyuan Liu, Yieun Yang, Yaojie Zhang, Junjie Chen, Chang Zou, Qingyuan Wei, Shaobo Wang, and Linfeng Zhang. dllm-cache: Accelerating diffusion large language models with adaptive caching. arXiv preprint wxXi::2506006295 2025.
Michael Luo, Sijun Tan, Roy Huang. Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepcoder: A fully open-source 14b coder at 03-mini level. https: httpps:/preetee wwweoepeennnnnnneeeeeeeeeeeeeeee 1e -at-reeee---ooor-a----------
.nction. s t wtt.iionnnttteeeeeeeeeeeeeeeeeee i-Level-1cfal (eto22214666bbbee
44aoa344455555555 2025. Notion Blog.
Xinyin Ma, Rumpeng Yu, Gongfan Fang, and Xinchao Wang. dkv-cache: The cache for diffusion language models. arXiv preprint arXiv:2505.15781, 2025.
Mathematical Association of America. American invitational mathematics examination (aime) 2024: Aime and aime ii.
com/ wiki/ indexx. php /AIME Pr Problems ems and Solutions, 2024. Competition problems used as evaluation dataset; original problems by MAA AMC.
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint 1rXiv:2502.09992 2025.
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint urXiv:2406.037366 2024.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language 2024.
models. Advances in Neural Information Processing Systems,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and general- ized masked diffusion for discrete data. Advances in neural information processing systems, 37: 103131-103161 2024.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, et Seed diffusion: large-scale diffusion language model with high-speed inference. arXiv preprint arXiv:2508.022 193, 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llss. arXiv preprint arXiv:2501. 2025.
13

---

Published as a conference paper at ICLR 2026
Xu Wang, Chenkai Xu, Yijie Jin, Jiachun Jin, Hao Zhang, and Zhijie Deng. Diffusion Ilms can do faster-than-at inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09192, 2025a.
Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, and Bryon Aragam. Scoreflow: Mastering Ilm agent workflows via score-based preference optimization. arXiv preprint S50iv:250020433066, 2025b.
Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving Ilm coder and unit tester via reinforcement learning. arXiv preprint arXiv:2506.03133 2025c.
Yinjie Wang, Tianbao Xie, Ke Shen, Mengdi Wang, and Ling Yang. Rlanything: Forge environment, policy, and reward model completely dynamic system. arXiv preprint (iXiv:2602.02488. 2026.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, al. Livebench: A challenging, contamination- free Ilm benchmark. arXiv preprint arXiv:2406. 29ti::2000.99114.. 2024.
Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleaation of diffusion I1m by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505. 22611 2025.
Zhihui Xie, Jiacheng Ye, Lin Zheng, Jiahui Gao. Jingwei Dong, Zirui Wu, Xueliang Zhao, Shansan Gong, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream-coder 7b, 2025. URL https: / ///kunlpp... github..hh....g.. 1o/blog /20///22////2 dream- coder.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 15Xiv:2412.15115. 2024a.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 00XY:25005.0938,, 2025a.
Chengran Yang, Hong Jin Kang, Jieke Shi, and David Lo. Acecode: A reinforcement learning framework for aligning code efficiency and correctness in code language models. arXiv preprint arXiv:2412.17264, 2024b.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao, Minkai Xu, Wentao Zhang, Joseph E Gonzalez, and Bin Cui. Buffer of thoughts: Thought-auggmntted reasoning with large language models. 2024c.
Advances in Neural Information Processing Systems,
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin Cui, and Shuicheng Yan. Supercorrect: Advancing small Ilm reasoning with thought template distillation and self- correction. arXiv preprint arXiv:2410.0900 2024d.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025b.
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: Discrete diffusion for complex reasoning and planning. arXiv preprint arXiv:2410.14157, 2024.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong, Dream 7b, 2025. URL https: /hwunlppppp..//// gi github. io/ bo/blog / 22025/2222//2 //dreaam
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu. Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint ar0iv:2503. /4476, 2025a.
14

---

Published as a conference paper at ICLR 2026
Rumpeng Yu, Xinyin Ma, and Xinchao Wang. Dimple: Discrete diffusion multimodal large language model with parallel decoding. arXiv preprint arXiv:2505. 16990, 2025b.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrap- ping reasoning with reasoning. In Proc. the 36th International Conference on Neural Information Processing Systems, volume 126. 2024.
Yizhe Zhang, Jiatao Gu, Zhuofeng Wu, Shuangfei Zhai, Joshua Susskind, and Navdeep Jaitly. Plan- ner: Generating diversified paragraph via latent language diffusion model. Advances in Neural Information Processing Systems, 18880188-80990 2023.
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint vrXiv:2504.122/11 2025.
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusson models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. arXiv preprint arXiv:2409.02900 2024.
Jiaru Zou, Ling Yang, Jingwen Gu, Jiahao Qiu, Ke Shen, Jingrui He, and Mengdi Wang. Reasonflux- prm: Trajectory-avee prms for long shain-of-thoughe reasoning in llms. arXiv preprint arXiv:2506.1888 18896, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
A Theoretical Results
16
A.1 Explicit Expressions Token-wise Returns and Advantages
16
A.2 Analysis of the Default Setting
17
B Related Work Extension
19
B.1 Enhancing Reasoning Capabilities of Large Language Models
B.2 Diffusion Models for Language Modeling
19
Sampling Acceleration
19
Additional Experimental Results
19
C.1 Additional Experiments on Different Models
19
C.2 Example of Long-Col outputs
20
Experimental Details
22
D.I Supervised Finetuning Methods Explorations
22
D.2 Prompt Templates
22
D.3 Evaluation Details
24
D.4 RL Sampling Details
24
D.5 Training Details
24
D.6 Process Reward Modeling
25
D.7 TraceRL. A Pipeline
26
Use of Large Language Models
26
A THEORETICAL RESULTS
A. EXPLICIT EXPRESSIONS OF TOKEN-WISE RETURNS AND ADVANTAGES
Proposition 1 (Token-wise return and advantage from step-wise recursions). Let trajectory be partitioned into trace steps r(1)...., T([r),, and let t, denote the unique step index with j E Ttt)). For moken-wise rewards Tj and boken-wise values Volid define the step-wise aggregates
Σ rb
Visold
Viold
T(()]
[(t))
LET(t)
IET())
Let the step-wise return and GAE be given by
R rt +
Rit1.
R
6j r Vitoold +V44
A
Σ
A =0 0, Vi-old 0.
k=
Define token-wise quantities
R₁ rj + 7
==
Voll 7V1,44 + +
16

---

Published as a conference paper at ICLR 2026
Then the following explicit expressions hold:
b
R2 = +
Σ n1, (E((+;+
(I)
[T) + k:)
k=1
|r (72)*
A; = Vilid
E (ET(e)th)
(1t vioid)
(2)
lT(ts + +k)
For the boundary case À 0, we have A₁ =1 Vigid + ty+1-
Proof. Return. Employ R
k== 71-11-1 7h +1+ and substitute into R, = rj + RR to
obtain Rg = r. + E Fritt Using r FoOT Eier(t) r gives Equation 1.
Advantage. Write A +1 Reindex to start at = 1:
->>)+ of +1++ and substitute 87 rr Ver.old + 7V44d
1 (7/)*
k=
1r|-6 Σ
17|-4
A₂ (r3 Voll
(1A)
71 - A)
Σ
( ty
The last line follows from collecting the Vooold terms into a telescoping series whose coefficient is 711 - ()( for step ły + k. Now use rf |-(el Eter(e) Ti and V, | Eier(e Viold and note that 7(1 - ()()A)=11 = (yA)* 1 for O. This yields Equation For = 0, the series vanish and the definition gives the stated one-step TD form.
Remark (Some special cases).
(i) (y, A) = (1, We have
|r|-6
1
Σ
Vold
= rj
rI:
(T(t
k)
k=1
LET(t,++
which is undiscounted Monte Carlo return and advantage with a token-wise baseline.
(ii) (r, A) = (1, The return remains the undiscounted form with 1 above, and
= - Vighd + V.*.old 11+11
which is the one-step TD advantage using the trace-level baseline Vi-old 1t+11
A.2 ANALYSIS OF THE DEFAULT SETTING
We choose discount factor 7 = and TD parameter = as the default setting, which is also commonly used in value models for LLMs (Hu et al., 2025a). We now analyze this choice of modeling diffusion-based value model in the following proposition.
(1, For simplicity, de- note F. = o(T(1:t)). Given the token-wise reward тke we have the objective J(0)
Proposition 2 (Analysis for the special case (7,
T D 1=1 rw/1 (xEr() Define the step-wise score gr Ejent Ve log Tolo; F(-1). Then
VoJ(O) E
Σ gt R
For any mrefux- scalar baselines b, be(F(-1) the estimator of gradient is unbiased:
T
gt R - bi) = VeJ(O).
(5)
Moreover, define jt := EE[ R2
Fr J. we have
E 11 112 R - be)² Fc-1] s 2E 11 (Ri - -me)² [ + 2 Fr (F4 - bz).
(6)
17

---

Published as a conference paper at ICLR 2026
Proof.
Firstly, we have
Σ [T(1]
Σ rz.
kET(t)
tm1
The trajectory density factorizes po(r) II IiRET()) Tolox Fu-1). Using the likelihood- ratio identity.
T
V / r(T)) Vepo(T)
(E
r
Va log Tolo FF-
==1 kEr(u)
Exchange the sums:
T
V Tog Foo1))
V&J = E
kEr(a)
[=1
Fix and split the inner L and
For / u, Fu-1, therefore
Elr Vo log (ok Fu-1)] E V log To(on Fw-1) Fu-1]] =0,
since log (-
F 1) Fu | = 0. Thus only t 2 remain:
V log To Fo-1)
VJJ= E
Σ
Σ ri
kEr(u)
Summing over i E ru)) gives gu. yielding the stated identity, For the baseline, for any bu (Fu-1),
Ligu 6.] Lg FF 6.]
E
E logmoon Fu-1) Fu-il) O,
kEr(u)
so subtracting by leaves the expectation unchanged, which completes the proof of Equation 5. In- equality can be derived by Jensen inequality.
Remark 2 (Link to the policy surrogate with vs. Tolds and to the value loss). (i) Connection to the na/Tou policy surrogate. Evaluating the gradient identity of equation at 0 = Oata and using the standard importance ratio r6) no(o; Fu-11/Tooal Fi-1), the first-order surrogate whose
gradient matches (o is
(P) = EEraa
Σ
Σ
r,(0) (RT - b.)
[())
SEr(E)
Replacing (0) by the clipped ratio Cdrr(()) yields the usual PPO-style surrogate, and adding a trust penalty ( -3 [LL[TT]]TToll
(ii) Connection between the value loss and the variance term. Let the step-shared baseline be the token average b, F Esercer V₁, where each V, Vy(F(-1). With μ4 E[R; Fr-1] and
the token-wise conditional targets μıj == E[R, F F.-11.3 E Tt)). we have
b, = [t(t)
Σ (mg-V,) -V)=
jEr(()
and by Jensen,
(14 bz)2 <
Σ (1y-
[T(t)
JET(()
Plugging this into the step variance term on the right-hand side of equation shows that minimizing the token-level value regression loss Joauue 1 1E (Va. R2 reduces upper bound on
E. [llge11 (R2 b.)2 [e-1]. thereby stabilizing the policy update.
18

---

Published as a conference paper at ICLR 2026
B RELATED WORK EXTENSION
B.1 ENHANCING REASONING CAPABILITIES OF LARGE LANGUAGE MODELS
Approaches to enhance the reasoning abilities of large language models (LLMs), spanning Chain- of-Thuught (CoT) prompting (Wei al., 2022; Yang al., 202ch)-hiich elicits step-by-step ratio- nales and self- oof--improveeett optimization algorithms (Zelikman et al., 2024; Hosseini et al., 2024; Yang et al., 2024d), in which the models iteratively refine their answers through verification, reflec- tion, or voting. Incorporating Long-Co'T methods, such as self-checking and self-reflection patterns, substantially improves the reasoning capabilities of L.MMs. Reinforcement learning (RI.) has also been shown to incentivize the reasoning ability of LLMs (Guo et al., 2025; Team et al., 2025; Hug- ging Face, 2025; Hu et al., 2025a) and to improve their ability to solve complex tasks (Jiang et al., 2025; Jin et al., 2025; Yang et al., 2024b; Wang et al., 2025c; Zou et al., 2025; Wang al., 2025b; 2026).
B.2 DIFFUSION MODELS FOR LANGUAGE MODELING
Extending the iterative-refinement paradigm of diffusion language models (Ho et al., 2020; Song & Ermon, 2019) to discrete language data is nontrivial. One line of research projects tokens into a continuous latent space where standard Gaussian diffusion can operate (Dieleman et al., 2022; Graves et al., 2023; Guirajani Hashimoto, 2023). parallel line directly defines the forward noising process the token simplex via state-transition matrices. Structured Diffusion LM (Austin et al., 1) introduced a categorical forward kernel (c.g., uniform corruption or bit-flip) and trained transformer to invert it. Subsequent work simplified or reparameterized the transition matrix to reduce variance and improve stability (Sahoo et al., 2024; Shi et al., 2024; Ou et al., 2024).
Masked Diffusion Models (MDMs) have emerged scalable architecture for large-scat scale diffusion language models (Nie et al., 2025; Ye et al., 2025; Yang et al., 2025b). By leveraging bidirectional attention, MDMs achieve stronger global consistency than left-to-right autoregressive transformers (Ye et al., 2024; Zhang et al., 2023), while their support for parallel decoding enables substantial inference acceleration (Labs et al., 2025; Google DeepMind, 2025).
B.3 SAMPLING ACCELERATION
Although the parallel decoding strategy of diffusion language models theoretically promises faster sampling than LLMs, current open-source implementations remain slower, largely due to less mature infrastructure compared with optimized engines such as vLLM (Kwon et al., 2023). To address this, KV-cache techniques for diffusion language models have been proposed (Hu et al., 2025b; Liu et al., 2025; Ma et al., 2025; Arriola et al., 2025; Yu et al., 2025b; Wu et al., 2025), and several closed-source systems have already achieved faster sampling than LLMs (Labs et al., 2025; Google DeepMind, 2025; Song et al., 2025).
C ADDITIONAL EXPERIMENTAL RESULTS
C.I ADDITIONAL EXPERIMENTS ON DIFFERENT MODELS
To demonstrate the broad applicability of our method by extended experiments. For Dream, we first fine-tune its instruction model on our sampled 1K demonstration examples using Dream's official SFT setup. and then perform RL for 160 steps with shrinkage value of 8. This yields a 9.4% accuracy improvement on math problems (from 49.2% to 58.6%). For LLaDA, we scale training on OpenCoder to 160 RL steps with a shrinkage value of 2. The OpenCoder dataset filtered so that each remaining problem causes LLaDA to make at least one error given three attempts. Under this setting, we obtain 7.7% improvement on MBPP. For MMaDA, we train with RL on a curated dataset (including GeoQA, CLEVR, MMK12, and GQA) for 60 steps with a shrinkage value of 8. The training dataset is further filtered to remove all binary decision problems in order to avoid spurious reward signals. This improves MMaDA's visual reasoning ability, yielding 5.1% gain on Math Verse (from 23.6% to 28.7%). ) For block diffusion models with different block sizes, we use SDAR-117 1-7B with block sizes of 4, 8, and 16, and RL. the same math training set for 160 steps.
19

---

Published as a conference paper at ICLR 2026
Table 8: Additional experiments on different models.
Model
Task / Dataset
Before RI
After RI
Dream
MATH500
49.2
58.6
LLaDA
MBPP
33.5
41.2
MMaDA
Math Verse
23.6
28.7
SDAR-1.7B-b4
MATH500
55.9
60.9
SDAR-I.7B-b8
MATH500
46.5
52.8
SDAR-1.7B-
MATH500
45.0
52.4
We report the dynamic sampling accuracy before and after RL in Table and observe consistent improvements on dynamic sampling across all block sizes.
C.2 EXAMPLE OF LONG-COT OUTPUTS
TraDo-8B- -hi-nkking Example Output
What is smallest positive perfect cube that can be written as the sum three consecutive integers? response: Okay, so I need to find the smallest positive perfect cube that can be written as the sum of three consecutive integers. Let me start by understanding the problem step by step.
task:
First, a perfect cube a number that can be expressed as n² where n is a positive integer. For example, 1, 8, 27, 64, etc. The question is asking for the smallest one of these that can also be written as the sum of three consecutive integers.
Now, three consecutive integers can be represented x, x+1, and x+2, right? So their sum would be x (x+1) + (x+2). Let me compute that first.
Adding those together: x x + + 3x +.. So the sum of three consecutive integers is 3x +3 3, which can be factored as 3(x + 1).
Hmm, so the sum always multiple of Therefore, any perfect cube that can be written as such sum must be divisible by 3? Because it's three times some integer.
So, the cube must be divisible by 3. Let me note that down.
Therefore, the smallest perfect eube divisible by is 27 (since 33 27), but I need to check 27 can be written as the sum of three consecutive integers. Let me check that. Suppose 3x + 27. Then 3x 24 =2 8. Therefore, the three numbers would be 8, 9, 10. Let's add them up: 8 + 9 + 10 = 27. Hey, that works! So 27 is indeed the sum of three consecutive integers.
But wait, hold on. The question says The smallest positive perfect cube that can be written as the sum of three consecutive integers.If 27 works, then is that the answer? But before jump to conclusions, maybe there's ' smaller cube that also satisfies this? Let me check. The next smaller cube after 27 is 8 (29). Let's see if can be written as the sum of three consecutive integers.
Setting up 3x + =.8. Then, integer, this doesn't work. So 8 is out.
=ú X = 5/3, which not an integer. Since x has to be an
The next is (19). Let's see, 3x + 3 =1 = 3x= -2 = 2/3.. Again, not an integer. So is out.
Therefore, the next cube 64 (49). Let's check that. Wait, but 64 is bigger than 27. Since 27 already works, but maybe there another cube between and 27? Since 8, 27, 64 the cubes after 1, and 27 is the first one here.
Wait, hold on. The cubes are So 27 is the first one after 8. But does 27 itself work? Yes, as we saw.
27, 64, 125, etc. So between 8 and 27 is 27 which works.
But hold on, maybe the question considering three consecutive integers in different way? For example, maybe the integers can be negative? Let me check.
Suppose we allow negative integers. Let's see if there's smaller cube that could ex- pressed as the sum of three consecutive (including negative) integers.
Let's check can't, as above. can't. t. 27 can. 64? Let's see. me check 64.
20

---

Published as a conference paper at ICLR 2026
3x + =4. 3x=61 = 20.333... Not integer. So 125? 3x 125 = 3x 122 =]
== 40.666 nope.
So even with negative numbers, the next cube after 27 might still be 27. But let me think again. Maybe there's s a different way to write three consecutive integers?
Wait, perhaps three consecutive numbers can be written as n-1, n, n+1? That would still be the same as X, X+1, X+2 where n-l. 1. So yeah, same thing.
Alternatively, maybe three consecutive even or odd integers? But that would complicate sum, but the problem didn't specify. The question just says three consecutive integers, so they can be any consecutive integers, whether positive or negative. Alternatively, maybe I made mistake in assuming that 27 is the first one. Let confirm again.
The sum of three consecutive integers is 3x 3(x 1). So the cube must be divisible by 3.
Therefore, the cube must be multiple of 3. The first cube divisible by is 27 (3?). So yes, that must be the case.
Therefore, unless there is smaller cube that is divisible by 3, but I not, 8 is not, so 27 is the first one.
Therefore, the answer should 27. But hold on, me check with another approach to be sure.
Suppose the three consecutive integers are n, n, n+1. Wait, no. Three consecutive integers n+1, n+2. So their sum is 3n + 3. So we need 3(n + 1) to be perfect cube.
Therefore, 3(n + 1) = k), where is an integer. Therefore, n =
Since must be an integer, k must be divisible by 3. Therefore, k must be divisible by 3, because is prime.
Therefore, let k 3m, where an integer. Then:
n 1= (27m))33 9m Therefore, n 1.
Therefore, the three integers would be 9m -1, 9m3, 9m3 +1. Their sum is 9m3 -1 + 9m3 9m3 +1 27m3, which is indeed cube. Therefore, the cubes are all multiples of 27. Therefore, the smallest such cube is when m=1, which is 23 27.
Therefore, 27 is indeed the smallest perfect cube that can be written as the sum of three consecutive integers.
Wait, but let check m=0? If m=0, then -1, the numbers would be -1, 0, 1. Their sum is O, which not a positive cube. So m=1 is the first positive case. Therefore, this seems solid.
Therefore, the answer 27. But hold on, let me think again. The problem says Smallest positive perfect cube.Since 27 is positive and perfect cube, so it's the first one.
Alternatively, maybe there's cube before 27 that can be expressed as the sum of three consecutive integers? Let me confirm again. The cubes are 1, 8, 27, 64, 125...
Checking 1: 3x +3 =1: 3x= -2, nope.
8: 3x +3=8: 3x=5, nope. 27 works, so yeah, 27 the first. Therefore, think that's the answer. Wait, but maybe I made mistake in the equation. Let me recast the problem.
(We omit some reasoning steps here.)
Therefore, 27 is indeed the first.
Therefore, think the answer is 27.
Wait, but the problem says šmallest positive perfect cube: Since 27 is positive and perfect cube, and there are no smaller cubes that work, then yes, 27 is the answer. Wait, but just make sure, 0 considered? But is not positive. So 27 is the first. Therefore, think can confidently say that the smallest positive perfect cube that can be written as the sum of three consecutive integers 27. #*Final *
27
21

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS
D.1 SUPERVISED FINETUNING METHODS EXPLORATIONS
Semi-AR SFT full attention
block attention
Position
Prompt
Response
Decoded
5
Random Masking
Decompose into sequences
0
1
Mask Token
1
2
3
No Attention
2
5
Attention
Forward about tokens with full attention
Loss
Forward about 2L tokens with block attention any
Figure 6: Semi-AR SFT for full attention and block attention model Overview. This example uses block size of B and sequence length of L 6. Block diffusion models naturally adapt to semi-AR SFT efficiently, whereas full attention models require slicing the data by B.
In the demonstration of Section 3, the training data consist of CoT responses generated by the Qwen2.5-32B-Instmnn model (Yang et al., 2024a) on 2,000 randomly selected tasks from the OpenR1 -MATH training set (Hugging Face, 2025), filtered to exclude problems solvable by the 1ann2.5--B-Insrruu model. We then collect the traces of these data using the model under eval- uation. For each instance, the model used to iteratively select the two tokens with the highest confidence, conditioned all previously processed tokens. This procedure produces trace, which then aggregated by grouping every 1/2 neighboring tokens to obtain the final trace, where each step has length I. We train the collected data for one epoch using 64 A100 GPUs, with learning rate of x 10 for Dream and 1 x 10-5 for SDAR. The semi-AR AR training objective is:
[L/B]
Tsemi (z, Q,0) = Σ
Trall (a (6- (6-1)):mi((BBB)) 1Q, 0:(1- (14-))11,0)
1=1
D.2 PROMPT TEMPLATES
Math Prompt Templates
Dream
stttinsst//ssssteellllllt Dsystem/nyou are a helpful assistant, dlaistaant((inneedd>llee need to put your final answer in Vbooxed (). This is the problem: And (problem) }<| in
LLaDA
***
(rttraatooftee//xlllaaaattttttt [ng i> need 4 to put your final answer in [boxed(). This is the problem: In |ottt><]set
4 (Iproblem)
assistant< header idj>\n
TraDo (non-thinking)
< im start Start/Puse >user\n/ (problem) AnPlease reason step by step, and + put your final answer within \boxed() Annend|\Ann ii__tart "n
22

---

Published as a conference paper at ICLR 2026
TraDo (thinking)
<imm aallimmsstarrt start/ >user'nYou need to put your final answer in Aboxed II. This is the problem: /nl (problem) |<| im end/>\n<| iim_strtt/
9 assistantsthinb>
TraDo (non-CoT)
< < im [em__mmsaaaaas start >user need to put your final answer in Aboxed 4 (). This is the problem: And (problem) i im__nd|>>nde assistantin!!'
Code Prompt Templates
Dream
<
iimisssattiiiiiiiiieeise_ttsiitt Saaatt//systemenYo are a helpful bebisstant<<in_eeddd S im start buser)aThis is the | (probblm) AnYou should put your code in "'python "., Use input to read input and print () produce output in your script. <|im_end end innstart//aassstaan
LLaDA
is the problem: /n( (probleem You should put your code in '.. python "., Use input to read input and print () to produce output in your script. <leot_id|>> id|>< startofttttt
TraDo (non-thinking)
" #r<liinstarrt >user\nthis is the problem: (problll em) \nYou should put your code in "pythhn "', Use input () to read input and print () to produce output in your script. <|im_end/ end| >>n<] im starttrt/ stsssst/assst
TraDo (thinking)
< jim_start >user\n/his is the preblem: (probllllee))) em) AnYcu should put your code in "'python "", Jse input to read input and print () to produce cutput your script. <lim_end [>An<] start "assssstanttthinkkk.
Process Reward Prompt
Process Reward (grading excerpt)
" { stssstttttssystseeea >system\nYou are helpful jssistant.<|in_ende < /tste///es the question below, you will be given entire solution (we have already known it ([if_correct)]) and an excerpt (a middle part) from it. Your task is to grade the excerpt for correctness using 1 (correct) or -1 ( incorrect), \n'nRules: An- Judge ONLY the excerpt's correctness given the question and the whole solution (which is !! [=_correct) as context. An- The excerpt may be truncated at the beginning or end due to chunking: de NOT penalize boundary truncation. An- If the excerpt contains any error, If it is completely correct, score 1n Ignore miner grammar issues that do not affect correctness. \n\nThis is the question: \nf (osstion))\nnnhh is the final ground truth answer: Anff gt_answer) \nAnThis is the given solution ([(if_correct] AAn
unjustified inference,
contradiction, score
23

---

Published as a conference paper at ICLR 2026
(whooe_ssoutte /tt.oo/ttito//aaaaaaaaaaaaaa AahiTTii is the excerpt need you to score (e((eeeerp / Vn need to put your final score in Vbooxed
(0 bic]]im_end>>>e iineiniiinnnssttrrt in_start passsssstssstttttttiiiiin
We use the same prompt template for both reinforcement learning and evaluation, with TraDo and SDAR models sharing identical templates. For long-CoT mode, we adopt the "thinking" prompt, while the non-CoT prompt is used only for SDAR in the toy experiment of Section 3, which evaluates the efficiency of different SFT methods in teaching the model to use COT. Prompts align closely with model preferences and minimize formatting errors (e.g.. failing to extract the final answer). For the process reward model, set the temperature to 0.6 and the maximum generation length to 4096 tokens.
designed to
D.3 EVALUATION DETAILS
For the LLaDA model, we use block size of 32, a response limit of 1024 (with no output truncation observed), temperature of 0.1, and further horizon size of 128. The further horizon size is defined as the number of tokens forwarded beyond the target block, excluding the tokens within the block itself. We find that this additional window mitigates the performance drop caused by employing the KV-cache. For dynamic sampling, we set the unmasking threshold to 0.95.
For the Dream model, also problems, the response limit is set to 1600, and for coding problems, it is set to 1024. When using dynamic sampling. we use block size of 4 with threshold T 0.95. Under static sampling, we use block size of 32.
temperature of 0.1, and further horizon size of 128. For math
For the SDAR and TraDo instruction models, we keep the pretrained block size of 4, response limit of 2000, and a temperature of 1.0. With dynamic sampling, we use threshold of T = 0.9 and top-k 0 (i.e., all tokens kept). For static sampling, we top-k 1, following (Cheng al., 2025). For the long-CoT model MaDDo-B-Thinning we set the response limit to 30,000 during evaluation, and only use dynamic sampling for evaluation.
For AIME2024, we evaluated each problem 20 times. For all other test datasets, we evaluated 3 times and report the average accuracy.
We use the K V-cache in all evaluations to accelerate inference. For full attention mask models, we adapt and improve the fast-dllm framework (Wu al., 2025). For block diffusion models, we use jetengine (Cheng al., 2025) to achieve acceleration.
D.4 RL SAMPLING DETAILS
We report the parameters used in our RL experiments. For full-attention diffusion models, we set the block size 32, the temperature to 0.8, and the further-horizon size to 128. At each step, we sample 56 problems with KV-Cache enabled, generating 8 responses per problem. For LLaDA, we adopt dynamic sampling to accelerate optimization with response limit of 256. For the Dream model, we employ static decoding to improve sampling quality (Gong et al., 2025), with response limit of 1024. For SDAR models, we use a block size of 4, dynamic decoding with threshold T = 0.9, top-k 0, temperature 1.0, and top-p 1.0 (also applied during evaluation). At each step, sample 128 problems and generate 32 responses per problem.
D.5 TRAINING DETAILS
For full attention models, we use a learning rate of X 10-6 for both semi-autoregrrssive and fully random masking fine-tuning methods. For block attention models, we use learning rate of 1 x 10 6, The masking probability is uniformly sampled between 0.1 and 0.9. Before RL training in figure 4(a),
collect 1.7k random SFT training samples from CodeContest (Li et al., 2022), with --enn.5-32B-nsstruu model. We find that including the eos token the SFT data is beneficial for stabilizing RL training.
solutions generated by the Qwen2.5-32B Instruct
24

---

Published as a conference paper at ICLR 2026
During RL training, we set the learning rate to x 10-6, with 3 = 0.01 and c 0.2. When using value model, we find no significant difference between the common parameter choices (Section C). By default, use the k estimator for KL. For math tasks, we use binary outcomes verifiable rewards and retain only those tasks with accuracy between 0.2 and 0.8 for training (Yu et al., 2025a). For coding tasks, we as the reward the proportion of unit tests passed by the generated solutions. To accelerate training, we use 64 A100 GPUs for our demonstration and ablation experiments. However, all experiments can be conducted on 8 A100 GPUs.
In figure 4 (a), for Trace RL we use a shrinkage parameter of - 8, together with an average response length of 380 during the RL process, resulting approximately 380/8 47.5 50 forward passes per data point. We augment the training data for fully random masking and coupled methods by applying 25 independent random masks, yielding 50 training samples per data point for the coupled method and 25 training samples per data point for the random masking method. Following Gong al. (2025), we keep the number of training samples for random masking at half that of the coupled method. Similarly, in Figure 4(b), the average response length is 60 and the average acceleration ratio (see Table is 2.0. Accordingly, we augment the training data for the coupled and fully random masking methods by factor of 15.
One noteworthy point is the number of pad tokens that need to be trained for each data point, which denote as Appad- Setting large ?ppad Ieads to excessively large logits for the pad token and can cause the model to terminate inference prematurely. In our RI. training, we set Uppad O, which works well. However, in our long-CoT SFT step, find that setting Nppdd 0 can potentially cause the model to never stop generating output during inference, although one remedy is to add an cos token the stop token. Therefore, choosing an appropriate ppd is vital for stable training.
D.6 PROCESS REWARD MODELING
In experiments trajectory process rewards, use Qwen3-4B to assign rewards for each 200-token segment. Specifically, we divide the model's response into excerpts of length 200 and query Qwen3-4B three independent times using the prompt described in Section D.2, with each query producing a binary score (+1 or -)). all three evaluations agree, we adopt the majority score the reward; otherwise, we fall back to the verifiable reward (+1 or 11) for that segment. After assigning rewards, we normalize them per problem and per segment: for each problem and segment index (e.g., the first 200-token segment), normalize that group jointly; segments are normalized independently. The normalized rewards are assigned the tokens in the last step within each segment.
pool rewards from all generated responses and
25

---

Published as a conference paper at ICLR 2026
D.7 TRACERL ALGORITHM PIPELINE
Algorithm 1: TraceRL (Twaeectory-Aaare RL for DLMs)
Input:
2: 1) Task set Dusk (Q.....QQN
3: 2) Policy TH, parameterized by 0g.
4: 3) (Optional) Value network Vo, parameterized by 0,; flag UseValue E (True, False).
5: 4) Iterations M, rollouts per task G. PPO clip KL coefficient 3.
6: 5) Discount y, GAE parameter X, learning rates (Πp- Me).
7: 6) Shrinkage parameter (aggregate every neighboring trace steps).
8: 7) (Optional) Value update interval E, (update value every E, iterations).
9: Initialize: 0p (and 0, if UseValuee
10: for == to M or not converged do Told < T0 // freeze old policy for PPO ratios & KL
12: if UseValue then
Vola <- freeze copy of Vo, // stop-gradiees baseline
14: end if
Sample rollouts (trajectory traces):
16:
for each minibatch of tasks Q Disk do
17:
for repeat G times do
18:
Generate response by policy decoding to obtain trace ( ((T))), where 7tt) the token set decoded at step
19:
Obtain verifiable reward (or optionally process-level token/step rewards).
20:
Shrink the trace: 7 +- shrink 8), i.e., group every neighboring steps. if UseValue then
Value inference: Voold + (VolaT))) E Ve + [F(0] Σ Vi
22:
23:
Build returns/advantages on the trace:
Calculate step-wise reward: r/. return Rt, advantage A
25:
Map to tokens (j E T(t)): R2 < rú R A₁ + (r, - Vold) + vV V.r,odd
7AAH+1-
26:
(Optional) normalize advantages: A, + normalize (Az).
end if'
28:
Store (Q,7", advantages,Taall for policy update.
29:
end for
30:
end for
31:
Build grouped dataset Derp f- ((Pg Og: Ag, vold)].
32:
for 1 1 to K do
Sample minibatch M C DRrpi compute ratios r,= 3e (0,1Pa)
33:
Toia(o,
34:
Policy step: maximize Tpolley (0p) (Eq. equation 3);
0p 0p + 1p V 8, J
35:
if UseValue and (t mod E, = 0) then
36:
Value step: minimize Tvalue (0.) (Eq. equation 4); 0, - 0, - 71 Vo, Jvalue- 37: end if
38: end for
39:
end for
40: Output: Trained policy TH, (and value Vo, used).
E USE OF LARGE LANGUAGE MODELS
We use large language models solely to polish English writing and improve clarity. All research ideas, methods, and results are entirely our own. All experiments involving large language models and diffusion language models were conducted in accordance with the statements of this paper.
26


Reviews:
[{"review_id": "JG1fBzssQF", "reviewer": "Reviewer_Nrqh", "summary": "This paper introduces TraceRL, a reinforcement learning framework designed to improve the performance of Diffusion Language Models (DLMs) by aligning their post-training objective with their inference trajectory. The authors argue that existing post-training methods, which often rely on random masking, create a mismatch with the more structured, confidence-based sampling process used during inference. TraceRL is a trajectory-aware framework that uses information from the entire sampling process to optimize the model. The paper also proposes a diffusion-based value model to stabilize training and incorporate process rewards. Using TraceRL, the authors develop a series of models called TraDo, which they show achieve state-of-the-art results on several complex reasoning benchmarks for math and coding, reportedly outperforming strong autoregressive models like Qwen2.5-7B and Llama3.1-8B.", "questions": "- Could the authors reevaluate the baselines by applying the same in-domain RL fine-tuning to the autoregressive models? What is the advantage of DLM RL compared to autoregressive RL, given that DLM RL is more demanding in implementation and expensive in training?\n- What is the inference speed of TraDo-8B-Thinking? Does it use static or dynamic sampling? Does it have any advantages in speed or performance, compared to OpenThinker models, which are also tuned with the OpenThoughts dataset?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "- The mismatch between the training objective and the inference process in DLM RL is a well-motivated issue. Aligning the RL training process with the model's actual inference trajectory is logical and well-supported.\n- The authors have conducted comprehensive ablation studies that analyze the contribution of different components of their method, such as the value model, the choice of RL optimization strategy, and the shrinkage parameter.", "weaknesses": "- Marginal Performance Improvement and Extra Value Model Compared to Autoregressive Models. TraceRL improves previous DLM RL methods, but the performance gain through TraceRL is still marginal (4%\\~5% on very simple benchmarks like MATH500). This largely lags behind common autoregressive LM RL recipes like GRPO/DAPO, which could easily bring 10%\\~20% gains for a 7B-scale model on MATH500 without any process reward. Post-training is critical to LLMs, while the significance of DLM RL remains questionable given the marginal improvement.\n- Incremental Novelty. The paper positions TraceRL as a novel framework that, unlike prior work, uses information \"embedded in the sampling trajectory.\" However, most policy gradient RL methods are inherently \"trajectory-aware.\", especially the diffusion RL methods like FlowGRPO/DanceGRPO. The objective function in Equation 3 appears to be a standard PPO-style implementation applied to the steps of a diffusion process, which has already been done in FlowGRPO/DanceGRPO. The novelty seems to lie in the specific application to DLMs rather than a fundamental innovation in RL methodology.\n- Overclaiming in the Title and Narrative. The title uses the word \"Revolutionizing,\" which sets an exceptionally high bar that the contributions, while solid, do not appear to meet. The work does not fundamentally change the paradigm of diffusion RL.\n- Confounding Effect of Long-CoT Fine-Tuning. The TraDo-8B-Thinking model achieves the most impressive results (e.g., 87.4% on MATH500). However, this model is the result of first applying TraceRL and then performing an additional semi-autoregressive fine-tuning step on the OpenThoughts dataset. A significant portion of the performance lift is likely attributable to the long-CoT data, and it is unclear how much TraceRL contributes. Even without TraceRL, the high-quality data of OpenThoughts itself could bring huge performance improvement. Notably, the OpenThinker-v3 7B autoregressive model tuned with the OpenThoughts dataset achieves an AIME2024 score of 69, significantly higher than TraDo-8B-Thinking's 35.5. This further undermines the necessity of using DLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bPvcL20Ofd", "reviewer": "Reviewer_YmJb", "summary": "This paper studies RL methods to reinforce the whole multi-step generation trajectories in DLLM -- this can be applied to both full-attention version and block-attention version. To accelerate training, the actual denoising steps are grouped into fewer actions to reinforce. Moreover, the paper introduce a step-wise value model to estimate an intermediate result to reduce the variance during. The experiments are conducted on TraDo, and achieve higher performance on math and code benchmarks.", "questions": "1. How to calculate advantage in the \"w/o value model\" setting?\n2. If the preference inference traces are all left-to-right ones from Qwen, what is the final generation order preference after our RL training?\n3. Since intermediate steps may contain mask tokens, how can Qwen3-4B be used as a process reward model to evaluate such intermediate outputs?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper provides a method to shrink the trajectory length to reinforce, accelerating training\n2. Introducing value model in DLM RL algorithm is novel, and it is reasonable to evaluate the quality of an intermediate steps.\n3. The paper is well written and easy to follow.", "weaknesses": "1. Why do we only shrink the steps after sampling trajectory? Actually, we can also just sample shorter trajectories by allowing more tokens to be predicted in each step.\n2. The paper does not discuss the effect of shrinking $s$ consecutive steps, both on the final performance and the training efficiency. [1] reinforces the whole trajectory without shrinking. It worth a comparison.\n3. The pretrained model in L178 is actually Qwen as in Appendix D.1, right? It is quite confusing and misleading, since usually such a phrase means the model before post training. Why not sampled from the DLM itself, just like the coldstart phase in Deepseek-R1?\n\n\n[1] Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "05YGlSaprT", "reviewer": "Reviewer_hx5e", "summary": "The paper introduces TraceRL, a trajectory-aware RL framework for diffusion language models (DLMs). It argues that standard post-training objectives for DLMs (fully random masking; semi-autoregressive SFT) are misaligned with the left-to-right inference trajectories used at decoding time, and proposes to (i) aggregate decoding steps into short “trace” segments via a shrinkage parameter , (ii) optimize a PPO-style policy objective over these segments, and (iii) pair the policy with a diffusion-style value model that provides variance-reduced advantages and supports process rewards. The approach improves math/coding benchmarks and can even adapt models to larger block sizes", "questions": "1. The authors are expected to provide an ablation on reward assignment (last-step vs. proportional attribution) and an analysis of how attribution interacts with acceleration ratios/response lengths. \n\n2. It would be better if the authors could give judge prompts/configs and inter-rater stats; run sanity checks (shuffled/adversarial segments) to quantify false positives.\n\n3. The authors are expected to include strong AR instruction-tuned models under matched decoding/compute", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper carefully motivates why training with random masking or generic semi-AR losses can be misaligned with the actual iterative unmasking/decoding procedure (static vs dynamic sampling), and shows a simple analysis/experiment to support this. \n\n2. The algorithm aggregates 𝑠 consecutive decoding steps into segments, computes advantages/returns at the segment (and token) level, and updates a PPO-style objective. This aligns learning signals with how the model is actually used at inference. The paper gives explicit value loss (clipped regression) and describes sliced training under block diffusion to keep training efficient. \n\n3. Extending variance-reduction baselines to DLMs is novel here, with the verifiable reward assigned at the final sampling step and zeros elsewhere; the same machinery accommodates mid-trajectory “process” rewards. Empirically this stabilizes training and accelerates optimization. \n\n4. On math/coding, TraceRL improves over strong DLM baselines; the TraDo models achieve the best results among compared DLMs, with detailed tables.", "weaknesses": "1. For the value model, the entire verifiable reward is assigned to tokens in the last sampling step, with zeros elsewhere. This is simple but may bias learning toward late segments and under-credit earlier reasoning。\n\n2. Process rewards come from Qwen3-4B judging 200-token segments with a fallback to the verifiable reward. The paper should quantify judge agreement rates, false-positive penalties (reward hacking) and robustness to prompt variations. \n\n3. While the paper compares to several full- and block-attention DLMs, it would help to include stronger autoregressive SOTA (e.g., top open 7–8B instruct models) under matched decoding/training budgets. \n\n4. The paper provides explicit forms for token-wise advantages and a clipped value loss, but there’s no convergence or bias/variance analysis for the “trace shrinkage” estimator or the “sliced” block training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nmGQX8BRKa", "reviewer": "Reviewer_bxFJ", "summary": "This paper studies reinforcement learning for diffusion language models. During the learning of diffusion models, one needs to sample noised intermediate samples to learn reconstructions. The authors highlight the importance of using noisy samples from diffusion models' sampling trajectories in training. They also introduce using value model to enhance training stability. The effects are validated on math and coding benchmarks, which surpasses strong autoregressive instruct models.", "questions": "1. Could you show more results on full attention diffusion language models without blocks (i.e., LLaDA, Dream instead of SDAR) ? How about the effect of RL on diffusion language models given different block sizes?\n\n2. How long is the responses of TarDo-Instruct compared to Qwen2.5-Instruct?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Enabling the use of reinforcement learning is an crucial steps to advance diffusion language models. This paper provides timely and systematic investigation into this topics.\n2. This paper validates the effect of several intuitive techniques in RL for diffusion language models, which is comprehensive and helpful for following research, including shrinkage steps, aligning training and sampling trajectories, introducing value models, and sliced training.\n3. The results is superior, matching strong autoregressive language models like Qwen2.5-7B-Instruct on representative math and coding benchmark. As far as I know, this is the first RL diffusion language models to achieve such results on these convincing benchmarks. \n4. To the best of my knowledge, this paper also implement the first diffusion language models capable of long-cot thinking.", "weaknesses": "1. The comparison to autoregressive models is not strictly fair. Strong results are only demonstrated through SDAR, which is a block diffusion language models with extremely small block sizes. This does not convince me on the potential of improving a diffusion language models. And the long-cot models are not compared to long-cot autoregressive models.\n2. The results do not support the claims about benefits of introducing value models. In figure 3, the results with value are almost identical to the ones without value model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)