Discrete Diffusion Trajectory Alignment via Stepwise Decomposition
Authors: Jiaqi Han
Austin Wang
Minkai Xu
Wenda Chu
Meihua Dang
Haotian Ye
Huayu Chen
Yisong Yue
Stefano Ermon
Venue: ICLR 2026 Poster (2026)
Keywords: discrete diffusion models
preference optimization
Primary area: generative models

Abstract: Discrete diffusion models have demonstrated great promise in modeling various sequence data, ranging from human language to biological sequences. Inspired by the success of RL in language models, there is growing interest in further improving the models by alignment with a certain reward. In this work, we propose an offline preference optimization method to approach trajectory alignment for discrete diffusion models. Instead of applying the reward on the final output and backpropagating the gradient to the entire denoising process, we decompose the problem into a set of stepwise alignment objectives by matching the per-step posterior. This framework enables efficient diffusion optimization, is compatible with arbitrary reward functions, and importantly, yields an equivalent optimal solution under additive factorization of the trajectory reward. Experiments across multiple domains including DNA sequence design, protein inverse folding, and language modeling consistently demonstrate the superiority of our approach. Notably, it achieves an up to 12\% improvement over the most competitive RL-based baseline in terms of predicted activity on DNA sequence design, and further improves the GSM8K score from 78.6 to 81.2 on LLaDA-8B-Instruct for language modeling.

Published as a conference paper at ICLR 2026
DISCRETE DIFFUSION TRAJECTORY ALIGNMENT VIA STEPWISE DECOMPOSITION
Jiaqi Han
Austin Wang?1
Minkai Xu'
Wenda Chu?
Meihua Dang
Haotian Ye!
Huayu Chen
Yisong Yue?
Stefano Ermon
Stanford University
Caltech
Tsinghua University
ABSTRACT
Discrete diffusion models have demonstrated great promise in modeling various sequence data, ranging from human language biological sequences. Inspired by the success of RL in language models, there is growing interest in further im- proving the models by aligmment with a certain reward. In this work, we propose offline preference optimization method to approach trajectory alignment for discrete diffusion models. Instead of applying the reward on the final output and backpropagating the gradient to the entire denoising process, decompose the problem into a set of stepwise alignment objectives by matching the per-step poste- rior. This framework enables efficient diffusion optimization, is compatible with arbitrary reward functions, and importantly, yields an equivalent optimal solution under additive factorization of the trajectory reward. Experiments across multiple domains including DNA sequence design, protein inverse folding, and language modeling consistently demonstrate the superiority of our approach. Notably, it achieves an up to 12% improvement over the most competitive RL-based baseline in terms of predicted activity on DNA sequence design, and further improves the GSM8K score from 78.6 to 81.2 on LLaDA-811 -----BBBBuC for language modeling.
INTRODUCTION
Diffusion models (DMs) (Sohl-Dickstein et al., 2015; Ho al., 2020; Song et al., 2021) have emerged as a powerful tool for modeling distributions and generating samples across an array of modalities such as visual contents (Rombach al., 2021; Saharia al., 2022: Ho al., 2022), natural languages (Nie et al., 2025; Lou et al., 2023; Shi et al., 2024; Sahoo et al., 2024), and geometric structures (Xu al., 2022: Han al., 2024b; Hoogeboom et al., 2022b), to name a few. Among them, discrete diffusion models (Austin et al., 2021: Campbell et al., 2022; Lou et al.. 2023; Sahoo et al., 2024; Shi et al., 2024; Hoogeboom et al., 2022a), those that are in particular grounded on masked discrete latent variables, have demonstrated remarkable promise for modeling sequence data in discrete space, achieving superior performance on tasks ranging from DNA sequence design (Wang al., 2024; Gosai et al., 2023) and protein inverse folding (Campbell et al., 2024; Wang et al., 2024; Hsu et al., 2022) to even text generation (Lou et al., 2023; Sahoo et al., 2024; Shi et al., 2024; Zheng et al., 2023; Gong et al., 2025) and chatbot (Nie et al., 2025; Ye et al., 2025).
Despite the promise, a critical question still remains unrevealed for discrete DMs: How to align pretrained discrete diffusion models towards target distribution, usually defined in the presence of certain reward? Such problem has been of core interest in finetuning modern Large Language Models (LLMs)) (Brown al., 2020; Achiam al., 2023; Team et al., 2023), paradigm usually referred to as Reinforcement Learning with Human Feedback (RLHF) (Christiano et al., 2017: Stiennon et al., 2020; Ouyang al., 2022) or preference optimization (Rafailov et al., 2023; Ji al., 2024). is vital in enhancing the applicability of the pretrained model on downstream tasks by biasing its distribution
"Correspondence to: ji anniiiann@ttafrrde..oo hanjq17 Equul contribution.
fifsiiiiion sdpo. Code available https: //gi thub.ccm

---

Published as a conference paper at ICLR 2026
towards that with higher rewards, e.g., higher enhancer activity for DNA sequence (Wang et al., 2024) or helpfulness and harmlessness for chatbots (Rafailov et al., 2023; Ji et al., 2024; Bai et al., 2022).
Existing alignment literature is primarily based on the left-to-right autoregressive modeling of sequences (Rafailov et al., 2023; Han et al., 2024a), and performing preference optimization is particularly challenging for discrete DMs, which hold the fundamentally different factorization with discrete random variables through a large number of diffusion steps.
Markov chain of sequence- Previous work explored using RL to fine-tune the model, but the inherent discrete representation makes it challenging to efficiently backpropagate the gradient to the entire sampling process, with the reward typically computed upon the final output. Furthermore, this nature also makes it prohibitive to efficiently compute exact likelihood and evaluate rewards when aligning the joint of latent variables on the chain, leading to suboptimal performance (Wallace et al., 2024; Zhu et al., 2025b). The chained sampling of discrete diffusion also makes online RI (Zhao et al., 2025) computationally exhaustive.
In this work, we propose principled approach for preference optimization of discrete diffusion models via stepwise decompesition. Our key innovation is to decompose the alignment of the entire diffusion trajectory po (x(:T))) into a set of subproblems, each of which responsible for aligning the per-step factorized approximation of the posterior (x (x,,, where Xg is clean sequence and x, is the latent variable at diffusion step t. Our stepwise decomposition takes the advantage of leveraging po(xo(xx) as the per-step alignment target, thus enabling both efficient and accurate likelihood computation and reward evaluation defined on clean sequence Xg. Furthermore, we also theoretically reveal novel connection between our stepwise decomposition alignment and the original problem by showing that the optimally aligned posteriors ø" (xo) xx) induce a joint p" (Xi:T) that is also an optimal solution of the diffusion trajectory alignment objective, when the reward of the trajectory takes an additive factorization over certain stepwise reward. In addition, we also develop a general form align the stepwise posterior po(x))xx /xx) that works with arbitrary reward models, as opposed to previous preference optimization approaches (Rafailov et al., 2023; Wallace et al., 2024) specifically tailored under certain simplified reward such as the Bradley-Terry model (Bradley & Terry, 1952).
Contributions. To sum up, we propose stepwise decomposition preference optimization (SDPO) for offline finetuning of discrete diffusion models, with the following detailed contributions. 1. We decompose the diffusion trajectory alignment problem into a set of subproblems that align the posterior (o(xo|xe) for each diffusion step, allowing for efficient and exact likelihood and reward evaluation. 2. We theoretically demonstrate the equivalence of SDPO and diffusion trajectory alignment through the bridge of certain stepwise reward. 3. We derive a general loss function that jointly optimizes the stepwise alignment problems under arbitrary reward functions. 4. We conduct extensive experimental evaluations on three different tasks, namely DNA sequence design, protein inverse folding, and language modeling. Our approach exhibits consistent enhancements, outperforming baselines by a significant margin across all benchmarks. Notably, we obtain a remarkable 12% gain in terms of predicted activity on the DNA sequence design, compared with the most competitive RL-based method ( (Wang et al., 2024; Borso et al., 2025) tailored for finetuning discrete diffusion models. Moreover, we adopt our approach to LLaDA-8B-Instrul (Nie al., 2025), which further enhances GSM8K 5-shot score from 78.6 to 81.2, further demonstrating its promise as large language models.
RELATED WORK
Discrete diffusion models. Discrete diffusion models, originally formulated in Austin et al. (202 I); Campbell et al. (2022); Hoogeboom et al. (2022a) and further extended by Lou et al. (2023); Sahoo et al. (2024); Shi et al. (2024); Zhao et al. (2024), have attracted growing interest in particular for modeling sequence data. Different from autoregressive models (Brown al., 2020; Achiam et al., 2023; Team et al., 2023), discrete diffusion models relax from the inherent left-to-right causal ordering, allowing for more flexible modeling and parallel decoding (Xu et al., 2025; Zheng et al., 2025). They have achieved remarkable performance design (Wang et al., 2024; Campbell et al., 2024) to human natural language modeling (Arriola et al., 2025; Nie et al., 2025; Ye et al., 2025; Zheng et al., 2023). Despite the promise, how perform preference optimization on pretrained discrete diffusion models to align with certain reward still remains challenge, which we aim to address in this work.
various tasks, ranging from biological sequence
Preference optimization for language models. Aligning language models with certain reward is a core problem to enhance their utility (Ouyang et al., 2022). Initial approaches under the paradigm of
2

---

Published as a conference paper at ICLR 2026
RLHF (Ouyang et al., 2022; Christiano et al., 2017) that employ RL-based algorithms (Schulman et al., 2017) for alignment have been proposed and successfully adopted. Direct preference optimization (DPO) (Rafailov al., 2023) and subsequent works (Ethayarajh et al., 2024; Meng al., 2024; Han et al., 2024a; Ji et al., 2024; Lai et al., 2024) leverage pairwise or ranking- perform offline optimization that further address the optimization instability and complexity. Whilst much progress have been made, they are developed upon autoregressive language models, while we instead focus on discrete diffusion models with a substantially different probabilistic factorization.
preference dataset to
Diffusion alignment. Preference optimization has also been explored for diffusion models. The pioneer attempt of Wallace et al. (2024); Yang et al. (2024) extend DPO to Gaussian diffusion and is able to promote image quality. Li et al. (2024); Gu et al. (2024) further improve the performance by employing different human preference modeling while Zhu et al. (2025b) proposes to align the score function. There are also works that resort to RI. (Fan et al., 2023; Black et al., 2024) or directly backpropagating through differentiable reward (Clark et al., 2024; Prabhudesai et al., 2024). Differently, we develop principled objective for discrete diffusion which pose unique challenges due to the discrete nature. Wang et al. (2024) approaches this problem through RL by backpropagating the gradient via the Gumbel trick, which leads to optimization overhead. Recent works also derive under pairwise preference based on DPO (Borso et al., 2025; Zhu et al., 2025a), resort to online sampling and verification (Zhao et al., 2025; Yang al., 2025), or perform optimization through posterior matching (Rector-Brooks et al., 2025). Besides these training-based approaches, inference-time guidance has also been explored for discrete diffusion to align sampling distributions. Nisonoff et al. (2024) adapts classifier guidance (Ho & Salimans, 2022) to discrete diffusion, while sequential Monte Carlo (SMC)-based approaches (Wu et al., 2023: Phillips al., 2024; Dou & Song, 2024) have also been introduced for more effective guidance. However, these seidance-based methods usually induce much higher sampling cost and easily suffer from suboptimal performance when the guidance signal is insufficient. Critically, our approach instead offers a generalized optimization objective, does not require online sampling at each iteration, and demonstrates enhanced performance.
bassiiier-guidaane
3 PRELIMINARIES
Discrete diffusion models. Discrete diffusion models (Austin et al., 2021; Lou et al., 2023; Sahoo et al., 2024; Shi et al., 2024) are a family of diffusion models with the latent variables residing in the discrete space X' with dimensionality m. With input data point xo. discrete diffusion features forward diffusion process in the form of Markov chain ((x; (xo) with
q(xx) xp) = Cat (x13 + (1 - -)))))
where π is the vectorized representation of certain prior distribution Cat (:) and og, usually referred to as the noise schedule, is a decreasing function w.r.f. satisfying that og and = O. The transition for any two timesteps 0 < S t S T that induces g xx)) is specified as ((x, xx) = Cat (xx; 0y,x, Xx + (1 - )T)))TT)) where Ozls = 02/ az..
Masked discrete diffusion models. Masked discrete diffusion models (Sahoo et al., 2024; Shi et al., 2024; Lou et al., 2023; Austin et al., 2021) are discrete diffusion models when the prior π is in particular instantiated as the absorbing state m == (0... 0, 1] where the last entry in m corresponds to special MASK token. The posterior has a simplified form (Sahoo et al., 2024; Shi et al., 2024):
(Ca((xs Xt)
X 7 m,
((x,|x2,,x0 1 Cat Xs;
(2)
au- Xo I-0.
1re m
X, m.
The reversal (x (x:) is then parameterized by a neural network fo (Xe: f) that predicts Xg in Eq, 2, which is optimized to approximate the posterior by minimizing the negative evidence lower bound E(=l
- ((x)) < LNLLBO = Extxx xx))
Ote 0e-00-1 log (xJ fg (x,,
I-ar
Reinforcement learning with human feedback. At alignment stage, a pretrained model (x(c)c) is finetuned to maximize certain reward r(x. c) subject to a -eiblec (KL) divergence regularization w.r.t. the reference model prot (x|c), leading to the following objective:
max Ex,e [r(x, c)] - 3KL. (pa( (Peer Pu
(3)
3

---

Published as a conference paper at ICLR 2026
where c is some context such prompt and 3 is the balancing factor. The choice of the reward model can be arbitrary, such as human or L-M-sssisted preference labels (Ouyang et al., 2022; Rafailov et al., 2023), or the predicted activity of the designed DNA sequence (Wang et al., 2024). This Kl-ccontrrained optimization problem has the optimal solution (Peters & Schaal, 2007)
exp (f)(x.c))
p¹ p'(x(c)= = Z(c) Prer (x
P(c)Prot(ee
where Z(c) = Ex
(xc)=Exxaan (x)e( exp (1 (fr(x,c) is the partition function that is intractable to evaluate.
Problem formulation. In this work, we aim develop an efficient offline alignment approach for discrete difflusion models. Specifically, the algorithm directly operates on pre-collected dataset D {(xo, c, (xx, c))} on clean data Xg without relying on on-policy generations during finetuning.
METHOD
In this section, we detail our approach for aligning discrete diffusion models through stepwise optimization. In 4.1, we first revisit the problem of discrete diffusion alignment and investigate the challenges. In $ 4.2, we propose a novel stepwise decomposition approach for discrete diffusion aligmment. In $ 4.3, we introduce principled way to solve the stepwise alignment objective through distribution matching. We offer additional in-depth analyses and discussions in š 4.4.
4.1 ALIGNING DISCRETE DIFFUSION MODELS
Different from autoregressive models that evaluate pe (x(c) efficiently in single forward pass, discrete diffusion models are grounded on a chain of random variables Xp:T == Xx: X1, Xr),
where the joint satisfies the Markovian factorization pa (Xorlcc) = Pa e) I Pe px(x,-1 |x,, c).
The alignment objective Eq. is therefore extended to the entire chain (Wallace et al., 2024):
max E (xerr e),e [F(xoer c)] 33 [po (xx=rle)]lpaaaxxxxxrl
Pe
where the reward f(XO.T, c) now considers the whole chain Xo:T. We hence refer to the optimiza- tion problem of Eq. 5 diffusion trajectory optimization. Akin Eq. 4, the optimal solution is p" (*X:ol/c) = Pret (x).Tlc) exp () c)). However, the optimization problem in Eq. 5 poses several challenges. First, the expectation is taken over the entire chain (=(o).)), making it computationally expensive. Moreover, the reward f(Xo:T, c) supposed to operate on the entire chain, while empirical rewards r(xp, c), egg., human preference (Rafailov et al., 2023) or DNA activity (Wang et al., 2024), are most commonly defined on the clean sequence Xg- We will introduce our stepwise decomposition approach that offers simplified and tractable measure to solve Eq. 5.
4.2 DIFFUSION TRAJECTORY OPTIMIZATION THROOGHI STEPWISE DECOMPOSITION
We propose a principled way to solve the prob- lem by decomposing the trajectory optimization into a set of subproblems, each of which corresponds to stepwise alignment objective for the factor- ized approximation of the posterior po (x c) II' po (x(") 1x, c) (Shi et al., 2024; Austin al., 2021) at diffusion step < t < T (see Fig. 1), where i the token index and is the sequence length. To be specific, the set of subproblems is
Trajectory Alignment
XT
X
x
Stepwise Decomposttion Py Kt G)
F(xass))
X
X
x0
Jac X7.c)
} fatta (..c)
Polxolie
X
X
X
rixa.c)
rixe.
e(x.,e)
L
max Eju (xo xx. c).e [r(xp.c)
Figure 1: The flowchart of our SDPO.
DKL (x Xx, c) (xlliror(xx))) Xx, c)], (6)
V1 < T, where Br = (/w(t) is the stepwise regularization reweighted by certain scheduler w(t).
The optimal solution for any / is given by 0r j0((xo/xx (xolxe, c) = Dio (Xo (xe, c) exp Jo. rx0, c))
4

---

Published as a conference paper at ICLR 2026
Such formulation enjoys several unique benefits compared with the trajectory alignment objective in Eq. 5. First, the expectation over the entire chain pe (xx:7 (c) has been decomposed into the stepwise posterior 6o (x, c). which can be computed both tractably and efficiently for discrete diffusion models. Furthermore, by grounding on the clean data Xo instead of intermediate latent variables x,, can readily reuse the reward model r(xo, c) without resorting to its biased estimates (Lu et al., 2023; Chen et al., 2024). More interestingly, below we reveal a critical connection between the stepwise decomposition alignment objective (Eq. 6) and the trajectory optimization objective (Eq. 5), as stated in the theorem below:
Theorem 4.1. The joint p" (xo:rle) induced by the optimal solutions { (xo|>x c) of Eq. 6 is also the optimal solution of the trajectory alignment objective in Eq. 5, with the chain reward /
c) where (xxx1ixxx, log P 1 m..0) [exp( dr (xxo,)))
F (XXoTT c) - 3 1 Ve(X
Ba (molmee)) [exp( B, ((x0,)))
Proof is in Appendix 1. Here Preet (xoo(Xol (X c) Proet (xco) Ies (x xx) is the posterior of xo Peet (x4 (x.)
w.t.t a specific choice of x, given x,. In the case of masked diffusion models, the posterior refers to the factorized conditional Pret Xx, c) constrained on the set of all possible xo that share the same decoded tokens with x,- Theorem 4.1 endorses our key finding that the intractable trajectory optimization can be alternatively approached by jointly optimizing the stepwise alignment objectives, under which the reward of the chain f(x):T, c) is effectively an additive factorization of the stepwise reward re (x+-1; x,, c). More interestingly, the stepwise reward also has intuitive implications. At each diffusion step & with the sampled x,, the denominator inside log is a constant and ry is therefore distinguished fully by the numerator, a term that effectively assigns higher reward to those x who are more likely to be obtained from the Xo with higher reward r(Xo. c). Furthermore, the stepwise rewards also serve as more fine-grained supervision that enables tractable alignment of each diffusion step, while previous works that operate fully on the trajectory- level confer no per-step guarantee.
4.3 GENERALIZED STEPWISE ALIGNMENT THROUGH DISTRIBUTION MATCHING
While the stepwise decomposition has introduced clear benefits, it is still yet unclear how to optimize Po (xoo Xt, c) towards the optimal solution 9² Xx, c). particularly under arbitrary reward r(xo, c). To this end, existing works seek to directly backpropagate the gradient from the reward model (Wang et al., 2024), which inevitably incurs optimization overhead and instability, or to simplify the reward into tractable forms such as the Bradley-Terry model (Wallace et al., 2024), which imposes additional constraints. Differently, we propose to perform optimization based on the following objective:
(0) Ex,,e [DKL [ö (xol x1, c) c)]],
where Py (xxoxx,,)) c) x Pret (Xy x c) exp(r(xx. c)) the Boltzmann policy (Laidlaw & Dragan,
2022; Peters & Schaal, 2007) induced by the reward then reweighted by pref, while (xol c) x Pret (Xo) (xo]X X2, (1 po (xo x,, c)& 9 is similarly the reweighted model policy. The rationale of Eq. 7 lies in that the minimizer of this Kiveeiveeeeece distribution matching (Han et al., 2024a; et al., 2024) problem is also the optimal solution of stepwise alignment (proof in Appendix. A.2):
Proposition 4.2. Let 0* = arg min (0) defined in Eq. 7. Then po- fo-((xolx (xo]Xx, c) (xxo|xx, c), the optimal solution of the stepwise alignment objective in Eq. 6.
Besides the guaranteed equivalence of the optimal solution, the definition of p, also enables impor- tance sampling by using Pref as the proposal distribution, from which the offline preference datasets are drawn. Expanding Eq. 7 with importance sampling (see Appendix. A.3), we have
cxp(r(Xo. c)) exp(ro Xx, c,)) (,(c)
(0) Prer
log
C,
(8)
Z X2, C, (c)
where re(Xo, Xz, c, ()) = 3e (log Pa (xo) x;, c) - log Pret (xo X2. c)) refers to the implicit re- ward (Rafailov et al., 2023; Cui al., 2025), Z,((c) Ep.a (xD e) exp(r(xi) c)) and (x c, B,)
Expre e) exp(ro(xo, ( c, Bc)) are the partition functions, and C is a constant irrelevant to 0
Empirical form. We leverage Monte-Carlo to estimate L. as well as the partitions using N samples
((x))) (8) (1) ,, drawn from Pret (Xo, x; /c) for each c. In form, employ
N
exp(r
x c))
exp(Fa(x)) x c, 80)
2N (0) = -E. Σ
log ΣØ= exp(Fx(x)) x1 c,,,,)
(9)
j=1 exp(r(x)) .))
0)
5

---

Published as a conference paper at ICLR 2026
Eq. 9 takes the form of cross-entropy loss (Ji et al., 2024; Lu et al., 2023) between the self-normalized Boltzmann policies induced by r(xo, c) and (g x2. c, 3)). As N - J0, the estimate for the policy of r becomes unbiased, while unbiased estimate of Z3 requires extensive sampling from the posterior (Xo Xt, c) for each Xt, which highly prohibitive in the offline alignment setup. In practice still favor the simplified MC estimate in Eq. which is efficient and performant, We henceforth employ L (0) to solve each subproblem of Eq. For the sample size N, view it as hyperparameter that trades off between efficiency and bias, depending on the task and dataset.
4.4 OVERALL OBJECTIVE
Since the final objective (Eq. 6) requires to jointly optimize for the subproblems across all diffusion steps, at each iteration we randomly select batch of diffusion steps, and optimize the corresponding C as per Eq. Furthermore, since offline settings the intermediate samples X; are not preserved, we instead keep track of the clean samples Xy obtained from Pret while approaching the corresponding X₁ via the forward process (xx, (xo) at each training step. Putting all together we obtain our final loss
exp(r(x,) (4)
c))
log exp(Fa (x)
c, B6))
((0)= = 1(xx,1x00
== exp(r(x) (1) .c) Σe exp(T8 (xo ,., 3x))
(10)
where Fe, by further leveraging the reversal factorized parameterization of masked diffusion mod- els (Shi et al., 2024; Sahoo et al., 2024) and the definition 3, B//w(t), takes the following form:
oa No. x,, c, B2) = B log(x) (x) fx t, c)) log(x, fret (xe, t, c))
(11)
w(t)
w(t)
We note that our method applies to general discrete diffusion, but we choose to focus specifically on the masked variant. Our final loss has several implications, which we will analyze below.
Pairwise preference data. Our loss possesses a generalized form w.E.. the reward model r(x,, c) and N, i..,, the number of samples for each context or prompt c. In particular, it subsumes the setting
in DPO where each prompt is provided with pair of winning and losing completions (x(=)) xx
by setting N = 2 and leveraging Bradley-Tery (BT) model as the reward, i.e., r(x, (m)) .) = 0 and rxx"") (1) c) oc. We provide detailed derivations of our loss this special case in Appendix A.4.
The role of wt). The coefficient w(t) is initially introduced as the weight for the per-step reward A. Interestingly. from Eq. we can also interpret wtt) as a factor that controls the scale of log(x, f(x,, t)), which is correlated to the number of masked tokens at step f. Therefore we set w(() a to amortize the loss to each token, and empirically find this choice effective.
The role of B. Eq. 5 reveals that B controls the strength of the KL regularization w.s. the reference distribution, which also widely reflected in literature (Rafailov et al., 2023; Wallace et al., 2024).
Iterative labeling. Empirically we have also explored a variant of our approach that updates the dataset with samples from the latest model and their corresponding rewards. We find this iterative labeling generally favorable since more useful rewards are progressively provided for samples of higher quality, as the training proceeds. We defer detailed justifications to $ 5.4.
EXPERIMENTS
We perform extensive evaluations on wide suite tasks, including DNA sequence design ($ 5,1,, protein inverse folding ($ 5.2), and language modeling ($. 5.3). We provide ablation studies in 5.4.
5.1 DNA SEQUENCE DESIGN
We aim to finetune our model to unconditionally generate DNA sequences that trigger gene expression in targeted cell types. This is task commonly seen in cell and gene therapy (Taskiran et al., 2024).
Experiment setup. We use publicly available dataset (Gosai et al., 2023) that contains the measured enhancer activity in 700k DNA sequences, each 200 base-pairs in length. Cell line activity is

---

Published as a conference paper at ICLR 2026
Table 1: Model performance on DNA sequence design. Our approach generates sequences with high activity while being natural-like. Results averaged across random seeds with standard deviations in parentheses. Numbers of baselines are taken from Wang et al. (2024).
Prevvictivty (med) ATAC-Ace 1 (R) 3-mer Corr
JASPAR Com App-Log-Lit (med) Entropy (med) 1
Pretrained (Sehoo et al., 2024)
0.17 (0.04)
1.5 (0.2)
-0.06 (0.034)
0.249 (0.015)
-261 (0.6)
390 (6.2)
CG (Nisenotf et al., 20224
(0.00)
0.0 (0.0)
.0.065 (0.001)
0.212 (0.035)
-266 (0.6)
(4.1)
SMC (Wu al., 2023)
(0.33)
39,9 (8.7)
0.840 (0.045)
0.756 (0,068)
259.2.5)
351 (6.5)
TDS (Wu et al.. 2023)
(0.21)
(16.4)
(0.00)
(0.054)
CFG (Ho & Salimans, 2022)
5.04-(0.6)
92.1 (0.9)
0,746 (0.001)
0.864 (0.011)
-263 (0.6)
363 (6.1)
D2-DPO (Borso et al., 2025)
2.97 (1.03)
35.6 (0.9)
0,944 (0.002)
0.83 (0.025)
-252 -2.4((1.4)
362 (4.9)
VRPO (Zhn et al., 2025a)
(.60(0.01)
15.8 (0.2)
0.838 (0.002)
0.865 (0.025)
-255 (0.8)
289 (13.5)
DDFF-IS (Rector- Brooks al., 2025)
4.07 (0.02)
(0.3)
0.711 (0.001)
0.723 (0.034)
-253
378 (5.8)
DRAKES (Wang et 2024)
(0:07)
92.5 (0.6)
(0.002)
(0.002)
325 (5.2)
diftc- GRPO (Zhao et al., 2025)
5.86 (0.04)
33.0 (0.5)
0.783 (0.001)
0.903 (0.004)
-245 (0.4)
310 (8.6)
SDPO
6.30 (0.003)
94.8 (0.01)
0.900 (0.003)
0,936 (0.0033
.216
365 (4.4)
measured for each sequence, quantified with massively parallel reporter assays (MPRAs) that record
the expression each sequence drives. The pre-trained masked diffusion language model (Sahoo et al.,
2024) is taken from Wang et al. (2024), trained on the entire enhancer dataset. The pre-trained finetuning and evaluation reward models predict the HepG2 cell line activity in a sequence, also taken from Wang et al. (2024) and trained on different splits of the dataset.
Baselines. We compare with the following baselines. Pretrained: the base pre-trained model (no finetuning). Guidance methods: classifier guidance (CG) (Nisonoff et al., 2024), classifier-free guidance (CFG) (Ho & Salimans, 2022) and two Sequential Monte Carlo-basee based methods (Wu et al., 2023), namely SMC, where the proposal is the pretrained model, and TDS, where the proposal is CG. D2-DPO (Borso al., 2025) and VRPO (Zhu al., 2025a): offline preference optimization algorithms that adapt DPO to discrete diffusion. DRAKES (Wang et al., 2024): an online RL algorithm that backpropagates the reward through the generated trajectory with Gumbel--Softmax diffu-GRPO (Zhao et al., 2025): policy gradient-based approach for discrete diffusion. DDPP- IS (Rector-Brooks et al., 2025): an importance sampling method to match the reward-tilted posterior.
Metrics. We use the metrics following the protocol in Wang et al. (2024) to evaluate the model's enhancer generation. 1. Pred-Activiry. The enhancer activity level in the HepG2 cell line is predicted by the evaluation reward model, trained on a held out evaluation set. 2. ATAC-Acc. We measure the proportion of generated sequences with high chromatin accessibility. This metric is typically correlated with the enhancer activity. 3. 3-mer Corr. We compute the 3-mer Pearson correlation between the generated sequences and the sequences from the enhancer dataset with the top 0.1% HepG2 activity. More natural, in-distribution sequences tend have higher 3-mer Pearson correlation values. 4. JASPAR-Corr We compute potential transcription factor binding motif's in the generated sequences with JASPAR transcription factor binding profiles CCastro-Mondrass calculate the Spearman correlation of motif frequency between the generated samples and the top 0.1% sequences in the dataset with the highest activity. 5. App-Log-Lik. The approximated log- likelihood of the generated sequences is computed with respect to the pre-trained model using the discrete diffusion ELBO presented in Sahoo et al. (2024). This metric evaluates the naturalness of the generations, as samples that over-optimize for the reward model tend to have worse log- lilkiiihohods.. 6. Entropy. Sequence entropy is computed following Wang et al. (2024) to measure the sample diversity.
et al., 2022), and
Results. Our method generates sequences that are both natural-like and have high predicted enhancer activity. Notably, we are able to significantly outperform all previous baselines in the predicted HepG2 activity, while also achieving strong 3-mer Pearson and JASPAR correlation numbers, demonstrating our method's 'ss robustness to over-optimizing for the reward model. In particular, we outperform the RL-based approach DRAKES by significant margin of 12.3% in terms of predicted activity. The ATAC accuracy, another metric correlated with HepG2 activity, provides further validation of the high quality of our generated samples, as we see that other baselines, such as the based methods, may achieve relatively higher predicted enhancer activity but suffer poor ATAC accuracy numbers.
Training efficiency. Besides the superior performance, another feature that worth highlighting for SDPO is its training efficiency. since it does not require on-policy sampling at each training iteration. We report the average wallclock time per training step, where DRAKES takes 6.02 sec, diffu-GRPO takes 1.51 sec, and SDPO only takes 0.77 sec, which verifies the superior training efficiency of SDPO.
5.2 PROTEIN INVERSE FOLDING
For the protein inverse folding task, we finetune a pre-trained model that predicts the protein sequence from 3D structure. We aim optimize the stability of the protein sequences.
7

---

Published as a conference paper at ICLR 2026
Table 2: Model performance on inverse protein folding, Our approach generates protein sequences with high stability and desired structure. Results averaged across 3 random seeds with standard deviations in parentheses. Numbers of basclines are taken from Wang et al. (2024).
Pred-cuGG (med) 1 ReddG 01 (
scRMSD (med) 1 RiscRMSDD 2)
Success Rate (56) Entropy [med) 1
Pretmined (Camphell al., 2024)
-0 54444444 (0.037)
36.6(1.0)
D:849 (0,013)
(0.9
34.4 (0.5)
1)
CG (Nisonoff et al., 20224)
-0.561 (0.045)
36.9 (1.1)
0,839 (0.012)
90.9 (0.6)
34.7 (0.9)
$4.6 1)
SMC (Wo et 2023)
0.659 (0.044)
68.5
D.841 (0.006)
93.8 (0.4)
63.6 (4,0)
24.9 (6.9)
TDS (Wu al., 2023)
0.674 (0.086)
(2.2
0.834 (0,001)
94,4
62.9 (2.8)
24,9 (7.2)
Salimans, 2022)
-1 (0.035)
(1.4)
(0.062)
29.4
(0.4)
(5.9)
D2-DPO (Borso et al., 2025)
0.500 (0.051)
(0.3)
0.909 (0.005)
(0.6
61.0 (0.5)
(7.4
VRPO (/hu et al., 2025aa
0.548 (0.022)
61.1 (0.1)
0 (0.004)
93.5 (0.7)
56.6 (0.3)
39.1 (9.3)
DIPP-IS
al. 2025g)
(0.047)
46.7
(08)
(0.6
43.3 (0.5)
DRAKES (Wang et al., 2024)
1.095 (0.026)
$6.4 (0.2)
0.918 (0.006)
91.8
78.6 (0.7)
33.3 (6.4)
difti-GRPO (Zhon al., 20255
1. 1,286 (0.021)
76.8 (0.3)
1,192 (0.005)
57.1 (0.8)
(1.4
(0.0
SDPO
1.400 (0.01-)
87.1 (0.01)
0.938 (0.005)
(0.3
75.5 (0.3)
(2.3
Experiment setup. The pre-trained diffusion model uses the ProteinMPNN (Dauparas et al., 2022)
architecture and is trained using the methodology from (Campbell al., 2024) on the PDB training dataset from Dauparas et al. (2022). The finetuning and evaluation reward models are trained on different splits of the Megascale (Tsuboyama et al., 2023) dataset. We take all checkpoints directly from Wang et al. (2024). For finetuning our model, we use the curated Megascale training dataset from Wang et al. (2024), which consists of 5500k sequences with stability measurements.
Metries. We use the following metrics (Wang et al., 2024) to evaluate the stability and naturalness of the generated protein sequences. 1. Pred-ddG. The evaluation reward model predicts the ddG (change in Gibbs free energy) of sequence, which is measure of the sequence's s stability. The finetuning dataset does not overlap with the evaluation dataset, so the model does not train on proteins used for evaluation. 2. scRMSD. The self-consistency root mean square deviation (scRMSD) measures the ability of a sequence to fold into the desired structure. We use the pre-trained ESMFold (Lin et al., 2023) model to compute the RMSD between the sequence's predicted 3D structure and the original backbone structure. 3. Success rate. We compute the success rate as the proportion of generated sequences with Pred-ddG measure the sample diversity.
and scRMSD 2. 4. Entropy. The sequence entropy computed to
Results. Our method is able to generate sequences with high stability that still remain in-distribution. We significantly outperform all baselines in the predicted ddG for stability, showing strong reward optimization ability, while still producing natural-like samples with scRMSD values and overall success rate comparable to the state-of-the-ar DRAKES method. The policy- griid-tt based method diffu-GRPO exhibits significant reward over-optimization with severe drop in metrics like Success Rate. Notably, the inverse folding problem is particularly difficult due to lack of labeled data in the curated Megascale dataset (only several hundred distinct 3D structure backbones). During evaluation, the model conditions on new backbone configurations not seen during training. Thus, our method is still able to generate high reward samples without over-opti in a limited-data setting.
5.3 LANGUAGE MODELING
Crucially, we also apply our approach to a large-scale discrete diffusion for natural language modeling, demonstrating its efficacy towards preference optimization of large language diffusion models.
Experiment setup. We employ LLaDA-88-Instrrae (Nie et al., 2025), a large-scale instruction- tuned chat model based on the masked diffusion framework, as the reference model. We use UltraFeeddacc (Cui et al., 2023) dataset annotated by Meng et al. (2024) as the preference dataset, and finetune the model on 8 Nvidia A100 GPUs. Detailed hyperparameters are deferred to Appendix.
Benchmarks and metrics. We compare our finetuned model against the reference model on three important language model benchmarks. 1. GSMSK (Cobbe et al., 2021), which benchmarks the math and reasoning capability of the model on graduate school math problems. The metric is the average accuracy of the answers. 2. IFEval (Zhou et al., 2023). which measures the model's capabil- ity of following human natural language instructions. We report IFEval score, the average of prompt and instruction- evaluates the chat response quality by comparing against certain baseline model on suite of prompts.
Table 3: Results on finetuning LLaDA-8B-
Instruct using dataset from Cui et al. (2023).
Instruct D2-DPO diffu-GRPO SDPO
Alpaca- LC (%)
10.6
12.1
12.6
14.2
Rval 2.0 WR (%)
6.8
7.5
7.8
GSMSK
78.6
78.1
80.5
81.2
IFEval
52.9
53.8
53.5
55.1
atrict-ccuuacy. 3. AlpacaEval 2.0 (Li et al., 2023; Dubois et al., 2024) that
8

---

Published as a conference paper at ICLR 2026
Pred Activity (median))
6.4
0.7
1,00
-0.6
Pred ddG (median)
1.0
6.2
0.75
0.5
6.0
0.4
0..50
0.3
0.0
5.8
0.2
0.25
5.6
-0.5
8:80
0.84
0.88
0:92
0.96
0.4
0.5
0.6
0.7
0.8
ATAC-Acc (%)
Positive Reward Portion (%)
(a) DNA sequence design.
(b) Protein inverse folding.
Figure 2: Ablation studies of /3 in (a) DNA design, and (b) protein inverse folding experiment.
The metrics on this benchmark are the winrate (WR) and length-controlled (LC) winrate against GPT-4--reviee--I11
The benchmark results are presented in Table 3. By finetuning LLaDA- ---------nntte using our proposed SDPO, we observe a consistent and remarkable enhancement across all three benchmarks, which underscores the efficacy of SDPO towards promoting the capability of mathematical reasoning. instruction following, and chat quality of the discrete diffusion language model. Notably, our approach improves GSM8K score from 78.6 to 81.2, surpassing the score of LLLMA-3--8 post-trained with RL (c.f. Nie et al. (2025)). Furthermore, and WR on AlpacaEval 2.0 benchmark, demonstrating the applicability of SDPO for building helpful discrete liffusion-based chatbot. Our results language modeling tasks open up new possibility towards building performant large language diffusion models through preference optimization.
Results.
obtain relative improvement of 30.9% averaged across LC
5.4 ABLATION STUDY
The effect of B. We study the effect of [ in aligning models. As shown in Eq. 5, choosing smaller 3 leads to weaker regularization and tunes the model further away from the pretrained reference distribution. We verify this by two ablation studies folding, fixing all hyperparameters except B. Fig. 2 shows that a lower 3 value results in stronger reward guidance, resulting in greater Pred-Activity for DNA design, and greater Pred-ddG values for protein inverse folding. Conversely, a larger 3 poses more regularization to the model and thus the reward remains closer to the pretrained reference model. However, choosing too small f may also steer the model too far away from the reference model and result in unnatural sequences. As shown in Fig. 2(a), the ATAC-Acc of the generated DNA sequences decreases as we over-optimizes Pred-Activity with small B, despite their being positively correlated for natural DNA sequences.
DNA sequence design and protein inverse
The effect of N. We first investigate the effect of the sample size N. The results in protein inverse folding task without any iterative labeling are presented in Table 4. Notably, we observe that as the value of N gradually increases, reduces the variance in Monte-Care Carlo estimate performed by Eq. 10, which is further supported by the increasing trend in Pred-ddG. In particular, compared with N = 2 which
Table 4: Effect of sample size N in the inverse protein folding task.
N
Pred-ddG Positive Reward Prop.
effectively
2
0.529
0.624
10
0.924
0.749
25
1.119
0.759
100
1.061
0.765
reflects the pairwise preference data setting adopted in DPO, leveraging a comparatively larger N is more beneficial. The performance plateaus as N further increases from 25 to 100, which is empirically not as favorable due the memory overhead incurred.
Iterative labeling. In $ 4.4 we additionally introduce a vari- of our SDPO that leverages iterative labeling to enhance performance. Specifically, during training we iteratively gen- erate 10,000 samples from the model and label them using the reward model in the DNA experiment. We then optimize the model on these labeled samples using the same objec- tive. We demonstrate the advantage of such an approach in Fig. 3. Compared with the baseline that does not scale up the labeling latest samples but always samples from the original model, we observe consistent increment over 2 rounds of iterative labeling. In particular, the predicted DNA
Prod
KOAC Aoourirry
Pred halate
10000
20000
Extra Sample Size
Figure 3: Iterative labeling.

---

Published as a conference paper at ICLR 2026
Full
DRAKES
dIffuGRRPO (p 0.011
SOPO (w/o Itar)
diffu-GRPO ( 0.1)
diffu-GRPO If 0.001)
=2=00610
0.781
6
P. c
2
HepG2
4
3-
-1
1-
10k Num. labeled samples
20k
<Ko. c)
(b) Correlation analysis between r(xo, e) and induced trajectory reward ((xo:T, c).
(a) Reward convergence w.r.t the number of labeled samples.
Figure 4: (a) The reward curve w.r.f the number of labeled samples throughout training. (b) The correlation analysis between the induced trajectory reward f(Xo:T, e) and the clean reward r(Xo, c).
activity improves by significant margin for SDPO with iterative labeling while the counterpart struggles predicted activity while also encountering a drop in ATAC accuracy, possibly due to overfitting. Furthermore, our approach is also remarkably more labeling efficient compared with DRAKES that uses 128,000 additional labeling on the DNA task. The result implies that, on certain tasks when the reward model available, performing SDPO an iterative manner with reward labeling will lead to improved performance.
Convergence rate comparison. We also perform a systematic head-to-head comparison of the reward convergence speed in Fig. 4a, where we plot the reward curve w.E.f the number of labeled samples throughout the training process. For fair comparison with the on-policy baselines DRAKES and diffu-GRPO, we implement variant of SDPO, i.e. SDPO w/ Full Iter, that performs iterative labeling after each training step. Notably, our SDPO with full iterative labeling achieves 6.2 Pred- Wed-HepG22 using only 1.5k labeled samples, while DRAKES and diffu-GRPO only achieve 5.6 and 4.2 Pred-HepCis with 25k labeled samples. SDPO without iterative labeling also exhibits fast convergence and high reward efficiency.
Reward correlation analysis. Here we provide more in-depth analysis on the DNA task regarding the relationship between the reward f(Xo:T, c) defined in Theorem 4.1 and the original reward r(Xo, c). Specifically, we sample 50 trajectories from the pretrained model, the model after first stage training, and the final model respectively, leading to 150 trajectories in total {x 1,10 For each (4) trajectory X we evaluate its chain reward p(i) = f(x).T c) using an unbiased MC estimator (see (6)
Appendix B.5) and the original reward y(0 r(x .). We then perform linear correlation analysis (=)) and show the plot in Fig. 4b, Despite the MC estimation, we observe relatively strong positive correlation between the rewards with Pearson correlation of 0.781. This indicates that the chain reward can be approximately viewed as RS a-r + b, which draws an interesting connection between the reward-tilted distributions: Per (c) exp( ( ( c)) x
for the set of datapoints {(r(() ((0
(xo:(Xo exp( r(xo, c) + ))) х Prer(Xo.Tlc) exp( 3ya c)). Therefore, this empirical investigation interestingly reveals that our trajectory-level chain reward is an effective surrogate of the original reward on Xo and, more importantly, our optimal reward-tilee tilttteed distribution is also decent approximation of the original optimal reward-tilted distribution without notable bias introduced.
CONCLUSION
We present SDPO for preference optimization of discrete diffusion models by decomposing diffusion trajectory alignment into set of subproblems for each diffusion step. Crucially, we propose to align the posterior po Po(xolxx (x:) for each step and draw an equivalence between the two objectives, with which we further derive principled loss function. Experiments
wide range of tasks including DNA sequence design, protein inverse folding. and language modeling consistently verify the efficacy of SDPO, showing its potential towards building performant and applicable discrete diffusion models.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
All authors have read and are committed to comply with the ICLR Code of Ethics (https: iclr. cc ec/pput heppubbicc i /CodeOfEthics). We present principled approach for discrete diffusion model alignment via stepwise decomposition. The goal is to develop a fundamental algorithm for alignment of discrete diffusion models, where we do not find major ethical concerns that need to highlight.
REPRODUCIBILITY STATEMENT
We have presented the detailed experimental setup in Sec. 5 and Appendix B. We also include the code in the supplementary material to ensure reproducibility.
ACKNOWLEDGMENT
We thank the anonymous reviewers for their feedback on improving the manuseript. This work was supported by ARO (W91INF-21-1-0155) ONR (N00014-23-1-2155), and the CZ Biohub.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXivv2303.08774 2023. 1,2
Marianne Arriola, Subham Sekhar Sahoo, Aaron Gokaslan, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Justin Chiu, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregres- sive and diffusion language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: httt:::///peeeeviicee ///pppppevvieeeeeeeeeeeeeeeeeeee net. wow.mmiddttyyrttaa....
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems, 14:17981-17999 2021. 2, 3,
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 25662522, 2022.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffussio models with reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: httts::/ /ttp:////pp//eeeeeee/m net/ forum? Yww.uu/.-CCCWW/CCFFFF 3
Umberto Borso, Davide Paglieri, Jude Wells, and Tim Rocktäschel. Preference-basee alignment of discrete diffusion models. arXiv preprint arXiv:2503.08295 2025. 2, 3, 7
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355 1952. 2,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1 33:1877-1901, 2020. 1, 2
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Amaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-28279, 2022. 1,2
Andrew Campbell, Jason Yim, Regina Barzilay. Tom Rainforth, and Tommi Jaakkola. Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design. arXiv preprint w:Xi::2402.04997 2024. 2,
11

---

Published as a conference paper at ICLR 2026
Jaime A Castro-Mondragon Rafael Riudavets-Puig, leva Rauluseviciute, Roza Berhanu Lemma, Laura Turchi, Romain Blanc-Mathieu, Jeremy Lucas, Paul Boddie, Aziz Khan, Nicolás Manos- alva Pérez, et al. Jaspar 2022: the 9th release of the open-access database of transcription factor 2022.
binding profiles. Nucleic acids research, 50(D1):D165-D17
Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui. Hang Su, and Jun Zhu. Noise contrastive alignment of language models with explicit rewards, 2024. URL https: /ttp///aaxvvv...a//.a//eeee.aav. org. h/s////////// 2402. 05369. 5
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulriee von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vishwanathan, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. httplpppcccppppppppeeeeeepe /tpppppppp//1111/// //hash/
4299 4307, 2017. URL https: http:////cocccecng...eoe neuripssss.nnnnns..oo..........r cc/ Abst html. 1,3
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations. 2024. URL https: htpp//ppeeeeeeeeeeeeeeeeeeeeeeee net/ forum? wawrrumiiddiiiVSiia..
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint viXiv:2110.14166 2021. 8, 23
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.0133
2023. 8, 21
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, al. Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.01455. 2025. 5
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, 378(6615):499555 -6, 2022. 8
Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations, 2024.
Yann Dubois, Balázs Galambosi, Perey Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arX1v:2404.04475, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. 3
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858-79885, 2023.
Shansan Gong, Shivam Agarwal, Yizhe Zhang, Jiacheng Ye, Lin Zheng, Mukai Li, Chenxin An, Peilin Zhao, Wei Bi, Jiawei Han, Hao Peng, and Lingpeng Kong. Scaling diffusion language models via adaptation from autoregressive models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ / opw.reviewwweeeeeoooep net/ forum ?id= iid=jiistykwwg......
Sager J Gosai, Rodrigo I Castro, Natalia Fuentes, John C Butts, Susan Kales, Ramil R Noche, Kousuke Mouri, Pardis Sabeti, Steven K Reilly, and Ryan Tewhey. Machine-guie guided design of synthetic cell type-specific cis-regulatory elements. bioRxiv, 2023. 1,6
12

---

Published as a conference paper at ICLR 2026
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. Diffusion-rpor Aligning diffusion models through relative preference optimization. arXiv preprint arXiv:2406606632, 2024.
Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, and Minkai Xu. f-po: Generalizing preference optimization with f divergeence minimization. arXiv preprint arXiv:2410.21662, 2024a. 2,
Jiaqi Han, Minkai Xu, Aaron Lou, Haotian Yc, and Stefano Ermon. Geometric trajectory diffusion models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024b. URL https: wwwp//ppeeeeeeeeeeeeeeeeeeeeeeep
net net forum? Ld0OYmd55vv
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12544 2022. 3, 7, 8
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. In International Conference on Learning Representations, 2022a. URL https:///// httpt::toopeeeeeeeeeeeeeee/e wow.revveeeeee.eeoeoeeeee net htt///ormmm// (tt.:dddmmmmmmmmmmmmm
1,2
Emiel Hoogehoom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867-8887. PMLR, 2022b.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. In International conference on machine learning, pp. 8946-8970. PMLR, 2022.
Haozhe Ji, Cheng Lu, Yilin NIu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment. The Forty-first International Conference Machine Learning, 2024. URL htt.t:/tpeeee wttttps///axxv..... org/ abs 2402 00856. 1, 2, 6
Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. Step-dpo: Step- wise preference optimization for long-chain reasoning of 1lms. arXiv preprint arXiv:2406.186299
2024. 3
Cassidy Laidlaw and Anea Dragan. The boltzmann policy distribution: Accounting for systematic suboptimality in human models. In International Conference on Learning Representations, 2022. URL https::///// wwwpreveeeeeeeeeeeeeeeeeeeeee.we wtw.eewwweee/teeeeeeemm id- _d=_1_ojPPNNyee
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.04440 044665, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follooinn models. https:: //////gi/ tttpttb..buu.ggg///////ggg.b..tt com/ cwom/tatsu--20 httaaaalpca_eeaaaaaa 2023.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary--caaee prediction of atomic-level protein structure with a language model. Science, 379(6637):1123-11133 2023. 8
Aaron Lou, Chenlin Meng. and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint arXiv:2310. 16834, 2023. 2,
13

---

Published as a conference paper at ICLR 2026
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive ergy prediction for exact mnergy-guided diffusion sampling in offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning. volume 202 of Pro- ceedings of Machine Learning Research, pp.
PMLR, 23-29 Jul 2023. URL
https: hppp:://prceeeiine www.iiinnnmgmm.....mm....m ml wtt.:.ll......s...... v2222/222///// lu23d. htt.://112dd..........3 ht.l 6,
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. In Advances in Neural Information Processing Systems (NeurIPS), 2024. 3, 8, 21
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji- Rong Wen, and Chongxuan Li. Large language diffusion models. arXiv preprint arXhv:2502.09992 2025. 1, 2, 8, 9, 21, 24
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. arXiv preprint arXiv:2406.01572, 2024. 7,
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 11730-27744, 2022. 3,4
Jan Peters and Stefan Schaal. Reinforcement learning by rewgrt-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745-750, 2007. 5,5
Angus Phillips, Hai-Dang Dau, Michael John Hutchinson, Valentin De Bortoli, George Deligiannidis, and Arnaud Doucet. Particle denoising diffusion sampler. arXiv preprint arXiv:2402.06320, 2024.
Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint arXiv:2407.08737, 2024.
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, (855378--53441 2023. 1, 3, 4, 6
Jarrid Brookr, Mohsin Hasan, Zhangzhi Peng, Cheng-Hao Liu, Sarthak Mittal, Nouha Dziri, Michael M. Bronstein, Pranam Chatterjee, Alexander Tong, and Joey Bose. Steering masked discrete diffusion models via discrete denoising posterior prediction. In The Thirteenth International Conference Learning Representations, 2025. URL httttp:////ooo.tt //ppn/eviie...... dd=0mmm8 OzN. 3,7
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. 2022 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10674-10685, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily I. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusson models with deep language understanding. Advances in neural information processing systems, 15536479-36499 2022.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances in Neural Information Processing Systems, 37:130136-1301888 2024. 2, 3, 6,
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (10.::/770.344 2017. 3
Jiaxin Shi, Kehang Han, Zhe Wang, Amaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in neural information processing systems, 37: 103131-103167, 2024. 2, 3, 4, 6, 17
14

---

Published as a conference paper at ICLR 2026
Jascha
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning. pp. 2256-2265. PMLR, 2015.
Yang Song, Jascha Sohl-Dickstein Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021. URL ht http:////pppeeeeeeeeeeeeeeeeeeee net.eeewwwweeeeeeeeeeeeeeeeeeeee forum? id=PxTIG12RRHS:
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008 30:3008-3021, 2020.
Ibrahim I Taskiran, Katina Spanier, Hannah Dickmänken, Niklas Kempynck, Alexandra Pančíková, Eren Can Ekşi, Gert Hulselmans, Joy N Ismail, Koen Theunis, Roel Vandepoel, al. Cell-type- directed design of synthetic enhancers. Nature, 626(7997)2112-223) 2024. 6
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023. 1,2
Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behbahani, Jonathan Weinstein, Niall M Mangan, Sergey Ovchinnikov, and Gabriel J Rocklin. Mega-scale experimental analysis of protein folding stability in biology and design. Nature, 620(7973): 434-444, 2023. 8
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. $228-8238, 2024. 2, 3, 4, 5, 6, 19, 20
Chenyu Wang, Masatoshi Uehara, Yichun He, Amy Wang, Tommaso Biancalani, Avantika Lal, Tommi Jaakkola, Sergey Levine, Hanchen Wang, and Aviv Regev. Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design. arXiv preprint arXiv:2410.. 1.110.1644. 13643, 2024. 3, 4, 5, 7, 8, 20, 23
Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling diffusion models. Advances in Neural Information Processing Systems, 36:31 36:31372-31403 2023. 3, 7,8
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference Learning Representations, 2022. URL httptttt///teeeeiiiiiiiee net/ / forum? 2orumm21-422cxxxxxxxg
Minkai Xu, Tomas Geffner, Karsten Kreis, Weili Nie. Yilun Xu, Jure Leskovec, Stefano Ermon, and Arash Vahdat. Ensrgy-based diffusion language models for text generation. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //tp::/s///peeeveeee net fortm?foum?ei fepprrmmid=ss2222F 9YCMXE.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 2025.
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Dream 7b, 2025. URL https: /hwun/ppppppp//pp gi thub. io it/ii///bll ht/blolg//2555 2tt////22////ree dream. 1,
Lingxiao Zhao, Xueying Ding, Lijun Yu, and Leman Akoglu. Improving and unifying dissretee&coominuousttine discrete denoising diffusion. arXiv e-prints, pp. arXiy-2402, 2024. 2
15

---

Published as a conference paper at ICLR 2026
Siyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. dI: Scaling reasoning in diffusion large language models via reinforcement learning. arXiv preprint arXiv:2504.12216, 2025. 2, 3, 7,
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming- Yu Liu, Jun Zhu, and Qinsheng Zhang. Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling. In The Thirteenth International Conference on Learning Representations. 2025. URL .et. ttee:/oooummi---TCCmimim....
attps://openreviln http:/wwwwwweeeeeeeeeeeeeeeeeeee
Lin Zheng, Jianbo Yuan, Lei Yu, and Lingpeng Kong. A reparameterized discrete diffusson model for text generation. arXiv preprint arXiv:2302.0573: 2023. 1,2
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followinn evaluation for large language models. arXiv preprint arXiv:231.007911 2023. 8
Fengqi Zhu, Rongzhen Wang, Shen Nie, Xiaolu Zhang, Chunwei Wu, Jun Hu, Jun Zhou, Jianfei Chen, Yankai Lin, Ji-Rong Wen, al. Llada 1.5: Variance-reduced preference optimization for large language diffusion models. arXiv preprint arXiv:2505. 0et.2/555.922.... 19223, 2025a. 3, 7, 8
Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for dif- fusion model alignment, In The Thirteenth International Conference on Learning Representations. 2025b. 2,3
16

---

Published as a conference paper at ICLR 2026
The appendix is structured as follows.
In Appendix A, provide detailed proofs of the theorems presented in the main paper and additional theoretical derivations.
In Appendix B, we provide more experiment details and hyperparameters for the experiments in the paper.
In Appendix we present more experiment results and ablations.
In Appendix D, we offer discussions on the limitations and broader impact of the proposed approach.
A PROOFS
A.1 PROOF OF THROREM 4.1
Theorem 4.1. The joint p² (Xu:le) induced by the optimal solutions {Ør (xo|xx, is also the optimal solution of the trajectory alignment objective in Eq. 5, with the chain reward
of Eq. 6
(XoOT, c) =B Te(xx-1, Xt. c) where 1, Xx, c) -log Br (30 10 () (exp( 37 (xx.c))]
Poot pxp( b, x(xo,)))
Proof. Leveraging Eq. 4, the optimal solution for each subproblem in Eq. 6 given by
I (()xxoc)) AI VSISST
1¹ (x(xxxxe, 7x(x,, c)
Pror 7
(xo((x c) exp
(12)
where Zx(x,, c) Ex, Pror (xo Xx, exp
o)) ( (xrr(x,,))
Enpe (xo (x0)), e), exp ( ci)x The
transition kernels p" (x1-1 Xxe, c) induced by the solutions be derived as
p" = Σ ()(xx)1110) xt),
(13)
Xa
= Σ Zi(xxe, c) Proe(xu c) exp ( (, r(xo, e)) g(xe- 1/0, xx),
(14)
(x X,, c)g()x)) (8: xx)
r(xx0,c) c)).
Pref (x X1,
exp
(15)
Zo c) (x xx)
Be
Prot (xo Xs, c)g(xx- (x0xxx)
Prof (roo((x--e-a) (x- Xx1, c)
exp
r(xxx
c)
7x( c)
Pret (xee(X4-11) (Xe,
30
(16)
exp (
(xx0 //)
2x c) Prot (x Ept (xolx
(17)
Br
where ples Xx,, =
Specifically, Eq. 13 holds due to the X parameterization of the transition kernel (see Austin et al. (2021); Shi et al. (2024)). Notably is straightforward to verify that Ther (Xo Xt, e) is a properly normalized distribution since
Peat (xi xx,c)
(x0 Σx, cxg xx0.x))
p(x, 18x.0) =
Deat (2 (x,c)
p X..)
17

---

Published as a conference paper at ICLR 2026
Plugging it back into the Markovian factorization of the reverse process, arrive at
t=
p (xxorrlc) = p(xr)
p (x,-1|x,,c)) c),
(18)
t=1
t=
= -p(x)) t=1
II
Prof Pros(Xs (x)- c)lEy, (x0xx [r(xo: c)]
(19)
Zx( c)
t=
Byl (N Ne- xe
exp
+ rx ())x,)))
= -p(xT) t=1
II
Pref (X,-
Xrx c)
(20)
Zi (x(),,c)
T
E (i 11x(..)
exp
De
r. cl)
= Prof Proff (Xo:T exp
log
(21)
(x,,
Ep E (x01 10000/1111110
xx.c)
exp
I xx(x ci)
PPeer(Xo c) exp
log
(22)
1=1
E (x
exp
Be
1 r(xo: c)
T
= -Peor c) exp
r (xp (x 1,,,
(23)
1
= Prof (Xo:T exp
13
rel X1 Xe, c)
(24)
3
f(xomr,e)
where (x Xx: c) = loog F 1.x [exp( [xxo,=]
Eq. 23 directly implies that the induced
( nn [exp( ( r(x,,)]
distribution (Xo:rl c) is the optimal solution of the trajectory alignment objective in Eq. 5 with F(Xo:T, = - - 1x c), which concludes the proof.
A.2 PROOF OF PROPOSITION 4.2
Proposition 4.2. Let f = arg min Le defined in Eq. 7. Then pa- (xolx4, c) is the optimal solution of the stepwise alignment objective in Eq. 6.
Proof. Recall the definition of (0) in Eq. 7:
(x) Exx, a [DKL (xol xr, (x) c)]]
(25)
Since the KL--divergence is minimized when the distributions are exactly matched, we have that the optimal 0* satisfies
pr c) po- (xol (xa/xes c).
(26)
By leveraging the definition of ÿ, and po. have
Zx(xx, c) Pref (xo x1, c) exp(r(xx, c)) Zo- (xxx c) Pret Pree (xoo Xx,, B1a (xol 8) (27)
which simplities
po- (xo (xe, c) Prof((X (x:: c)
Be Zo- (xe, c)
Z, exp(r(x0, c));
(28)
and finally gives us
po. (xol Xx, 1x0]Xe,,0 c),
Z'(xx, c) Pret exp(( r(xo: c)),
(29)
where Z'(x,, c) = (x..e)
1/3,
The proof is therefore completed.
18

---

Published as a conference paper at ICLR 2026
A.3 PROOF OF THE LOSS IN EQ. 7
Here we provide the detailed derivation on how to derive Eq. 7 from Li(0) step-by-step
We start from the definition of (() Eq. 6:
(0) Ex,,c [DRL (Xu X,, ll (xn) Xg, c)]l
(30)
Pr (xo Xx,
log
(31)
Po (xol Xx, C)
Pr (xol Xx, c)
log
(32)
Po (Xul X,, c)
Pr Xo: xx log
Pr (Xol Xx. c)/Pref (xol Xc. c)
(33)
Pref Xo, x1
po (xo) X1, c) Pret (xo) xt,
exp(r(X0, c))
Pr (xo X Prot X,,, C)
log
E (roe(Xoxxx1o) exp(r(Xx..)) c))
po (Xo x Prot (x Xx,, c)
(34)
exp(r(Xu, c))
(x X c) Pref (Xo Xc: c)
)]
IE (xor(x exp(r(xo, (xp(((xx)))) log =xx, (x)
(x Xe, c) Pref Xx Xr. C)
(35)
exp(r(X0: c))
Po(xoo1a Xcc c) Prof Xo Xe, C)
1r
)
= -xo,(xxoxx) (x0
Zr(c) log
+ +C,
(xe, 3t)
(36)
where the last step extracts the constant C' out of the numerator of log since is irrelevant to Recalling the definition of the implicit reward, which is given by fo(Xo, Xe, C,
B.(log (x x,, c) - log Pret (xolxe, c)), we have that [o(xxoxx.c)) 8t -exp(Fo(Xo Xx, c, (,)) Therefore, we can further simplify
(Poe((x|(x.c))
eex((r(xo: c))
Zz(x,, B,) (exp(Fo(xo: Xx, c, 8.)) c,,))) (()))) +C, (37)
(0) -Ee, Dee
Zr(c) log
where C' a constant irrelevant 0 and (c) and Zg (xe; c, Be) are the partition functions.
A.4 DERIVATION OF SDPO LOSS IN THE DPO SETTING
Recall our proposed loss function L(0):
exp(r(x)) c)) log exp(fo(x (1)
Xx (1)
[(0) = - -E, (
1=1 Ej=1 exp(r(x)) .c)) .))
Ey exp(Fo
( x 9c)
(38)
with
[log(x) fx t, c)) w(t)
log(x, (xe, c))
xx, B
(39)
w(t)
Here we derive specific instance of C(0) in the DPO pairwise preference setting, and draw connec- tion of it to Wallace et al. (2024).
In particular, in DPO preference pair setting for each context there are two completions, namely, N = 2 in our case. Furthermore, one completion is labeled as the preferred (chosen) response (w) and the other rejected sample x (1) Since explicit real-valued reward on the chosen and Xi
rejected sample is provided, the Bradley-Terry (BT) model (Bradley & Terry, 1952) is adopted, which
corresponds to, in our case, setting r(x(w) .) and r(x()) c) -oo. Under this specification,
19

---

Published as a conference paper at ICLR 2026
(() is simplified
(()) - c.x xo)l
exp(r(X0 (we) .c)
exp(Fe(x) (w)) xx(()))
llog
exp(r(x)))) .c) + exp(r(x))) !1) c))
cxp (w) x )) + exp(fp exppfo((xt !!! (x
xx
+ exp(r(x0 c))
exp(řn(Xp)
log
exp(r(x)) w) ,c) + exp((rxx) c))
exp(Fa(x)) [c) x ))
+ exp vo (x x (1))
= - - xo)
log
+
(40)
1 1 cxp(fo(xx) !1 xx Fo (w) x (1)))) 1+0
-_ xo)
log
(41)
1 1 exp((ixsxx)) (1) xx(1)) ) - i6 =x(xx (") x (m))
= -B xo) log (fo(x)) (1)) xx,") (ar)) ) -o(xx) x; xx
(1)
(42)
where Fe (x(x) (w) x((r) is shorthand for (x(xx)
(w) 3,) and similarly for the losing sample. Eq. 42 underscores an interesting connection of our loss to that of Wallace et al. (2024) specifically in the
(w))
(w)
DPO setting, since both share the same form of negative logsigmoid over the margin between the implicit rewards of the winning and losing sample, with the difference being the definition of the whether using Gaussian diffusion (Wallace al., 2024) or
implicit reward (Eq. 11), depending discrete diffusion as in this work. Notably, since we leverage a general formulation of stepwise decomposition that reduces the problem stepwise distribution matching objectíve, are able generalize to the setting with arbitrary number of samples and reward model, which is not revealed Wallace et al. (2024).
B MORE EXPERIMENT DETAILS
B. DNA SEQUENCE DESIGN
We use the pre-trained model and fine-tuning reward oracle from Wang et al. (2024) for finetuning with SDPO. In the first stage of finetuning, we train on the original enhancer dataset (also used for pre-training), without using re-labeled samples. In the next two stages, generate 10000 samples from the finetuned model, label the samples with the reward model, and continue finetuning on the relabeled data. In all three stages, use the original pre-trained checkpoint the reference model. We provide hyperparameter configurations in Table
Table 5: Detailed hyperparameters for DNA design task.
Stage
# relabeled samples
|3
N
# Epochs
Learning Rate
Stage
0
0.92
25
10
9e-5
Stage
10k
0.4
200
2
le-5
Stage
10k
0.064
918
2
7.4e-6
B.2 DETAILED HYPERPARAMETRE FOR PROTEIN INVERSE FOLDING TASK.
We also use the pre-trained model and fine-tuning reward oracle from Wang et al. (2024) for finetuning with SDPO. Likewise, we finetune on the original pre-training dataset without re-labeling any new samples. In the second stage, we re-label 12800 generated samples with the reward oracle, then continue finetuning with SDPO. Differing in setup from the DNA experiment, find that using the previously finetuned checkpoint as a reference model during Stage 2 results in superior performance. We provide hyperparameter configurations in Table 6.
20

---

Published as a conference paper at ICLR 2026
Table Detailed hyperparameters for protein inverse folding task.
Stage
# relabeled samples
|3
N
# Epochs
Learning Rate
Stage
0.047
25
9
5.7e-6
Stage? 2
12.8k
0.063
200
5
8.5e-5
Algorithm SDPO Training Algorithm
Input: Initial dataset D( pretrained discrete diffusion model Pref and trainable model po-
I: for iter & = 0, ... K do
2:
for step / 0, ... do
Sample [(x) c) ((2) each prompt the batch
Sample x g(x: (x)) given x
5:
Compute loss [(0) via Eq. 10
P SDPO loss
6:
AV)((0) Generate ()++1)
i
Gradient update
7:
via po (xo:r//c) = p (0 c)
Optional iterative labeling
8:
Pref Po
b Optional reference model update
Return: Optimized model ps
B.3 LANGUAGE MODELING
We leverage the open-source checkpoint' of L'LLLD--B-IIntrrucc (Nie et al., 2025) as the base model perform our SDPO. We use UltraFeedback (Cui al., 2023) dataset labeled by Meng al. (2024) as the finetuning dataset We operate in the pairwise setting with N = 2 on the dataset, with labeled pairs of winning and losing samples with rewards. We use 8 Nvidia 80G A100 GPUs with DeepSpeed enabled during finetuning, due to the scale of the model. We use per device batch size 2 and gradient accumulation 16 steps, leading to an effective global batch size of 256. We set the learning rate reuse the inference
to 1e-6 and 3 to 1.0 and train the model for 2 epochs. At inference time, peperaaramesess adopted in Nie et al. (2025) for GSM8K without any additional tuning, which include total length 256, block size 8, and total number of steps 256. For IFEval and AlpacaEval 2.0, we keep the same set of hyperparameters except setting block size to 32. We always adopt the low confidence remasking strategy, following Nie et al. (2025).
B.4 COMPLEXITY ANALYSIS
Computational and memory complexity. As an offline preference optimization approach, SDPO not bottlenecked by online data generation during training, and the offline data generation can be fully parallelized. In detail, it is of ONN + LDD²)) for computational complexity and O LD + D2)) for memory complexity, where N is the number of Monte-Care samples, M is the number of attention blocks, L is sequence length, and D is the latent dimension. The complexity comes from standard Transformer-based architecture, on top which the coefficient of is multiplied for Monte-Carlo estimation, making it irrelevant of diffusion steps. The inference complexity remains unaffected.
B.5 ALGGRRIHH DETAILS
For completeness, include the entire training algorithm in Alg. 1. We also include the MC estimator (Alg. 2) to compute the trajectory-level reward r, used in our reward correlation analysis in ablation study. Note that the function Constrain xx-1) means setting every unmasked token in x to the same place in x with the same token value.
tra Crafeebacck
21

---

Published as a conference paper at ICLR 2026
Algorithm 2 SDPO Trajectory-level Reward Evaluation
Input: model pa and prot, number MC samples K, reward model r(-)
1: for step = T. ,0 do
Sample X (X
3: Sample {x/) ]i=1 from Peet (Xolxee
(2) )
4: Estimate the denominator d = (xol()) exp o. (x0, c))] 2 + Eiz exp (xx 1 .c)
(
5: (()
Constrain (x) .x(-1)
+
6: Estimate the numerator (x0xxxxx)
exp
3. (xxx x E exp (xx)) c))
Compute the stepwise reward r log
Return: chain reward
-E,, Tt
Table 7: Dataset size ablation results. Even in highly limited data settings (< 10% of original dataset), SDPO achieves strong results.
Pred-Activity
ATAC-Ace
3-mer Corr
App-Log-lik
25k + 20k relabeled
5.56
0.40
0.795
-237
50k + 20k relabeled
6.02
0.756
0,793
-248
700k + 20k relabeled
6.30
0.948
0.900
-246
Table 8: Detailed results on different values of 3 on DNA design task.
B
Pred-Activity (median)
Pred-Activity-std
ATAC-Acc t (%)
0.10
6.33
0.68
0.84
0.15
6.29
0.68
0.85
0.20
6.26
0.70
0.86
0.25
6.22
0.74
0.89
0.30
6.17
0.74
0.91
0.35
6.15
0.75
0.92
0.40
6.08
0.76
0.92
0.45
6.01
0.85
0.92
0.50
6.00
0.83
0.93
0.55
5.97
0.85
0.94
0.60
5.90
0.84
0.94
0.65
5.78
0.90
0.96
0.70
5.79
0.90
0.95
C MORE EXPERIMENT RESULTS
C.I ABLATION STUDY ON DATASET SIZE
To study the effect of the data quantity on model performance, we perform additional ablation on the DNA sequence task Table 7. Our results demonstrate that SDPO can achieve strong results even in highly limited data settings, where the first stage of fine-tuning uses small random subset of the original training dataset (700k samples). We follow this with two stages of iterative re-labeling and fine-ture -uningn according to our established setup.
C.2 MORE RESULTS ON B
We provide detailed ablation results on different values of 3 Table 8 and Table for DNA sequence design and protein inverse folding tasks, respectively. We observe that when / becomes smaller, which indicates less regularized distribution w.r.f. the reference distribution, the model granted more flexibility in optimization and generally achieves higher reward. Meanwhile, some other metries such as ATAC-Ace that relates to the stability the generated sample will tend to drop due to over-optimizing the model.
22

---

Published as a conference paper at ICLR 2026
Table 9; Detailed results on different values of on protein inverse folding task.
beta
Pred-ddG (median) f
Pred-ddG-std
%(ddG>0) (%) 1
0.05
1.026
1.001
0.742
0.10
1.119
1.093
0.760
0.25
0.348
1.176
0.569
0.50
-0.058
1.010
0,461
0.75
-0.350
1.215
0.430
1.00
-0.410
1.192
0.418
C.3 MORE RESULTS ON PROTEIN INVERSE FOLDING TASK
We provide additional experimental results the protein inverse folding task, where demonstrate strong performance in both the re-labeling setting (a.k.a. iterative labeling) and the re-labeling-free setting (only using the original dataset for finetuning). The results are depicted Table 10. this set of experiments, we obtain the runs that achieve better balance between the target reward (Pred-ddG) and other metrics, such as scRMSD and Success rate. In particular, the experiment with re-labeling achieves better Pred-ddG (1.179) compared with the baseline DR A AKES Wang et al. (2024) (1.095) while obtaining better scRMSD and matching its success rate of 78.6%. Furthermore, we observe that without iterative relabeling, SDPO can already obtain strong performance, which additionally underscores the robustness of our design.
Table 10: Additional evaluation on the protein inverse folding task.
Stage
# relabeled samples Pred-ddG (median) $(ddG)>> 0 scRMSD (median) *(scRMSD)< 2 Success rate
w/o Re-labeling
0
1.165
85.5
0.936
90.5
77.0
w/ Re-labeling
12.8k
1.179
84.8
0.914
92.0
78.6
C.4 RESPONSE SAMPLES ON LLADDA-B--NSTRUSS
Here we present several samples using the questions from GSM8K Cobbe al. (2021). It is observed that by employing our SDPO to LLaDA-8B-Instrue the finetuned model exhibits clearer reasoning logic and better presentation of the response.
23

---

Published as a conference paper at ICLR 2026
Response Samples GSM8K
Question [Few-shot Prompts] I Janet's is ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?
Answer by LLaDA-8B-Inss -Bs-rust Nie et al. (2025)
Janet's ducks lay 16 eggs per day. She eats eggs for breakfast every morning, so she has 16 -3 1==ii1--==13 13 eggs left. She bakes 4 eggs for her friends every day, she has 13 4 1;13-4=9;;9 eggs left. She sells the remaining eggs at the farmers' market for $2 per egg, so she makes 9 $2 $ 1 18 every day.
#### 18
Answer by
-A-DD-----Isttut-SDD-- (Ours)
To determine how much Janet makes every day at the farmers' market, we need to follow these steps:
1. Calculate the total number of eggs laid by Janet's ducks per day. 2. Subtract the number of eggs Janet eats breakfast and bakes for her friends. Calculate the number eggs left for sale, Multiply the number of eggs left for sale by the price per egg to find her daily earnings.
Let's go through these steps:
1. Janet's ducks lay 16 eggs per day. 2. She cats 3 eggs for breakfast every morning. 3. She bakes
4 eggs for her friends every day.
Now, let's calculate the number of eggs left for sale:
16 3 - 4 = 9
So, Janet has 9 eggs left for sale each day.
Next, we multiply the number of eggs left for sale by the price per egg:
9 x 2 18
Therefore, Janet makes $18 every day at the farmers' market.
The final answer 18
Response Samples on GSM8K
Question [Few-shot Prompts] Carlos is planting a lemon tree. The tree will cost $90 to plant. Each year it will grow 7 lemons, which he can sell for $1.5 each. It costs $3 year water and feed the tree. How many years will it take before he starts earning money on the lemon tree? Answer by -LaDDA-B-IInstruee Nie et al. (2025)
To determine how many years will take for Carlos to start earning money on the lemon tree, we need calculate the profit per year and then how many years it will take for the net profit to cover the initial cost of planting the tree.
** *aaccllate the net profit per year:**
Revenue from selling lemons: 7 lemons $1.5 per lemon $10.50
Cost to water and feed the tree: $3
Net profit per year: $10.50 $3 = $7.50
** #etermmin the number of years needed to cover the initial cost:*
Initial cost of planting the tree: $90
Net profit per year: $7.50
Number of years needed: $90 / $7.50 per year 12 years
Therefore, will take Carlos 12 years to start earning money on the lemon tree.
Answer by -..DA--B--nstrut--SSS
(Ours)
First, let's calculate the annual revenue from selling the lemons:
-The tree grows lemons each year.
Each lemon sells for $1.5.
Annual revenue
lemons * $1.5/lemmon $10.5.
Next, let's consider the annual cost of watering and feeding the tree:
It costs $3 per year to water and feed the tree.
Now, let's calculate the net annual profit:
Net annual profit Annual revenue Annual cost
24

---

Published as a conference paper at ICLR 2026
Net annual profit = $7.5.
Finally, let's determine how many years it will take for the net annual profit to cover the initial cost of planting the tree:
Number of years = Initial cost / Net annual profit
Number years $90 / $7.5 12 years.
Therefore, it will take Carlos 12 years to start carning money on the lemon tree.
D DISCUSSIONS
Limitation. Our framework relies on the reward model while, in practice, such model may be noisy even harmful, which will potentially lead to undesired consequence.
Broader impact. As demonstrated in the paper, our approach can help finetune pretrained discrete diffusion models for better alignment towards certain reward, which can have significant practical impact in various domains, such as natural language modeling and biochemical sciences. Our approach can building helpful and harmless chatbots and even performant and effective large language model agentic systems.
critical building block towards designing useful DNA and protein sequences,
E
THE USE OF LARGE LANGUAGE MODELS
We did not use Large Language Models for research ideation and paper writing in this work.
25


Reviews:
[{"review_id": "f1SffJ8obj", "reviewer": "Reviewer_WErC", "summary": "This paper proposed an offline preference optimization method for aligning the discrete diffusion models with rewards. By decomposing the loss for trajectory alignment into a set of stepwise objectives, the proposed method achieved efficient optimization. Fine-tuned diffusion models outperform RL-based and inference-time guidance methods in terms of reward optimization while maintaining strong sequence quality/naturalness. The efficacy of the method was verified across different domains including DNA, protein, and natural language.", "questions": "1. Some clarification questions about methods details.\n- Are you directly using CG which involves a time-dependent classifier as proposals in TDS? It seems that in the original TDS paper, they were using the reward model on the clean data $x_0$ which was approximated from $x_t$.\n- Is $p_{ref}$ the pertained diffusion model?\n- Is each token of $X_0$ a one-hot vector?\n- In the first two tasks, DNA and protein, what is the number of Monte-Carlo samples $N$? is $c$ in $r(x,c)$ empty?\n2. In the proposed framework, do the reward models always need to be differentiable? From my reading of Eq. 11, differentiability is not necessary.  If not, did you try to apply your method on non-differentiable targets like the docking scores of small molecules?\n3. Could you please provide more intuitions about why the ATAC-Acc of the generated DNA sequences decreases but\nPred-Activity increases with a small β, despite their being positively correlated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well structured and clearly written. The overall presentation flows smoothly and is easy to follow. Most details are clear. \n2. The experiments covered diverse relevant domains and are comprehensively evaluated. Beyond achieving the best rewards among all compared methods, the sequences generated by the propose method also maintain comparable naturalness (close to natural sequences, or low NLL) and diversity (entropy).", "weaknesses": "1. Missing parts in the related work section. Although the authors did experiments with some inference-time guidance methods in the experiments for DNA and protein, I think it is still worth to mention such baselines in the related work about diffusion alignment. The pros and cons of these two different alignment paradigms, i.e., adding guidance during the inference time without training diffusion models like TDS or CG, or fine-tuning diffusion model through preference optimization like SDPO, could be further clarified in the main text. \n2. Although the authors mentioned that dividing a trajectory alignment loss into the objectives for individual steps would make the optimization more efficient, it is not well supported by the experimental results. It could be good if some proxies for training efficiencies, e.g., training speed per batch/convergence speed , are used for comparison between previous preference optimization methods like D2-DPO or DRAKES and SDPO\n3. It would be clearer if the complete training algorithm was listed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pHfe62OGx1", "reviewer": "Reviewer_Gnse", "summary": "The work aims to fine-tune a discrete diffusion model (with trajectory-level distribution $p_\\theta(x_{0:T})$) to sample from the tilted distribution  $p_{\\theta}(x_{0:T}) \\exp(\\frac{1}{\\beta}r(x_{0:T}))$. It proposes a surrogate objective which decomposes the trajectory-level problem into subproblems where each posterior over $x_0$: $p_\\theta(x_0|x_t)$ is fine-tuned to the target $p_\\theta(x_0 | x_t) \\exp(\\frac{1}{\\beta_t}r(x_0))$, which side-steps issues with expensive likelihood evaluations over the trajectory. The work focuses on the offline-setting where a reward-labeled dataset (with data drawn from $p_\\theta$) is available, and solves each subproblem using a variant of forward-KL/cross-entropy loss, where samples from the data-set are importance weighted using the reward. The method is shown to outperform other baselines on the tasks of DNA sequence design, protein inverse folding, and preference alignment for language modelling.", "questions": "1. Why not formulate Equation (8) as $D_{\\mathrm{KL}}( p_\\mathrm{ref}(x_0 | x_t, c) \\exp(r(x_0, c)/\\beta_t) \\mid \\mid p_\\theta(x_0 | x_t, c))$ directly? Is there an advantage to the loss as its presented?\n\n2. In Appendix B, it is mentioned that the first stage of training is done on existing/original datasets for fine-tuning. This in theory creates a mismatch with the training objective since it assumes the pretrained model’s distribution perfectly coincides with its pretraining dataset, which is likely not true. Does the performance of the method degrade if samples from the pretrained model are used instead, for the initial fine-tuning stage?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Side-stepping the trajectory-level nature of discrete diffusion fine-tuning is an important problem, and the proposed method is a useful step in this direction.\n2. The work is well organized.\n3. The preference alignment task in language modelling demonstrates scalability of the method to larger models.\n4. The extension to iterative training is useful.\n5. The ablations section is well-done, and clarifies important aspects of the method (namely its sensitivity to the sample size $N$). \n6. The evaluations are done on a comprehensive set of metrics for DNA sequence design and protein inverse folding.", "weaknesses": "1. The method assumes access to $p_\\theta(x_0|x_t)$ - however, in discrete diffusion, we only have access to a factorized approximation (or equivalently, the **mean**) (eg. see section 3 of (Shi et al., 2024)) of this posterior (through $f_\\theta(x_t,t)$). Equation 12 conflates these two things as being equal. \n    - This additionally calls into question the applicability of the theoretical analysis in Theorem 4.1 - does the result still hold for the factorized approximation of the posterior $p_\\theta(x_0 | x_t)$?\n2. The variance of the importance weights (or equivalently the effective sample size) should be evaluated for various tasks - since a reduced sample size appears to be a potential limitation of the method for more challenging rewards.\n3. The bias in using the stepwise approximation should be better analyzed:\n    - Are there real rewards for which the method is unbiased (at sampling from the trajectory level reward-tilted distribution)?\n    - If not, a visualization of how the method is biased would clarify the methods drawbacks (for instance on a toy task where the exact reward-tilted posterior can be visualized).\n\n4. The choice of baselines lack a more standard policy gradient RL method. I understand that the method tackles an offline setting, but I think at least one comparison to such a method (such as diffu-GRPO (Zhao et al., 2025)) would be helpful in understanding the sample efficiency of the proposed method. In particular this is because such methods have been shown to be more scalable to challenging problems than DRAKES.\n\n5. The work is in need of proofreading for various grammatical mistakes.\n\nFor now I am recommending a weak reject, mainly due to point 1 and its impact on the theoretical results (and secondarily, point 4 as an important missing baseline) above. \n\nJiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and\ngeneralized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329, 2024.\n\nSiyan Zhao, Devaansh Gupta, Qinqing Zheng, and Aditya Grover. d1: Scaling reasoning in diffusion\nlarge language models via reinforcement learning. 2025. arXiv:2504.12216.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yqULWfOKIw", "reviewer": "Reviewer_pkbk", "summary": "The paper propose a preference optimization method for discrete diffusion models using diffusion trajectory alignment. It decompose the alignment problem into a set of stepwise objectives, instead of directly optimization for the reward on the final output. The authors provide the optimality guarantee under the assumption of the additive factorization of the trajectory reward. Experiments on DNA sequence, protein sequence, and language demonstrate its effectiveness.", "questions": "Please refer to the **weaknesses** section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Experiments on different domains demonstrate strong performance across different metrics.\n- The idea to leverage the intermediate step information for alignment of discrete diffusion models is well-motivated.", "weaknesses": "- Although claimed as a trajectory alignment algorithm to maximize the stepwise reward, the algorithm does not really use the stepwise reward. Instead, it only uses the reward of the clean sample $r(x\\_0,c)$ . Actually, the trajectory alignment here is more like a simple decomposition of the joint distribution for the data itself, while irrelevant to the reward-tilted part.\n- In Theorem 4.1, the optimality of the trajectory alignment objective is achieved when the reward $\\hat{r}(x_{0:T},c)$ is defined as the complicated chain reward formulation. Therefore, the theorem actually defines a very strong condition specialized for the trajectory alignment objective to prove that the trajectory alignment objective is optimal under this specialized condition. Although we may gain some interesting insights from the condition, the theorem itself is not very meaningful.\n- The final objective relies on the coefficient w(t). Could the author provide more ablations on different choices of w(t) to better demonstrate its effect?\n- The authors turn this trajectory alignment objective into an off-policy preference optimization objective. For general RL algorithms of LLMs, it is known that on-policy RL works better than off-policy RL methods. Why does the author choose off-policy preference optimization as the final objective, instead of an on-policy one, and does SDPO suffer from the same issue as off-policy RL for LLMs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wn9x4D2o5Y", "reviewer": "Reviewer_KTkz", "summary": "This paper proposes an offline fine-tuning method for discrete diffusion, without rely on the reward gradient calculations.", "questions": "- In Line 262-263, the authors show that $x_t$ cannot be calculated in offline setting, so that the authors drop it. Although it’s understandable, could the authors provide more in-depth analysis on this? Will this cause big problems?\n- In iterative labeling, authors present that they will update the datasets from latest models. Does it mean that the overall framework is not completely offline anymore?  \n- While I appreciate the authors add diversity quantity in fine-tuning diffusions, in Table 2, why the pretrained model doesn’t have the highest entropy? My understanding is that, no matter how you can keep diversity, during fine-tuning of reward maximization, adding diversity measure from pretrained model of 35.2 to 42.3 seems strange to me. \n- One important measurement is the computation cost. Compared with your baselines, did your methods add additional computation costs, especially the reward calculations? (since scientific reward calculation may be very cost) How about the convergence rate under same computation compared with the baselines?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The overall framework and results are technical sound. The proposed fine tuning methods without gradient calculation are important in scientific area.", "weaknesses": "- Performances: in Table 1, it’s reported that DRAKES achieve Pred-Activity as 5.61 while proposed SDPO is 6.30. However, in the original paper, DRAKES have two versions (with and without KL), which is 5.61 and 6.44, respectively. Why the authors only pick up the lower value one? If DRAKES can achieve 6.44, does it mean the performance gain by SDPO is low? Since the primary optimization reward here is Pred-Activity, I think this is a big experimental flaw.\n- Baselines: authors may miss some important baselines. For fine-tuning discrete diffusion without gradient, DDPP [1] and DDPO [2] serve as strong baselines. Why not add them as the baseline methods?\n\n[1] Rector-Brooks J, Hasan M, Peng Z, et al. Steering masked discrete diffusion models via discrete denoising posterior prediction[J]. arXiv preprint arXiv:2410.08134, 2024.  \n[2] Black K, Janner M, Du Y, et al. Training diffusion models with reinforcement learning[J]. arXiv preprint arXiv:2305.13301, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)