Diffusion Alignment as Variational Expectation-Maximization
Authors: Jaewoo Lee
Minsu Kim
Sanghyeok Choi
Inhyuck Song
Sujin Yun
Hyeongyu Kang
Woocheol Shin
Taeyoung Yun
Kiyoung Om
Jinkyoo Park
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Model
Alignment
RLHF
Test time search
Primary area: generative models
TL;DR: Diffusion Alignment as Variational Expectation-Maximization (DAV) alternates test-time search (E-step) and forward-KL distillation (M-step) to align continuous and discrete diffusion models.

Abstract: Diffusion alignment aims to optimize diffusion models for the downstream objective. While existing methods based on reinforcement learning or direct backpropagation achieve considerable success in maximizing rewards, they often suffer from reward over-optimization and mode collapse. We introduce Diffusion Alignment as Variational Expectation-Maximization (DAV), a framework that formulates diffusion alignment as an iterative process alternating between two complementary phases: the E-step and the M-step. In the E-step, we employ test-time search to generate diverse and reward-aligned samples. In the M-step, we refine the diffusion model using samples discovered by the E-step. We demonstrate that DAV can optimize reward while preserving diversity for both continuous and discrete tasks: text-to-image synthesis and DNA sequence design. Our code is available at https://github.com/Jaewoopudding/dav.

Published as a conference paper at ICLR 2026
DIFFUSION ALIGNMENT AS
VARIATIONAL EXPECTATION-MAXIMIZZTONN
Jaewoo 2
Minsu Kim
Sanghyeok Choi Taeyoung Yun'
Inhyuck Song'
Sujin Yun Jinkyoo Park 1.6
Hyeongyu Kang'
Woocheol Shin'
Kiyoung Om'
KAIST
"MonooseAA
Miila- Quebec AI Institute Omelet
University of Edinburgh
Mila, Université de Montréal
ABSTRACT
Diffusion alignment aims to optimize diffusion models for the downstream objec- tive. While existing methods based on reinforcement learning direct backpropa- gation achieve considerable success maximizing rewards, they often suffer from reward over-optimization and mode collapse. We introduce Diffusion Alignment (DAV), a framework that formulates diffusion alignment as iterative process alternating between two complemen- tary phases: the E-step and the M-step. In the E-step, we employ test-time search generate diverse and reward-aligned samples. In the M-step, refine the dif- fusion model using samples discovered by the E-step. We demonstrate that DAV optimize reward while preserving diversity for both continuous and discrete tasks: text-to-image synthesis and DNA sequence design. Our code is available at https: /ttts:://githke com wwwooopdddindgg dav.
as Variational Snpectation-Maximizaa
INTRODUCTION
Diffusion models (Ho al., 2020; Song et al., 2021) excel at generating diverse domains, from image synthesis (Rombach et al., 2022; Ho et al., 2022), robotics (Chi et al., 2023) to computational biology (Sahoo et al., 2024). Beyond generating high-likelihood samples, many real-world applications necessitate samples optimized for external criteria; the aesthetic qual- ity of images (Schuhmann, 2022), or the biological activity of DNA enhancers (Gosai et al., 2023).
samples across
To align the diffusion model with downstream objectives, various fine-tuning methods for diffusion models have been proposed. Prior works can be typically divided into two categories: (1) RL-based fine-tuning and (2) direct backpropagation. RL-based approaches (Fan al., 2023; Black et al., 2023; Venkatraman et al., 2024) optimize the parameters diffusion models with a reverse- KL ob- jective using on- on-policy data. This combination is prone to mode-seeking behavior, which may cause premature convergence and mode collapse, severely degrading sample quality and diversity (Kim al., 2025c). Direct backpropagation (Clark et al., 2023; Prabhudesai et al., 2023) attains higher sample efficiency but depends on sharp, brittle gradient signals from learned reward functions (Tra- buceo et al., 2021), often leading severe over-optimization (Skalse et al., 2022). Thus, there is pressing need for a fine-tuning framework that can effectively maximize rewards without sacrificing the diversity and naturalness of the pretrained diffusion model.
To this end, we propose Diffusion Alignment as Variational Expectation-Maximiiaan Inspired by (Levine, 2018), our framework is based on the variational Expectation- -aaitttiiii-aaiiizzattiooo (EM) algorithm (Neal & Hinton, 1998; Jordan et al., 1999), iteratively alternating between the E- step for exploration and the M-step for amortization. The E-step (Exploration) aims to discover diverse and high-reward samples from the variational posterior. To effectively capture the multi- modal structure posterior distribution, we invest additional test-time computation (Kim et al., 2025c; Zhang al., 2025) via techniques such as gradient- based guidance (Grathwohl et al., 2021; Guo et al., 2024) and importance sampling, enabling a thorough exploration promising regions.
(DAV).
The subsequent M-step (Amortization) updates pe pv- by distilling the knowledge from the discovered samples back into the parameters of the diffusion model. Unlike conventional RL-based
"Correspondence jaewon kci

---

Published as a conference paper at ICLR 2026
methods that optimize reverse-KL. divergence, mode-seeking objective that concentrates single dominant mode (Fan et al., 2023; Venkatraman et al., 2024; Uchara et al., 2024b), our M-step update corresponds to minimization of the forward-K KL divergence, mode-covering objective that encourages the model to cover all diverse modes discovered through the E-step (Chan et al., 2022). Therefore, the iteration of EM steps leads to synergetic cycle, where the M-step adaptively refines the model towards a multi-modal aligned distribution, and the improved model in turn enables the E-step to gather samples from more aligned distribution while preserving diversity.
To demonstrate the versatility of our method, we apply DAV to align diffusion models in both continuous (text-to-image synthesis) and discrete (DNA sequence design) domains. For the image synthesis task, we fine-tune the Stable Diffusion v1.5 model (Rombach et al., 2022) to optimize for given external rewards-including aesthetic quality (Schuhmann, 2022) and non-differentiable ob- jectives like image compressibility and incompressibility (Black et al., 2023)-while mitigating the mode collapse of the images. For DNA sequence design, we fine-tune discrete masked diffusion model (Sahoo et al., 2024) to design DNA enhancers (Gosai et al., 2023) that achieve high target activity while preserving the naturalness and diversity of generated sequences.
RELATED WORKS
2.1 DIFFUSION ALIGNMENT
Fine-tuning approaches. RL-based fine-tuning frames the denoising process as a sequential decision-making problem, optimizing a policy maximize black-box reward function (Black et al., 2023; Fan et al., 2023; Uehara et al., 2024b; Su et al., 2025). In parallel, direct backprop- agation approaches propagate the gradient signal from the differentiable reward function through the diffusion denoising chain, greatly improving sample efficiency (Xu et al., 2023; Clark et al., 2023: Prabhudesai et al., 2023). Despite progress. both fine-tuning methods suffer from reward over-optimization (Skalse et al., 2022), due to the mode-seeking behavior of reverse-Ki KI. optimiza- tion in reinforcement learning (Korbak et al., 2022) and the brittle gradient signal from the reward model (Kim et al., 2025b). Recently, Liu et al. (2024); Domingo-Ente Enrich et al. (2025) propose to fine-tune the continuous diffusion model using the gradient signal of the reward function to follow the tilted distribution Ppretrained (x) exp(r(x)), where r(z) is the reward function. While effective at improving performance and mitigating over-optimization, these methods require a differentiable reward function and are not straightforward extend to discrete diffusion models.
Test-time inference approaches. Test-time inference approaches allocate additional computation during generation to find aligned outputs without altering model weights. Techniques include guid- ance (Dhariwal& & Nichol, 2021; Chung et al., 2023; Yu et al., 2023; Bansal et al., 2023), and test- time search methods (Ma et al., 2025; Zhang et al., 2025; Li et al., 2024b; Singhal et al., 2025; Kim et al., 2025a; Li et al., 2025; Jain et al., 2025). Despite their effectiveness in bypassing additional post-training phases, these methods face drawbacks: guidance-based approaches often suffer from underoptinization (Kim et al., 2025a), and search-based algorithms demand substantial computa- tional overhead, rendering them impractical for real-world applications.
DAV unifies the strengths of these two paradigms by leveraging test-time search to collect diverse, aligned samples and then distill the gathered knowledge back into the model through principled Expectation- Expectation-Maximizattee Maximization algorithm. By amortizing test- test-time search into the parameters of the dif- fusion model, DAV achieves strong alignment and sampling diversity without extensive computation requirements at inference time. While recent methods (Liu et al., 2024; Domingo-Enrich et al., 2025) focus on continuous diffusion models and rely on the differentiability of the reward function, DAV naturally extends to both continuous and discrete diffusion without requiring any assumption on the differentiability of the reward function, making it a more general and widely applicable framework.
BACKGROUNDS
3.1 DIFFUSION MODELS
Diffusion models are generative models that learn data distributions by adding noise to data and then training how reverse the noising process (Sohl-Dickstein al., 2015). They comprise two
2

---

Published as a conference paper at ICLR 2026
opposing Markov chains, forward process that gradually adds noise the data, and learned reverse process py that denoises the state to recover the original data:
((r zo) II ( (x-1)),
po( pr(er) II po xx);
where Zpp ~ Qdata is clean data, 2r is pure noise, and pr(xr) is a simple base distribution from which can easily sample. Depending the data modality, can parameterize diffusion processes with a Gaussian distribution for continuous diffusion or Categorical distribution for discrete diffusion.
The details are elaborated in Appendix G.
The reverse process Pe is learned by optimizing the evidence lower bound (ELBO) on the data ogg-iikelhood r [log (xo)) which can decomposed as follows:
Egg 2000 [_(Dxu((((=11)0),0))) (p((xe11==))) (ka(((pr(zo) pr(r)))
3.2 MARKOV DECISION PROCESS
We formulate the fine-tuning of the reverse diffusion process as a Markov Decision Process (MDP). Following prior works (Fan & Lee, 2023; Black et al., 2023), define Aiite-hhrizon MDP with sparse reward and deterministic transitions, denoted as the tuple (S, AA P, r, 7, Po). where S the state space, A the action space, P the transition dynamics, ? [0. 1] the discount factor, x A + R the reward function, and Po the initial state distribution. Unlike previous approaches, incorporate the discount factor y. Within this MDP, we want to fine-tune our diffusion policy m. The specific formulations are as follows:
St - (IT-I, t) TT(at (s) _ po( P(s++1|51, at) & 6127
R((ro) = 1 10
A TT
po(s)) (pr,ōr)
r(st, ax)
otherwise.
R(zo) is an external reward function defined on the clean data space. For brevity, we mainly use notations with 3y (rather than (st, (t)) )) in the subsequence sections. Also, with slight abuse of notation, we often denote r (II2) instead of r((2,, (), xt))
3.3 KL-DIVERGENCE REGULARIZED REINFORCEMENT LEARNING
Following Abdolmaleki al. (2018); Wu et al. (2019); Kumar al. (2019), we consider maximiza- tion of the KL-regularized RL objective for given diffusion model Pprior Pa. i.e.,
arg max E Peeaxe Po'
(r(z,, xx-1) - (pr (xx))
|Priior (Ie (xx
(1)
where 7 (IT, 2T ... 20) trajectory sampled from Po and Pptiors and >> controls the strength of the regularization. We define soft Q-function of statc-action pair (æe, xz-1) 1) as follows:
(Fe, r( xt-1)
E Eropp-
Σ
t- (r(#a) DKL. (po (-12ra) (Ppiiorr Tt, Te-
(2)
Following Uchara al. (2024a), KL-regularized soft Bellman equations are given by:
V (xx) = log E Er,_ Ex..sppporr(ee) exp( & (xe, 21-
(3)
= ((r,, + By, - 1o b- ( V (x:-1)]
(4)
Under the discounted diffusion MDP with sparse reward and deterministic transition of Section 3.2, we can set terminal conditions (x)( = 0. Q (x1. vo) = r(æu), and approximate the soft Q-function by using Tweedie's formula (Efron, 201 Li al., 2024b) follows: (2n xx-1) -*-1 -(-1r((0(((11))
(5)
where (xx) = Exo-penne [zo] denotes the approximated posterior mean. (See Appendix for details.)

---

Published as a conference paper at ICLR 2026
E-step) Sampling from 7/
M-step) Adjust Pot
Potrt
E-step) Sampling from new 7/++1
(t)(t) (t)
(t
Pytei (t)
(E(r)
Pore: (T)
()
Prior Distribution
Posterior Distribution
Test-time Search
Conceptual illustration of DAV. DAV alternates between E-step, where trajectories are obtained via test-time search, and M-step, where the diffusion model parameters are updated by amortizing the posterior into the policy. By iterating these two steps, DAV progressively refines the diffusion model toward a multi-modal aligned distribution.
Figure 1:
DIFFUSION ALIGNMENT AS VARIATIONAL EM
In this section, we introduce a detailed mechanism of Diffusion Alignment as Variational Expeetation-Maxiniaaioon (DAV). As illustrated in Figure 1, DAV aligns diffusion models by iteratively alternating between an E-step and an M-step. The E-step involves a test-time search guided by a soft Q-function to effectively discover high-reward, multi-modal trajectories from the variational posterior distribution. Subsequently, distilling the information from searched trajectories through the E-step, progressively learning to generate outputs that are aligned with the reward function. The training procedure our method is summarized in the pseudo code provided in Appendix A.
the M-step, the diffusion model is updated by
4.1 VARIATIONAL
FORMULATION
Inspired by Levine (2018), we cast diffusion alignment as maximizing the likelihood of a binary optimality variable O, defined (O = 1/T) exp(E> (2, This directly implies that the alignment objective is to maximize the marginal likelihood of optimality as follows:
lax pe (O = 1).
0
Importantly, 9 does not directly parameterize this marginal likelihood, but instead induces a distri- bution over trajectories by (T( (T(T)) Iln=1 Pa (++-1 In other words, since 0 specifies the diffusion reverse process, alignment can be formulated by introducing latent trajectory variable that bridges between the model parameters and the optimality variable. The reverse process yields trajectory т, which contains zu that is subsequently evaluated by the reward function. Hence, acts as unobserved latent variable connecting 0 to the observed optimality outcome O. The resulting incomplete log- aggliikel expressed as Po 1) log (o O 1)dr.
Directly maximizing the incomplete log-likelihood is intractable due to the hierarchical struc- ture of denoising trajectories. Instead, we introduce a variational distribution 77(7) =
Pr (xT) I[==1 v)(x,-1 2t). to approximate the intractable posterior Po Po(1 = 1) and convert the marginal likelihood optimization into tractable variational inference problem as follows (See Appendix C for details):
T Σ
(((1) + log
pe po xz))
() xx )
log (O = 1) Erany
= Ja(n). Po).
(6)
The high stochasticity of the diffusion reverse process makes early denoising steps less impactful on the final outcome (Ho al., 2020), necessitating discounting mechanism to attenuate credit assignment of z, with large timestep I.
Introducing discount factor.
Proposition (Lower bound on likelihood of O with discount factor 7). Let n E (0, be the discount factor. The likelihood of the optimality variable O admits the following lower bound:
T
7 T-1 (r(Ee, 2:-1) + +log Po(x,-1 Iza)
Ja, Bray(T)
(7)
n7 xz)
Proof. See Appendix D.
4

---

Published as a conference paper at ICLR 2026
Optimizing the ELBO, Ja.y (1,,((1,1 (n. Po). is often approached with an EM-based RL algorithm, alterntt- ing between the E-step (posterior inference) and the M M-step (model update) (Dayan & Hinton, 1997). However, prior EM-based RL approaches exhibit critical weakness in the E-step. They approxi- mate the posterior by reweighting on-policy samples or past experiences from a replay buffer (Peters & Schaal, 2007; Abdolmaleki al., 2018; Nair et al., 2020). However, if the behavioral policy devi- ates significantly from the posterior distribution pe( = 1), this approach critically misspecifies the posterior, guiding the M-step toward biased and suboptimal distribution.
Therefore, we redesign the EM-based RL for diffusion alignment. Let 0 be the model parameter the k-th iteration. In the E-step, we first determine the posterior distribution that maximizes the ELBO, i.e., (T) = arg max, Ja.y (1) Po+ ). Then, we employ test-time search (Singhal et al., 2025; Kim et al., 2025c) to obtain approximate samples that follow (T). In the M-step, we max- imize the ELBO by updating the model parameter, 0k+1 arg m&X, Ja.n (.p.Pa), which corre- sponds minimizing the forward KL divergence using the samples from the Mp.-
4.2 E-STEP: TEST-TIME SEARCH FOR POSTERIOR INFERENCE
In the E-step, we sample trajectories from the variational posterior distribution 7i by employing test- time search. We first determine n%. (T) arg max, Ja,n Po+). Since our objective is equivalent to the KL-regularized RL objective in Equation (1), the optimal variational distribution, (T),), is
the production of the soft optimal policy that takes the form of reward-tilted distribution:
m (Z)-1(z)) x Po* (It-1 (x)) exp(Q son, ou (Ze, 21-1)/a).
(8)
The derivation is detailed in Section E. However, directly sampling from mi is intractable. To overcome this, we approximate using a two-stage local search to generate the subsequent state, It-1. The first stage of this search involves sampling set of candidate particles. Specifically, draw M intermediate particles, {-/m } from a proposal distribution, fj.. If the gradient signal of the reward is available, we can construct an effective proposal distribution by using gradient guidance (Grathwohl et al., 2021; Kim et al., 2025a). Subsequently, we refine the intermediate particles via importance sampling, which effectively pushes the samples closer nh (Li et al., 2024b). We detail our search procedure in Section E.2.
Note that the test-time search in DAV is modular component. This design allows for any algorithm capable of approximating the target posterior distribution to be substituted into the E-step. There- fore, DAV is not tied a specific search technique and can directly benefit from future advancements in test-time search methods (Jain et al., 2025; Zhang et al., 2025; Yoon et al., 2025).
4.3 M-STEP: AMORTIZING TEST-TIME SEARCH INTO DIFFUSION MODELS
In the M-step, we update by distilling trajectories from the E-step. This corresponds to maximizing ELBO by projecting vi onto the diffusion pa via forward KL minimization on searched trajectories:
g++1 arg max Ja,, (T), 0) = arg min D (n (IL))))))
Minimizing this forward KI. divergence equivalent to maximizing the log- log-likelihood of the trajec- tories from % Since the policy update is performed via gradient ascent rather than exact max- imization, it constitutes a partial M-step in a Generalized EM framework (Dempster et al., 1977). Nevertheless, as long as the gradient ascent increases the ELBO, the monotonic improvement prop- erty of EM is preserved. The training objective for DAV thus:
LDAV( = E, [- log Po())] = Er-nn
Σ log po (2z-1
(10)
To prevent the capability loss of the pretrained model, we introduce DAV-KL, which adds a KL- divergence term to penalize deviation from the initial pretrained policy P&n:
T
DDAV (L) = E
Σ
l - log po (x (xx) + (po.a(oo(==..... Iox Ilpo ( (
(11)
where the coefficient À controls the trade-off between aligning with the expert policy of and pre- serving the knowledge of the pretrained model.
5

---

Published as a conference paper at ICLR 2026
(a) Aesthetic vs. ImageReward (b) Aesthetic vs. CLIP Score
(c) Aesthetic vs. LPIPS-A
(d) Acsthetic vs. LPIPS-P
0.65
0.5
0.24
0.60-
0,55
Imagettewar
CLJP Soure
0.0
0.22
0.55
LPIPS-
0.50
-0.5
0.20
0.50
0.18
-2.0
0.16
0.40
0.35
Aesthetic Score
Aesthetic Score
Aesthetic Score
Aesthetic Score
DDPO
TDPO
DRaFT
DAV
DAV-KI.
Figure 2: Training dynamics of our methods and baselines, with performance marked every 10 epochs. All methods were trained for 100 epochs, except for DDPO, which was trained for 500 epochs. Our approaches successfully preserve alignment score and diversity compared to baselines.
Method
Aesthetic (t)
LPIPS-A (f)
ImageReward (t)
Pretrained
5.40 (0.01)
(0.01)
0.90 (0.01)
DDPO
6.83 (0.16)
0.48 (0.05)
0.27 (1.01)
TDPO
6.78 (0.28)
0.39 (0.10)
0.51 (0.47)
DRaFT
7.22 (0.22)
0.46 (0.04)
0.19 (0.64)
DAV
8.04 (0.07)
0.53 (0.03)
0.95 (0.21)
DAV-KL
6.99 (0.04)
0.58 (0.01)
(0.04)
DAS
7.22 (0.01)
0.65 (0.01)
1.07 (0.03)
DAV Posterior
9.18 (0.07)
0.53 (0.01)
0.91 (0.21)
DAV-KL Posterior
8.66 (0.09)
0.58 (0.01)
1.14 (0.08)
Table 1: Comparison on text-to-image synthesis benchmarks. All results are reported as mean with standard deviation in parentheses. Our methods and TDPO are shown at the 100th epoch, while DDPO and DRaFT are taken from the last checkpoint before over-optimization. Methods above the midline are fine-tuning approaches, and those below are test-time search methods.
5 EXPERIMENTS
This section empirically evaluates the ability of our framework to optimize rewards while preserving sample diversity and naturalness continuous and discrete diffusion. We demonstrate the versatility of our framework across two distinct data modalities: text-to-image synthesis using latent diffusion model (Rombach et al., 2022) and DNA sequence design via a discrete diffusion model (Sahoo al., 2024). We denote the amortized policies as DAV and DAV-KL. and their posterior samples, obtained through test-time inference as explained in Section 4.2, as DAV Posterior and DAV-KL Posterior, respectively. Implementation details and hyperparameter settings for all experiments are provided in Appendix H. To reflect the versatility of DAV, our baselines mainly consist methods that are agnostic to data modality and reward function differentiabilityy We also compare approach with a representative rest-time search method, as it is the core mechanism of the E-step.
5.1 CONTINUOUS DIFFUSION: TEXT-TO-IMAGE SYNTHESIS
We use Stable Diffusion v1.5 (Rombach et al., 2022) as our base pretrained model across all text- to-image experiments. We employ set of 40 simple animal prompts for fine-tuning provided by Kim et al. (2025c). All results are averaged over three random seeds.
5.1.1 EXPERIMENTAL SETUP
Metrics. We use the differentiable LAION aesthetic score (Schuhmann, 2022) as our primary re- ward. We evaluate the results for two key failure modes: reward over-optimization and diversity col- lapse. To detect over-optimization we measure prompt alignment using CLIPScore (Radford al., 2021) for semantic consistency and ImageReward (Xu al., 2023) for human preference. Sample

---

Published as a conference paper at ICLR 2026
DAV
8
DAV-KL
(R 7.21)
DAV-KL Posterior
(R - 8.76)
DAS (R - 7.16)
DRaFT-1
- 7.09)
DDPO
(R 6.79)
Pretrained
(R 5.71)
Qualitative comparison of our methods with DAS, DRaFT-1, DDPO, and pretrained model. Results for our methods are reported after 100 epochs fine-tuning. For DDPO and DRaFT, sample images from the last checkpoint before significant collapse. All images in the figure collected from multiple runs, and the score reported under each method is the average aesthetic score of the seven sampled images.
Figure 3:
diversity is quantified using LPIPS (Zhang et al., 2018), reporting the average distance across all samples (LPIPS-A) and within samples from the same prompt (LPIPS-P).
Baselines. We compare DAV against representative baselines include DDPO (Black et al., 2023) for RL-based fine-tuning. DRaFT (Clark al., 2023) for direct backpropagation, and DAS (Kim al., 2025c) for test-time search. We also include TDPO (Zhang et al., 2024b), a gradient-free RL-based method specifically designed to mitigate reward over-optimization.
5.1.2 RESULTS
Figure shows the training dynamics for each method, demonstrating how alignment and diversity metrics evolve as the aesthetic reward is optimized. The results in Figure 2-(a) and Figure 2-(b) indicate that DAV and DAV-KI. maintain a high alignment score while the baselines exhibit a sharp degradation in alignment scores. Similarly, Figure 2-(c) and Figure 2-(d) demonstrate that DAV and DAV-KL are substantially better at preserving sample diversity throughout the fine-tuning process.
Table provides quantitative comparison of our methods against the baselines. The first key observation a comparison between our methods and the fine-tuning baselines. DAV achieves a significantly higher reward (8.04) than both DDPO (6.83) and DRaFT (7.22), while maintaining high ImageReward score (0.95) comparable to the pretrained model. In DAV-KL, KL-regularization induces trade-off: it enhances diversity and ImageReward while incurring reduction in reward.

---

Published as a conference paper at ICLR 2026
(a) BLBO
Aesthetic Score
6.5
6.0
5
10
15
20
25
30
5
10
13
20
25
30
Epoch
DAV
Search and distill
Reweight
Figure 4: Comparison of ELBO and aesthetic score trends for DAV and its ablated baselines.
As illustrated Figure 3, both DAV and DAV-KL generate well-aligned samples without producing the repetitive backgrounds seen in the outputs of DDPO and DRaFT-1.
The second key observation from Table is the comparison among test-time search methods. DAV Posterior achieves the highest aesthetic score (9.18), substantially outperforming DAS (7.22), while DAV-KL Posterior obtains the best ImageReward (1.14) with competitive aesthetic performance (8.66). Although our methods exhibit slight decrease in diversity compared to DAS, our methods deliver clear gains in reward and alignment quality.
Effect of E-step Variants in DAV. In Search and distill, the model is updated using samples from "6 instead of n skipping the update of the posterior distribution. In Reweight, we replace test-time search with trajectory reweighting by exponentiated reward, following Abdolmaleki et al. (2018). By ablating key components of the E-step, we evaluate the effectiveness of test-time search their ability to optimize the ELBO.
Figure illustrates the ELBO and the corresponding aesthetic score trends for each variation. We empirically validated that DAV consistently improves ELBO, although cannot guarantee mono- tonic improvement due to approximation errors in the E-step. The ablated baselines not only fail to optimize ELBO, as expected, but also fail increase the aesthetic score. These results suggest the importance of test-time computation in the E-step for our variational EM framework.
Additional analysis. We conduct sensitivity analyses of the hyperparameters of DAV in Appendix Lower a increases reward but risks over-optimization, while higher values improve diversity. Set- ting >" stabilizes optimization by limiting early-step credit assignment. We analyze the effect of the number of distillation steps in the M-step and the number of particles in importance sam- pling. We also evaluated DAV with non-differentiable objectives, such as the compressibility and incompressibilityy rewards from Black et al. (2023), and we report the results in Appendix J.
5.2 DISCRETE DIFFUSION: DNA SEQUENCE DESIGN
We pretrained the masked discrete diffusion model (Sahoo et al., 2024) on the large-scale DNA enhancer dataset from Gosai et al. (2023). The dataset consists of 700k DNA sequences (200-bp length) and their corresponding enhancer activity in human cell lines, as measured by massively parallel reporter assays (MPRAs). All subsequent results averaged over six random seeds.
5.2.1 EXPERIMENTAL SETUP
Metrics. Our metric setup follows the methodology of Wang al. (2025). As the target reward, we use a trained Enformer network (Avsec et al., 2021) to predict enhancer activity (Pred-Actie Pred-Activity (t)). To avoid data leakage, we train two distinct Enformer models on disjoint splits of the enhancer dataset (Gosai et al., 2023): one provides rewards during fine-tuning, and the other serves solely as a held-out evaluator. We further assess the generated sequences on three criteria: diversity, nat- uralness, and biological validity. We quantify diversity using the average Levenshtein distance (f) (Haldar & Mukhopadhyay, 2011) between generated sequences. We assess naturalness using the 3- mer Pearson Correlation (3-mer Corr (?)) against the top 0.1% most active enhancers in the dataset. Finally, to measure biological validity and detect over- optimizaiii,,nn we use an independently trained classifier for Chromatin Accessibility (ATAC-Ace (f)) (Consortium al., 2012; Lal al., 2024).
8

---

Published as a conference paper at ICLR 2026
(a) Reward vs, Diversity
(b) Reward vs. Naturalness
(c) Reward vs, Validity
0.40
Levenstein Diversity
80-
d-mer Correlation
0.35
0.8
ATAC-Ace
70
0.6
60-
0.4
50-
0.2-
40
0.20
0.0
8 Prod-Activity
Pred-Activity
Prod-Activity
* DRAKES VIDD
DDPO
DAV
DAV Posterior
Figure 5: Performance comparison of DAV and baseline models. The x-axis represents the reward (Pred-Activity), while the y-axis shows (a) Diversity (Levenshtein Diversity), (b) Naturalness (3- mer Corr), and (c) Validity (ATAC-Acc).
Baselines. We compare DAV against representative baselines of discrete diffusion model alignment. For direct backpropagation on discrete models, our baseline is DRAKES (Wang et al., 2025). For RL-based fine-tuning, we compare against DDPO (Black et al., 2023) and VIDD (Su al., 2025).
5.2.2 RESULTS
As shown in Figure 5, methods outperform baselines in generating enhancer DNA sequences. In Figure 5-(a). DAV achieves the best trade-off between reward and diversity. Figure 5-(b) shows that methods achieve the best performance in both reward and naturalness. Figure 5-(c) further confirms that our methods effectively balance reward and validity. Moreover, DAV Posterior further enhances reward optimization while maintaining diversity and naturalness. These results showcase the versatility of DAV in the discrete diffusion alignment, where it effectively optimizes for reward while maintaining high sample diversity. The entire table of results presented in Table 2.
DISCUSSION
Conclusion. We introduced DAV, novel diffusion alignment framework based the variational Expectation- Expeetation-Maxiiizatiio Maximization algorithm. The E-step performs test-time search guided by a soft Q- function discover high-reward, diverse, and natural samples; the M-step amortizes the result of test-time search by forward-KL. distillation, fine-tuning diffusion models to the high-reward dis- tribution while preserving the diversity and naturalness. modalities: continuous diffusion for image synthesis and discrete diffusion for DNA sequence de- sign. Our results show that DAV effectively fine-tunes diffusion models to optimize rewards while mitigating over-optimization and diversity collapse in both domains.
validated DAV on two distinct data
Limitations and future works. Our work presents two primary avenues for future research. The first addresses the main limitation of our current framework: the computational overhead of the test-time search in the E-step. Fortunately, the E-step is modular, allowing for the direct integration of more efficient search algorithms. We anticipate that leveraging recent and future advances in test-time search (Zhang et al., 2025; Li et al., 2025) will substantially mitigate this bottleneck. The second limitation is the approximation error in the soft Q-function, which may guide the test-time search toward suboptimal distributions. This error primarily stems from the inaccuracy of Tweedie's formula for approximating the posterior mean at Ze with high noise levels (Chung et al., 2023). Future work could address this by employing distillation techniques (Salimans & Ho, 2022: Song et al., 2023). Distillation techniques substantially reduce the number denoising steps required for accurate æp prediction, expected enhance posterior mean approximation at large timesteps and yield more reliable Q-function approximations.

---

Published as a conference paper at ICLR 2026
AKNOWLEDGEMENT
We thank the anonymous reviewers for their insightful comments and suggestions, which signif- icantly improve our manuseript. This work is supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024- RS-2024-00410082)) and NRF grant funded by the Korea (oVE))men((MSIT) (No. RS-2025-00563763),
THE USE OF LARGE LANGUAGE MODELS
Large Language Models were employed exclusively for two auxiliary tasks: (1) minor polishing of the manuscript text for improving grammar and readability, and (2) limited assistance in code implementation for debugging syntax or refactoring functions. Importantly, LLMs did not contribute to the conception of the research problem, the development of the methodology, or the design and execution of experiments. All critical ideas, methods, and analyses presented in this paper are the original work of the authors.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learn- ing Representations, 2018.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in neural information processing systems,
2021.
Ziga Avsec, Vikram Agarwal, Daniel Visentin, Joseph Ledsam, Agnieszka Grabska-Barwinn Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 (10):1166-1203, 2021.
Arpit Bansal, Hong-Min Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 843-852, 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Rep- resentations, 2023.
Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A Rupam Mahmood, and Martha White. Greedification operators for policy optimization: Investigating forward and reverse kl diver- gences. Journal of Machine Learning Research, 23(253):1-79, 2022.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusson policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, pp. 02783669241223668, 2023.
Wenda Chu, Zihui Wu, Yifan Chen, Yang Song, and Yisong Yue. Split gibbs discrete diffusion posterior sampling. arXiv preprint arXiv:2503.01 161, 2025.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. The Eleventh Interna- tional Conference on Learning Representations, 2023.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representationss, 2023.
ENCODE Project Consortium et al. An integrated encyclopedia of dna elements in the human genome. Nature, 489(7414)557, 2012.
10

---

Published as a conference paper at ICLR 2026
Peter Dayan and Geoffrey E Hinton. Using -axiiiiionmmmmmiiiiiiiioonoo for reinforcement learning. Neural Computation, 9(2):271-278, 1997.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):
1-22, 1977.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 2021.
Carles Domingo-Entn Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. In The Thirteenth International Conference on Learning Representations, 2025.
Bradley Efron. Tweedie's formula and selection bias. Journal of the American Statistical Associa- fion, 106(496):1602-166) 2011.
Ying Fan and Kangwook Lee. Optimizing ddpm sampling with shorteut fine-tuning. In International Conference on Machine Learning, pp.
PMLR, 2023.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858-79885 778855, 2023.
Sager Gosai, Rodrigo Castro, Natalia Fuentes, John C Butts, Susan Kales, Ramil R Noche, Kousuke Mouri, Pardis C Sabeti, Steven K Reilly, and Ryan Tewhey. Machine-guided design of synthetic cell type-specific cis-regulatory elements. bioRxiv, 2023.
Will Grathwohl, Kevin Swersky, Milad Hashemi, David Duvenaud, and Chris Maddison. Oops i took a gradient: Scalable sampling for discrete distributions. In International Conference on Machine Learning, pp. 3831-3844 PMLR, 2021.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08998, 2023.
Yingqing Guo, Hui Yuan, Yukang Yang, Minshuo Chen, and Mengdi Wang. Gradient guidance for diffusion models: An optimization perspective. Advances in Neural Information Processing Systems, 37:90736-90970 2024.
Rishin Haldar and Debajyoti Mukhopadhyay. Levenshtein distance technique in dictionary lookup methods: An improved approach. arXiv preprint arXiv:1101.125:: 2011.
Jonathan Ho and Tim Salimans. Classifier-fre free diffusion guidance. In NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-663 6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in newral information processing systems, 35:8633- 8646, 2022.
Edward J Hu. yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Vineet Jain, Kusha Sareen, Mohammad Pedramfar, and Siamak Ravanbakhsh. Diffusion tree sam- pling: Scalable inference-time alignment of diffusion models. In Second Workshop Test-Time Adaptation: Putting Updates to the Test! at ICML 2025, 2025.
Michael Jordan, Zoubin Ghahramani. Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-25) 1999.
11

---

Published as a conference paper at ICLR 2026
Jaihoon Kim, Tachoon Yoon, Jisung Hwang, and Minhyuk Sung. Inference- time scaling for flow models via stochastic generation and rollover budget forcing. CoRR, 2025a.
Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of score- conditioned generator for offline design of biological sequences. Advances Neural Information Processing Systems, 36:67643-67661 2023.
Minsu Kim, Jiayao Gu, Ye Yuan, Taeyoung Yun, Zixuan Liu, Yoshua Bengio, and Can Chen. Offline model-bard based optimization: Comprehensive review. CoRR, 2025b.
Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment of diffusion models without reward over-optimization. In The Thirteenth International Conference on Learning Representa- lions, 2025c.
Tomasz Korbak, Ethan Perez, and Christopher L Buckley, RI with penalties is better viewed as bayesian inference. In EMNLP (Findings), 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. Designing realistic regu- latory dna with autoregressive language models. Genome Research, 34(9):1411-144 2024.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv: 0e0.11000000000 00909, 2018
Shufan Li. Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Align- ing diffusion models by optimizing human utility. Advances in Neural Information Processing Systems, 37:24897-249222 24925, 2024a.
Xiner Li, Yulai Zhao, Chenyu Wang, Gabriele Scalia, Gökcen Eraslan, Surag Nair, Tommaso Bian- calani, Aviv Regev, Sergey Levine, and Masatoshi Uchara. Derivative-free guidance in continuous and discrete diffuusion models with soft value-based decoding. CoRR, 2024b.
Xiner Li, Masatoshi Uchara, Xingyu Su, Gabriele Scalia, Tommaso Biancalani, Aviv Regev, Sergey Levine, and Shuiwang Ji. Dynamic search for inference-time alignment in diffusion models. arXiv preprint arXiv:2533.0200 2025.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint Stt.:25005004700 2025.
Zhen Liu, Tim Z Xiao, Weiyang Liu, Yoshua Bengio, and Dinghuai Zhang. Efficient diversity-preserving diffusion alignment via gradient-informa gflownets. arXiv preprint arXiv:2412.07775. 2024.
lyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Nanye Ma. Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xubui Jia, et al. Inference-time time scaling for diffusion models beyond scaling denoising steps. arXiv preprint mxXiv:2501.0933 2025.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09359 2020.
Radford M Neal and Geoffrey E Hinton. A view of the algorithm that justifies incremental, sparse, and other variants. In Learning in graphical models, pp. 355-368. Springer, 1998.
Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, and Jennifer Listgarten. Unlocking guidance for discrete state-space diffusion and flow models. In The Thirteenth International Conference on Learning Representations, 2025.
12

---

Published as a conference paper at ICLR 2026
Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning. pp. 745- 750, 2007.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 01153728--37444 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. Advances Neural Information Processing Systems, 37:130136-1301 2024.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
Chrisoph Schuhmann. Laion aesthetics. URL htlps://laion. aihlooglaiion-aesthiiiics 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprini 2.Xiv:2402.033000 2024.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittweeser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844889 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.0188 2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang. Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017b.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Penning- ton, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models. Transactions Machine Learn ing Research, 2024. ISSN 2835-8856. Expert Certification.
Raghay Singhal, Zachary Horvitz, Ryan Teehan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. In Forty-second International Conference on Machine Learning. 2025.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9447 2022.
13

---

Published as a conference paper at ICLR 2026
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, pmlr, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- fional Conference on Learning Representations, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Iya Sutskever. Consistency models. In International Conference on Machine Learning, pp- 3221 -32222. PMLR, 2023.
Xingyu Su, Xiner Li, Masatoshi Uchara, Sunwoo Kim, Yulai Zhao, Gabriele Scalia, Ehsan Haji- ramezanali, Tommaso Biancalani, Degui Zhi, and Shuiwang Ji. Iterative distillation for reward- guided fine-tuning of diffusion models in biomolecular design. arXiv preprint MrXii:2507.00445, 2025.
Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective mod- els for effective offline model-based optimization. In International Conference on Machine Learn- ing. pp. 10358-10368. PMLR, 2021.
Masatoshi Uehara, Yulai Zhao, Tommaso Biancalani, and Sergey Levine. Understanding reinforce- ment learning-based fine-tuning of diffusion models: A tutorial and review. CoRR, 2024a.
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous- time diffusion models as entropy-regularized control. CoRR, 2024b.
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, al. Amortizing intractable in- ference in diffusion models for vision, language, and control. Advances in neural information processing systems, 37:76080-76114 2024.
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision 2024.
and Pattern Recognition, pp.
Chenyu Wang, Masatoshi Uehara, Yichun He, Amy Wang, Avantika Lal, Tommi Jaakkola, Sergey Levine, Aviv Regev, Hanchen, and Tommaso Biancalani. Fine-tuning discrete diffusion models via reward optimization with applications to DNA and protein design. In The Thirteenth Interna- tional Conference on Learning Representations, 2025.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning- 2019.
arXiv preprint arXiv:1911.11I
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903-1515 15935, 2023.
Shuchen Xue, Chongjian Ge, Shilong Zhang, Yichen Li, and Zhi-Ming Ma. Advantage weighted matching: Aligning with pretraining in diffusion models. arXiv preprint arXiv:2509.2505/ 2025a.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint wxXx:22505.078/8 2025b.
Tachoon Yoon, Yunhong Min. Kycongmin Yeo, and Minhyuk Sung. e-sampler: Initial parti- cle sampling for sac-based inference-time reward alignment in score models. arXiv preprint arXiv:2506.00 01000,0.322.... 2025.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training -ree -nergy-guided conditional diffusion model. In Proceedings s of the IEEE/CVF International Con- ference Computer Vision, pp. 23174-23188 2023.
14

---

Published as a conference paper at ICLR 2026
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D Goodman. Star: Self-taught reasoner bootstrap- ping reasoning with reasoning. Proc. the 36th International Conference on Neural Information Processing Systems, volume 1 126, 2024.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*; Lim self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64777 2024a.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Xiangcheng Zhang, Haowei Lin, Haotian Ye, James Zou, Jianzhu Ma, Yitao Liang, and Yilun Du. Inference-time scaling of diffusion models through classical search. arXiv preprint arXiv:2505.23614 2025.
Ziyi Zhang. Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, and Dacheng Tao. Confronting reward overoptimization for diffusion models: perspective of inductive and primacy biases. In International Conference on Machine Learning, pp. 60396-60444 60416. PMLR, 2024b.
Huaisheng Zhu, Teng Xiao, and Vasant G Honavar. Dspo: Direct score preference optimization for diffusion model alignment, In The Thirteenth International Conference on Learning Representa- fions, 2025.
15

---

Published as a conference paper at ICLR 2026
A PSEUDO CODE OF DAV
Algorithm Diffusion Alignment as Variational EM (DAV)
Require: Pretrained diffusion parameters 0
1: Initialize model parameters <- 0
2: for k= ==..... N do
3: Initialize dataset of trajectories +-
4: E-step: Posterior Exploration via Test-Time Search
5: for = 1, do
6: Sample initial noise FT Po(xT) Initialize trajectory T + {
7:
8:
for 8 T, ..,, 1 do
9:
Sample M particles from gL (-(aa) with Equation (56) or Equation (59) Compute weights (wr Jm=1 with Equation (62)
11:
Resample - ~ Categorical [w" [w"))=)) mm=1)
Append - to trajectory т end for
13:
14:
Add completed trajectory T to
15:
end for
16:
17:
M-step: Amortization via Forward KL Projection
18: for E Dr do
Update by minimizing Equation (10) Equation (11)
20: end for
end for
22:
return Optimized parameters VNv
16

---

Published as a conference paper at ICLR 2026
B APPROXIMATION OF THE SOFT Q-FUNCTION
In KL-regularized RL for diffusion fine-tuning, prior works approximate the Soft Q-function for the undiscounted MDP case where 7 1 (Uchara al., 2024a; Li al., 2024b). By leveraging the recursive soft Bellman equation and the posterior mean approximation by Tweedie's formula (Efron, 2011: Chung et al., 2023), the soft Q-function can be approximated as:
Qusa (Tez 21-1i7 = 1) ~ r())(1+))
(12)
However, this approximation does not hold in the more general discounted setting where 7 * 1. We find that for a discounted MDP, the recursive form of the Bellman equation instead yields lower and upper bounds on the Soft Q-function. For t 2 and < 7 we derive the following inequalities:
(2' (200)())) ]
r((oo)
an log E Ex Ppeiee exp
Szon 12-1) S ory'-1 log
exp
For inducing upper inequalities, we start by substituting the definition of the soft Q-function into the soft value function, and we obtain the recursive expression of the soft value function:
r (xte xr))
T Vion (Xz-1))
Vi (It) log Ex, 1 (.1z,) exp(
+
(13)
a a
Based on the sparse reward MDP defined in Section 3.2, the immediate reward r(x,, 11-1) is zero for all steps t 2. This simplifies the recursion to
V (2x) = a log En, 1 P-Ppii (m))) exp( 1 V (z))) for t 2 2.
(14)
With the terminal condition V (3r)) 0, the soft Q-value at 1 is simply the immediate reward, Quen = r(æ1, zo) = r(zo). By substituting this into the definition of the soft value function, we can derive the value = 1 and subsequently the Q-function
as:
(on(a)) ) log E. Fe Po (-1e,) (exp(((za))/a))
(15)
r(Fn) )
Qwnn (12, 01) = ya log E.g ppan (m1)
exp
(16)
To simplify the notation, we define the auxilary variable (xx) = cxp(Vin (az)/a). From the definition of V the case at = is 3, (x1) Enor (exp (rao)//)// Then the recursion of soft Bellman equations for 2 2 can be written
(xx))) 1 1
(17)
Since the function 2 2 is concave for 0 < y 1, applying Jensen's inequality yields the fol- lowing relationship: (E|Z))² 2 E[Z"]. By iteratively applying this result to the recursive definition of (e (2:.) ), we establish lower bound as follows:
(x(x.)) Ez, Presa [3,-1 (0,(((x))) (x1=1)"1
(18)
Ex, 1 Ppone (EM, 2 Ppiie 13 2 (Ix=-
(19)
2 Eze 2u 2a-11- Pripp (1 2 (1x (1)
(00)
D Ex, au Paiia 1B, 3( [It 3))= )
(21)
(22)
2 Exg 1e [BB (111))))
(23)
(24)
By applying Jensen's inequality again, we obtain the lower bound of the (xx)):
E Empse- 1 Panie ((exp((((x)))) ((exp(((=o))/)))) (x
(25)
17

---

Published as a conference paper at ICLR 2026
In similar manner. the upper bound of (x(x)) for 2 2 is derived using B|Z Σ (B[z)))
((x+) = - [3 [t 11
(26)
I E P Pran B [1 1)])
(27)
(Ezr pma
(28)
(29)
(Bzn 1mppo (3r )))
(30)
(Bene-1PPas [E (ex)(((2))/))]]
(31)
By the law of total expectation, the nested expectations can be combined into single expectation over the trajectory T0:--1, which yields the final upper bound for (x,)):
(z())) (Ezon- (esp-//ammm///ssp///a///
(32)
Combining these results, the bounds on 3: for t 2 2 is summarized as:
Exg. PrPaii ((exp(((ao)//))))""""] (x(x)) S Emo: 1~Ppiia (exp(r(xo)/a)))"
(33)
exponentiated soft value function: (2) (a))/a) back into the inequality, we obtain the bounds on the
By substituting (x,) exp(V
- r(2o) }
Vinn (2ze)
r(20)
Ero. Pppix
exp
Sexp
EE
Ero.
exp
α
a
(34)
Taking the logarithm of all parts and multiplying by yields the bounds on the soft value function (
(d (2''1((a))
}
r (2o))
log B
exp
log SVin (xxe) S ar E
(35)
Using the relation Q (Ie, 1) VV for 2 3 from Equation (4), we can derive the corresponding bounds for the soft O-function:
(2(0)
]
r(xo)
an log E. Exo.t Preice
exp
Lai xt-1) S "01 log E
E
exp
CK
A remarkable simplification occurs when approximate the log-sum-exp terms in both the lower and upper bounds using Tweedie's formula (Efron, 2011; Chung et al., 2023). We find that both complex bounds approximated the same simple, closed-form expression: 71-1r(0n(2,
This result, which is also consistent with the exact solutions for the boundary cases at 1 and t 2, yields the following unified, first-order approximation for the soft Q-function:
(2, 2t-1) (--1((((((111)
(36)
18

---

Published as a conference paper at ICLR 2026
C DERIVATION OF EVIDENCE LOWER BOUND
We derive the ELBO of the marginal log-likelihood of the optimality variable, log = 1). The derivation begins by introducing variational distribution 17(T) over the trajectories. By applying Jensen's inequality, we can derive the ELBO as follows:
log po (O 1) = log / po(T)p(O 117) dr
(37)
pO(T)p(( = 1/7) dt
log
(((T)
(38)
1T(T)
> f n(())
logp(0 = 1|T) + log (T(T)
dr
(39)
11(T)
((12 log (((3)-1122)) 1/(x (z))
x E Eron
Ja(n po)
(40)
This final expression constitutes the ELBO, which serves tractable surrogate objective for alignment.
19

---

Published as a conference paper at ICLR 2026
D PROOF OF PROPOSITION 1
Proposition (Lower bound on likelihood of O with discount factor 7) Let ? E (0, be the discount factor The likelihood of the optimality variable O admits the following lower bound:
(r(0t; F2-1) + +log po xz)
(n,P0) - Byones)
Σ T-
(7)
a
n x0))
Proof. For clarity, we stick to MDP notations in Section 3.2. First, as suggested in Levine (2018), we introduce an absorbing state Saborc to take into account the discount factor and let S =
S U {Sahsorh ). Each action a make transition to s' S following the original dynamics P
with probability yy and to Saborb with probability 7, i.e., the modified transition probability P becomes:
P(s' 8,
E E
(
(41)
1 1-
if Sabsorb
and (S Sabsorb Q) = 1.
Similarly, the modified reward becomes:
F(s, a) (r(s a) if E S
(42)
if Sabsceb-
Also, po and 7j define distributions over trajectories the modified MDP, with pa(-I 7(- (absorb))) deancs (-) and Pe s) pe(-ss), 7(1ss) n7(ss) if E S.
=
We can derive the ELBO for the modified MDP in the same way as Equation (40), which gives:
F(8,, az) + log Ro(ag19e) ) na(els))
log (O = 1) Tw P
Σ
=: (7.
(43)
t=
By linearity of expectation, we can swap the sum and the expectation:
F(st,ar) at)
po(a (s))
Ja (7) pa)
E (e
llo
(44)
a
17( 8t)
Consider the event E,, meaning the agent has not yet reached the absorbing state up to time i.e., Vt
8, E S, E [0, 1,..., ,, t] and Pr(Ee) = 7. Recall the law of total expectation:
E (. [+] Pr(Ee) 18 (Bs., 1. [. + (1 - Pr(E))) (. H-
If - then 8t Saburb, F(se, at) O, and log Re(ax (s.) 0. On the other hand, when conditioned ((),)))
on Es, 7(-[s;) = n((1se), (-(- = p-(- F(5t, at) = r(s,, and E(en Ec] =
( [.]. Now, we can rewrite Ja.y in the original MDP:
F(s
+ +log po(ar(s)) n((e(s)) st)
Ja,((1110)
Pr(Et) - Be
E.
(45)
a
a() + log n(aelss)
(a(()()))
Σ
E E
(46)
t=
T-1
(t)
+ log pa (a(a((8))
Σ
= E
(47)
n(ar(s))
By rewriting the last line using notations with 24 's:
(1(21, log po(
Ja Bray
Σ 7
T-t
(48)
a
17 (1t)
20

---

Published as a conference paper at ICLR 2026
E DETAILS OF TEST-TIME SEARCH FOR THE E-STEP
This section details the test-time search procedure for DAV. the E-step, we are required to sample trajectories from the optimal posterior distribution, 17" (T). We first simplify the problem by defining an optimal policy, 17" (Zt). and decompose the optimal posterior distribution as (r) = Vhr (ZT) I1 77 (xt-11 We then draw samples from this policy using a two-stage pro- cess: (I) Constructing proposal distribution f)(2;=1 using gradient guidance from the reward function (Grathwohl et al., 2021; Dhariwal & Nichol, 2021), and (2) Employing importance sam-
pling to refine these samples, correcting for mismatches with the true posterior.
E.1 DERIVING SOFT OPTIMAL POLICY FOR EXPLORATION
For the fixed policy parameters of diffusion model at the k-th iteration, Pos, the optimal posterior policy n ( for < t < T is defined as the policy that maximizes the regularized objective Ja,(11 Pg=). Because our objective Ja.; Par) ) is equivalent to the KL-regularized RL objective in Equation (1), (x(-1(1zz) is the soft optimal policy. Let Qun,oo denote the soft Q-function for the objective regularized with respect to the prior policy Par. Then, the soft optimal policy is found by solving the following maximization problem:
Yoot (It, xt- po (I (tp)
a log 7 121))
(1(--1(11)) = arg max Ex (m)(aa) vi xx)
(49)
por (14-1 exp soft, (Ft, Tie 1))
arg max Exg mm)()
log
(50)
7) (z:))
1 7 (It (Fr-1 exp onn (Ze, (t11/a)))/a) log Z
arg min DRL DKL (n(2(-1 n(12))
(51)
z por (x)) exp(Q
(xx=1 so (It, (11,2))1)/o))
(52)
where = Por (2+-1 1 (arz) exp(Qr sont / To summarize, the soft optimal policy - (2xx) takes the form of a Boltzmann distribution, where the prior policy is re-weighted by the exponentiated soft Q-function to favor actions with higher expected soft returns.
E.2 GRADIENT- GUIDED TEST-TIME SEARCH WITH IMPORTANCE SAMPLING CORRECTION
We now detail the two-stage procedure for sampling from the soft optimal policy, 7 (=t-1 (x:), which is designed to generate high-reward samples while preserving the diversity of the prior model:
Proposal construction: If the reward function differentiable, construct proposal distri- bution îi for 1i using a first-order Taylor expansion.
Importance sampling correction: We employ importance sampling to correct for the distributional mismatch between our proposal distribution fly and the optimal policy 7p.
E.2.1 CONSTRUCTING PROPOSAL DISTRIBUTION VIA TAYLOR EXPANSION
For complex reward functions where the target posterior deviates significantly from the current pol- icy, leveraging the gradient of a differentiable reward function can dramatically improve the effi ciency of the E-step. This gradient-gun -uided exploration provides more direct path to discovering high-reward samples.
Continuous diffusion. In the case of continuous diffusion models, we incorporate the reward gra- dient into the reverse process by leveraging a first-order Taylor expansion, as previously suggested by Dhariwal & Nichol (2021); Kim et al. (2025c). We derive our proposal distribution from the
21

---

Published as a conference paper at ICLR 2026
optimal soft policy vi by first approximating the soft Q-function with the reward function:
1
(I
Z por (Z exp
(1
(53)
1
7 Pot (21 (22) exp
r(āo(xc-1))
(54)
where (x)-11 Byg [200 and po (10- (2 paa (2t, 1), o) To maintain tractable Gaussian form, we approximate r(fo(E(-1)) with first-order Taylor expansion around the mean of the prior distribution, pagi:
r(āo(x=-1)) ( r(to(zz)) + V V2,, (())(z)))) (x Mig (xx,
(55)
Since r()) does not depend on 2t-1, it is absorbed into Z, leaving only the linear term in the exponent. This process effectively incorporates the linearized reward gradient as a guidance term, shifting the mean of the prior distribution (Dhariwal Nichol, 2021). We therefore define our proposal policy nn this modified Gaussian:
71--1
(x))))) == Z Por (x x4) exp
r(60(21-1)))
(56)
o + L - Vo o:1
=N Tt-1i Mo (xr,
(57)
Discrete diffusion. Drawing inspiration from Grathwohl et al. (2021) and Nisonoff et al. (2025), extend gradient-guided E-step discrete diffusion model by incorporating Taylor expansion- based gradient exploitation a differentiable soft optimal Q-function. Similar to the derivation process of proposal distribution of the continuous diffusion, our derivation begins with the approxi- mated optimal policy, where the soft Q-function is approximated as posterior mean approximation:
(7'-1 (2-- r((0(((-1))
M ( xx)) = z pox Por (IC-1 xz) exp
(58)
To create tractable gradient signal, we apply first-order Taylor expansion to the reward function ((fo(x;=1)) around the current state estimate dt (zz)). After substituting the expansion, we can sim- plify the expression. Terms that are constant with respect Iz-1 1 are absorbed into the normalization constant Z. This isolates the influence of the gradient signal as a linear term that directly modifies the log- probabbbbillssiss of the prior distribution Pok.
n (3+-1 wx) Po* (x4-1 1x)eexp exp
[r((o(z))) ((o((((())) xt)]
(59)
x Pot ( 24) exp
Vz, ((o
(60)
a
The final expression defines new categorical distribution whose log-probabilities are those of the prior policy, shifted by a linear guidance term. Finally, the proposal distribution is Categorical with logit-additive guidance:
ny (x1-1 x1) II
Cat Cat softmax (zt)
(61)
2-1
where the logits vector z, K for position has components:
(ze); log Fot (04-1,e = r' [Va,, (fo (ax))];
a
and e (1,... L.): dimension, E (1,.... K}: category index.
22

---

Published as a conference paper at ICLR 2026
E.2.2 IMPORTANCE SAMPLING CORRECTION
Since the proposal distribution fix is constructed via Taylor expansion of the reward function, the resulting samples may deviate from the true optimal distribution l. To correct for this mismatch, we employ importance sampling. Given M samples {=--))m=1 from the proposal distribution îjk- assign importance weight to each particle as:
77 (==""" (2+)
w
(62)
Pyk (ot-1 (2re)
expl Sion (ft, x(-1)) a
(63)
(x (1=="11z0 (2)))
(xy (1x)
exp( y-
(64)
1, (2+)
corrected sample from og is then obtained by resampling according the normalized weights:
W
Ft-1 Cat
(65)
m=
==
23

---

Published as a conference paper at ICLR 2026
F EXTENDED RELATED WORKS
F.1 SELF-TRAINING
The self-training paradigm, pioneered by the AlphaGo series (Silver et al., 2016; 2017a;b), alternates between expert rollout stage to generate high-quality data and distillation stage to fine-tune the model on that data via maximum likelihood estimation (MLE). This framework has been success- fully adapted to Large Language Models in methods like STaR (Zelikman et al., 2024), which boot- straps reasoning from self-generated rationales, and ReST (Gulcehre et al., 2023; Singh et al., 2024; Zhang et al., 2024a), which distills policy from filtered set of high-quality outputs. In the same vein, DAV translates the self- trafiiraining paradigm diffusion models. Within the DAV framework, the test-time search (E-step) acts the expert rollout to discover diverse, high-reward samples, which are then used for distillation (M-step) via MLE. To our knowledge, DAV is the first to apply the self-training framework to the problem of aligning diffusion models.
F.2 GRPO AND DPO BASED GENERATIVE MODEL ALIGNMENT
Recently, Group Relative Policy Optimization (GRPO) has emerged as strong method for aligning LLMs, often outperforming PPO-based approaches (Shao al., 2024). Following this trend, several GRPO-based alignment methods for diffusion fine-tuning have been introduced, achieving strong performance in reward optimization (Liu et al., 2025; Xue et al., 2025b;a). However, their primary objective is reward maximization rather than preventing over-optimization. DAV is designed to mitigate over-optimization, providing a balanced trade-off between reward optimization, diversity, and naturalness.
There is also a line of work aligning diffusion models using Direct Policy Optimization (DPO; Rafailov et al., 2023). While these methods achieve strong human preference win rates pre- trained models (Wallace et al., 2024; Li et al., 2024a; Zhu et al., 2025), they are fundamentally constrained by the quality of pre-collected preference datasets, which often limits their performance frontier (Xue et al., 2025b). Moreover, prior GRPO and DPO-based approaches focus primarily on visual generative tasks, whereas DAV provides unified framework that is empirically validated on both continuous and discrete diffusion models.
24

---

Published as a conference paper at ICLR 2026
G ADDITIONAL BACKGROUNDS
G.1 CONTINUOUS DIFFUSION
Diffusion models (Sohl-Dickstein et al., 2015; Ho al., 2020) class of hierarchical generative models that learn to approximate the data distribution. A denoising diffusion model generates sample zo by Markov generative process-often referred to as the reverse process-that starts from a standard Gaussian prior (FTr) = ().
T po(2)))) - (T((T) II po ( xt): pa ( x4) = M( pa( 1), of.).
The forward process defined by fixed Markov chain that gradually corrupts the data xy with Gaussian noise according variance schedule {B,)==1
T
q(11 m0) = II
g(<e 21-1),
((x, xt-1) = -M,; /1 I1-1, Be1).
t=1
A useful property of the forward process is that oa can be obtained directly in closed form without simulating the entire chain:
((x vo) = Mar vorzo, vo, (1 - I.))))),
where 1 Br, ār = IT Cts-
Training diffusion model is performed by optimizing the evidence lower boLnd(ELBBO) on the data log-lik log-likelhhood Egane [log po (Io)], which can be decomposed as follows:
log pe Po (20l/z11
Σ
p0 12)1) - DKL || pr(xT)). 1e
G.2 DISCRETE DIFFUSION
Our discrete diffusion model follows the framework defined in Masked Diffusion Language Models (MDLM) (Sahoo et al., 2024).
Notation. We begin by considering single discrete variable 2n {1, K} that belongs finite vocabulary of size K. We denote the mask token as [M]. which serves as an absorbing state our forward diffusion process. The diffusion process operates over T discrete time steps, with t E {1,2.... .... T}. We use Q E [0, 1] x to denote row-stochastic transition matrices where each row sums to 1. The noise schedule is parameterized by a, = = (t)) where ot)) [0, 1] + R+, and B 04 controls the corruption rate.
Forward diffusion process. The forward diffusion process gradually corrupts the clean data v0 through a markov chain over discrete time steps:
q(Eur xr) II
a (ax x4-1)
1=1
Each forward step follows categorical distribution:
q (x, xt-1) Cat (z.: Q:)
where Cry- the one-hot encoding of æ;-1- The choice of Q₂ determines the corruption process. Austin et al. (2021) deals with various options for Q such uniform, absorbing, and Gaussian; however, we are mainly considering an absorbing kernel as Sahoo et al. (2024):
B
j [M] and + [M]
1,
if j M
(Q0)s
Br,
if = + [M]
0.
otherwise
25

---

Published as a conference paper at ICLR 2026
Following prior works, log-linear schedule (t) - log(1 (1- t) gives a, and 33 meaning the corruption probability increases linearly with time. Additionally, marginal forward distribution can be computed as:
q (z: ro) Cat (2t;; Cazo Qu)
where Q = Q1Q² ... Q₂ is the cumulative transition matrix. This analytical tractability enables
sample-cfficient training as we can directly sample 2; from zo without simulating the entire forward chain. Let a₂ II 0₂ II (1 B.). Then:
if j = [M] and + [M]
1,
if j = M
@1)ss =
Ci.
if j = i + [
0.
otherwise
This means the marginal distribution simplifies to:
if Dt 2o * [M]
(<<e)2o)=
Cl
if 2 [M and zo # [M]
1,
To = [M]
0.
otherwise
In other words, each unmasked token Zo remains unchanged with probability ar. This simple binary choice makes sampling and likelihood computation extremely efficient.
Reverse diffusion process. Austin et al. (2021) introduces the Zo- prrammtteeriziioonn for the reverse process. Sahoo et al. (2024) further simplifies the process by introducing substitution- (SUBS) parameterization.
simplifies the reverse diffusion process by explicitly pre- venting the model from predicting the absorbing state ([M) when conditioned on an intermediate sample, and also by ensuring that once token is denoised, it is permanently fixed and cannot revert to a masked state in subsequent sampling steps. The reverse process in this paper is parameterized as:
(at((ss) Cat( (I-a.)m )m (o. (m)
æf * [M]
po(#s(11)) qlaaalte 20 =f( ()0)
x4 [M]
1-ac
where the (x (x)) for t denoises from time back to time s, and m denotes the one-hot encoding of the mask token [M].
Loss function. From the original loss in Ho al. (2020), Sahoo et al. (2024) derives a simplified training objective through Bao--Baakwellization of the discrete-ting time ELBO. We define a sequence of time step pairs (s(1), (s(T), {(T)) where 0 S s(i) t(i) < for all i, which partitions the diffusion trajectory into segments. The key result is:
T
EMDLM
Bg DKL (g (Zate) Ie()) mo) Il (z 2x()))
[ai(s) - as(1)) log (fio (Z1(0):0) m0)
Eg
1-0x() (4())
Multivariate discrete variables. Now we extend to multivariate discrete data, where we have sequence of discrete variables. We redefine our notation for this multivariate case x, (at .xf) where each of E (1, K). L is the sequence length, and each position é represents a discrete variable the sequence. The forward process assumes independence across positions:
g (It xz-1) =
II g (2) E E=1
26

---

Published as a conference paper at ICLR 2026
Each component follows the same absorbing state transition in the single- varialee case: q (af xf-1) = Cat (af; e Qr). The reverse process allows for dependencies across positions:
Pe (x, x1) = I po (x' 27-2), 1=1
sct.
Each position
depends
the entire sequence
context: Pg (=f xt
Cat (arf; softmax (Jo ())).
Thereby, the training objective extends the single-variable
SUBS formulation:
T
MDLM
Σ
B
at(i)) (x(1) (c((s)
(
= [M]]]
log pa (x k
where T is indicator function, and is essential to ensure that we don't compute the loss based on the unmasked tokens.
27

---

Published as a conference paper at ICLR 2026
H EXPERIMENTAL DETAILS
H.1 TEXT-TO-IMAGE DIFFUSION MODELS
Diffusion model. Based on Stable Diffusion v1.5 (Rombach al., 2022), we adopt 50-step DDPM sampling (Ho et al., 2020) with classifier- free guidance (Ho & Salimans, 2021), setting the guidance weight 5.0 for all experiments. For parameter-efficien fine-tuning, we apply LoRA (Hu et al., 2022) to the denoising UNet with rank of 4, consistently across all baselines.
Details of evaluation. To assess prompt- image alignment, we use CLIPScore (Radford et al., 2021) and Reward (Xu et al., 2023). To evaluate diversity, we employ two LPIPS-based met- rics (Zhang et al., 2018), 8), which quantify perceptual differences between images. LPIPS-A measures diversity across all generated images, irrespective of the prompt, while LPIPS-P measures diversity within each prompt by computing the mean LPIPS distance among images conditioned on the same prompt. For all evaluations, we sample 32 images per prompt.
Baselines. We reproduce results for DDPO (Black et al., 2023) and TDPO² (Zhang et al., 2024b) using their official codebases. As the source code for DRaFT is not publicly available, we use faithful implementation based on the AlignProp codebase" (Prabhudesai et al., 2023). We set the batch size to 64 for all fine-tuning methods. For DAS* (Kim et al., 2025c), we adjusted its sampling steps from 100 to 50 match other methods. Accordingly, we set its tempering parameter y to 0.014 to ensure (1 + +)) 1 for T = 50, following the hyperparameter setting guidelines of the authors.
Training details of baselines. Most baselines are trained for 100 epochs. However, due to its slower optimization, we trained DDPO for 500 epochs. For all methods, we report the best perfor- mance prior to collapse. In Table 1, the reported results correspond to epoch 400 for DDPO, epoch 40 for DRaFT, and epoch 100 for TDPO. In the case DAS, we directly report the performance obtained from its test-time inference without additional training.
Aesthetic score optimization. We employ the Adam W optimizer (Loshchilov & Hutter, 2019) with a learning rate of le-3, 31 0.9, and |32 0.999. We set the training batch size to 64. The core hyperparameters for DAV are set as follows: a = 0.005. = 0.9, and an importance KL-rregularization coefficient of = 0.01. In each iteration, the M-step performs a single update using the dataset collected from the corresponding E-step. We train the model for 100 epochs, which takes approximately 14 hours on eight RTX 3090 (24GB) GPUs. A sensitivity analysis for these hyperparameters is presented in Appendix I.
sampling particle size M 4. For the DAV-KI. variant,
Compressibility and incompressibility optimization. We use the same hyperparameter configu- ration as the aesthetic score optimization experiment, with the only exception being the importance sampling particle size, which we set to M = 16. Training for six epochs takes approximately hours on eight RTX 3090 (24GB) GPUs.
28

---

Published as a conference paper at ICLR 2026
H.2 DNA SEQUENCE DESIGN
Diffusion Model. We used the pretrained Masked Diffusion Language Model (MDLM) (Sahoo al., 2024), following the pretrained diffusion from the Li et al. (2024b). This model is trained on the Enhancer dataset (Gosai et al., 2023), which consists of DNA sequences of length 200.
Reward Oracle. Our reward modeling and evaluation setup follows that of DRAKES ( (Wang et al., 2025). To prevent data leakage, we use two separate reward oracles trained on distinct data splits from Lal et al. (2024). Both oracles are Enformer models (Avsec et al., 2021) initialized with pretrained weights; one serves the reward signal during fine-tuning, while the other is reserved for held-out evaluation. Further details on the evaluation protocol can be found in Wang et al. (2025), Appendix F.2.
Evaluation Metrics. We assess the generated sequences on three criteria: biological validity, nat- uralness, and diversity. We generate sequences for evaluation using batch size of 640. To measure biological validity and detect wver-optimization. we use an independently trained classifier to pre- dict Chromatin Accessibility (ATAC-Ace) (Lal et al., 2024; Wang et al., 2025; Su et al., 2025). We assess sequence naturalness using the 3-mer Pearson Correlation (3-mer Corr), which compares the k-mer frequencies of generated sequences to those of the top 0.1% most active enhancers in the dataset (Gosai et al., 2023). Finally, we quantify diversity using the average Levenshtein distance between generated sequences (Haldar & Mukhopadhyay, 2011; Kim et al., 2023).
Predicted activity optimization. We use the AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of le-3, 31 = 0.9, and 83 = 0.999. Hyperparameters for our method are set to a = 0.01, 7 I, and an importance sampling particle size of M 10. In the M-step, posterior mean approximation at early timesteps is inherently unreliable. Consequently, directly maximizing the likelihood in Equation (10) may compel the model to replicate these erroneous estimates, resulting in unstable optimization. To mitigate this issue, we omit the first 80 steps from the original 128 steps and perform maximum likelihood estimation only on the remaining steps. We train DAV for 200 epochs, which takes approximately 15 hours on single RTX 3090 (24GB) GPU.
Experiment Result Table 2 shows that DAV and its posterior variant, DAV Posterior, achieve superior balance between reward optimization and validity, and naturalness compared to baseline methods. While strong RL-based methods, such as DDPO and VIDD attain high target rewards, they suffer from significant drop in diversity and ATAC-ace, which indicates reward over-optimization. In contrast, our amortized DAV policy achieves a higher reward (7.71) than baselines while main- taining high diversity (87.91) and naturalness. Furthermore, the DAV Posterior achieves the highest scores in both the target reward (9.04) and validity (0.865), while still maintaining high diversity. This demonstrates the ability of our methods to generate high-reward sequences without sacrific- ing naturalness or diversity. For the test-time search baselines (Li et al., 2024b; Chu et al., 2025), although SGDD approaches DAV in Pred-activity and even achieves higher diversity, DAV still sub- stantially outperforms it in ATAC-acc and 3-mer correlation. This demonstrates that DAV Posterior is significantly more robust to reward over-optimization compared to these baselines.
Pred-activity (1)
ATAC-acc (f)
3-mer Corr (f)
Levenstein
Method
(turget)
(Validity)
(naturalness)
Diversity (1)
Pre-trained
0.13 (0.03)
0.018 (0.012)
0.000 (0.081)
111.58 (0.27)
DRAKES
6.05 (1.09)
0.784 (0.354)
0.229 (0.221)
90.38 (16.20)
VIDD
7.31 (0.09)
0.545 (1.425)
0.261 (0.074)
64.31 (10.06)
DDPO
7.51 (0.05)
0.129 (11266)
0.287 (0.114)
49.12 (13.83)
DAY
7.71 (0.26)
0.552 (0.109)
0.397 (0.145)
87.91 (3.73)
SVDD
5.06 (0.03)
0.244 (0.008)
0.675 (0.004)
64.72 (0.42)
SGDD (8 = 30)
8.66 (0.04)
0.225 (0.008)
0.090 (0.020)
110.14 (0.07)
SGDD (5 = 50)
8.77 (0.07)
0.223 (0.021)
0.090 (0.019)
109 94 (0.11)
DAY Posterior
9.24 (0.23)
0.920 (0.067) 0.347 (0.160)
87 (4.63)
Table 2: Comparison of sequence design methods.
29

---

Published as a conference paper at ICLR 2026
I SENSITIVITY TEST
This section provides a sensitivity analysis for the hyperparameters governing the test-time search in DAV. We examine four key parameters: (1) a, a temperature parameter that controls the exploration strength, where lower values lead to sharper sampling by amplifying the influence the soft Q- function; (2) y, the discount factor used for credit assignment; (3) Distillation Steps, the number of training cycles applied to the dataset collected a single E-step; and (4) Number of particles used for the importance sampling step in Equation (62). The experimental setting, except for these four hyperparameters, follows the setting in Section H.1.
Temperature As shown in Figure 6, lower yields higher reward scores. However, excessively small values may cause over-optimization, degrading ImageReward, CLIP score, and diversity. In- creasing a mitigates this effect, producing more diverse samples.
Discount factor 7, Figure shows that setting > Rs 0 stabilizes optimization. contrast, high values such = 0.95 fail to suppress early-step credit assignment, leading over-optinization. Interestingly, before collapse, reward and alignment scores are positively correlated with y, while diversity metrics exhibit negative correlation.
Distillation steps. illustrated Figure 8, increasing the number of distillation steps in M-step improves the reward when set above one. However, excessive steps reduce both alignment scores and diversity.
Number of particles. Figure 9 shows that the number of particles used for importance sampling has limited impact on overall performance. Yet, large values can harm alignment scores such as lewaeReward and CLIP. For computational efficiency, we set the number of particles to four.
ImageReward
0.55
0.20
8:50
0.15
0,022
0.0055
0,023
5006
0.035
0.305
04075
0.023
0005
30075
0,002
0066
30075
Figure 6: Effect of the inverse temperature parameter a performance metrics.
Acsthetic
ImageReward
CLIPScore
LPIPS-A
LPIPS-P
0.20
0.50-
8:15
6.12
8:5
8.4
640
Figure 7: Effect of the discount factor y on performance metrics.
Acsthticc
ImageRoward
CLIPScore
LPIPS-A
LPIPS-P
640
8.55
9.50-
8.0
Desilloon stes
Disiilatin steps
Disillaion steps
Disillatin
Disilbaion stags
Figure 8: Effect of the number of distillation steps on performance metrics.
Imapekeeard
CL.PSScore
LPIPS-A
0.22
0.55
0.204
9:0
Numbrra pariciio
Number pamidiza
Nambar of pamidia
Nambur particiio
particion
Figure 9: Effect of the number of particles on performance metrics.
30

---

Published as a conference paper at ICLR 2026
J
NON-DIFFERNNTIALE REWARDS
(a) Compressibility
(b) Incompressibility
350
25
300
-5
200
100
150
125 0
2
100
2
3
4
Epoch
Epoch
DAV-KI.
DDPO (100 epoch)
Figure 10: Optimization trends of DAV on compressibility and incompressibility rewards.
Compressibility
Incompressibility
DAV-KL (6 epochs)
DDPO
(100 epochs)
Figure 11: Qualitative comparison between DAV-KL and DDPO when optimized for the compress- ibility and incompressibility reward function.
DAV extends naturally non-differentiable rewards by skipping the gradient-based proposal struction described in Section E.2.1. As shown in Figure 10, DAV-KL. effectively optimizes both compressibility and incompressibility rewards. This result demonstrates the versatility of DAV in black-box reward optimization.
Figure 11 presents qualitative comparisons between DAV-KL and DDPO. DAV-KL produces more aligned samples with fewer training epochs. Under the compressibility reward, DAV-KL preserves only the core parts of the animals, whereas DDPO retains redundant background textures. For incompressibility. DAV-KL better captures the essential semantic features of the animals, while DDPO often fails do so despite DAV-KL achieving higher reward scores. These results confirm that DAV is effective for optimizing the non-differentiable rewards.
31

---

Published as a conference paper at ICLR 2026
K
COMPUTATIONAL COSTS ANALYSIS
Aesthetic score Table 3 reports the RTX 4090 GPU hours and performance of our method com- pared DDPO, DRaFT, and their KL-regularized variants. KL-regularized DRAFT and DDPO optimized following Equation 18 of Uchara et al. (2024a):
T
arg max E Pop(p() r(20) Pe
DKL(po( [x)]lppo (x)))
While DAV requires a substantial computational budget, its runtime remains comparable to the high- epoch DDPO and KL-regularized baselines. Crucially, DAV justifies this cost by achieving supe- rior trade-off: it attains the highest aesthetic scores while preserving LPIPS-A and ImageReward. In contrast, KL-regularized baselines suffer sigmificant degradation diversity and ImageReward even when consuming comparable or greater GPU hours.
Method-epochs
Aesthetic (t)
LPIPS-A (1)
ImageReward (1)
GPU hours
Pretrained
5.40
0.65
0.90
DDPO-100
6.08
0.63
0.96
18.1
DDPO-200
6.44
0.57
0.85
36.1
DDPO-300
0.54
0.67
$4.2
DDPO-400
0.48
0.28
72.2
DDPO-500
0.44
-0.44
90.3
DDPO+KL-400 (a=0.3)
6.93
0.47
0.47
82.7
DRaFT-42
7.22
0.46
0.19
1.7
DRaFT+KI--2001 (o=0.035)
6.78
0.59
0.23
220.0
DAV-100 (M=4)
8.04
0.53
0.95
82,4
DAV-KL-100 (M=2)
7.11
0.58
1.11
91.2
DAV-KL-100 (M=4)
6.99
0.58
1.13
98.7
Table Comparison of computational cost and performance for optimizing aesthetic score.
Compressibility and incompressibility As shown in Figure 10, DAV-KL trained for epochs substantially outperforms the DDPO baseline trained for 100 epochs. In terms of compute, DAV-KL requires 14.3 GPU hours RTX 3090, which is roughly half the cost of DDPO at 28.7 GPU hours.
DNA sequence design For discrete diffusion model fine-tuning we reproduce DDPO (Black et al., 2023) and VIDD (Su et al., 2025) using the official codebases of VIDD S and repro- duce DRAKES (Wang al., 2025) following its official implementation 6. All hyperparameters are set exactly specified in the original papers. On single RTX 3090 GPU, the training times approximately: 14 hours for DDPO, hours for VIDD, 43 hours for DRAKES, and 15 hours for DAV. Notably, DAV achieves comparable training time while yielding higher reward and naturalness with preserved diversity.
/a/DRAKSS
32


Reviews:
[{"review_id": "aqOIkbDs2Z", "reviewer": "Reviewer_8DWZ", "summary": "This paper presents a diffusion alignment method (DAV) that alternates between test-time searches as Expectation steps and online refinement of the diffusion model as Maximization steps. Specifically, the diffusion alignment problem is formulated as a soft RL objective, whose evidence lower bound is optimized an EM algorithm. In the E-steps, DAV draws posterior samples given a reward-tilted distribution; while in the M-steps, DAV distills the sampled trajectories into the diffusion model. Experimental results show the effectiveness of DAV compared to existing RL and direct preference optimization methods for both continuous and discrete diffusion models.", "questions": "- For the discrete diffusion model alignment, I am curious of how DAV compare to test-time sampling algorithms such as [3,4], which also consider the same DNA enhancer task, and also alignment methods designed for discrete diffusion models (e.g., [5,6]).\n\n[3] Li et al. \"Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding\", arXiv: 2408.08252.\n\n[4] Chu et al. \"Split Gibbs Discrete Diffusion Posterior Sampling\", NeurIPS 2025.\n\n[5] Borso et al. \"Preference-Based Alignment of Discrete Diffusion Models\", ICLR 2025 Bi-Align Workshop.\n\n[6] Zhu et al. \"LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models\", arXiv: 2505.19223.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper offers a fresh perspective by aligning diffusion models with the EM algorithm. I especially appreciate this idea because the multi-round iterative alignment could potentially help in settings where the reward is costly or intractable to evaluate—for example, when it requires human evaluation or expensive wet-lab experiments.\n- The proposed method is accompanied by rigorous derivations and theoretical guarantees.", "weaknesses": "- Experiments only include one example for continuous diffusion and one for discrete diffusion. The case for generality would be stronger with additional tasks (e.g., compressibility or prompt alignment as in DDPO). Moreover, some of the recent methods are not included or discussed as well, such as DSPO[1], DanceGRPO[2].\n- The EM algorithm may be substantially more expensive than the methods it is compared to (e.g., DRaFT or DDPO), given the test-time search required in each expectation step. However, there is currently no analysis on the runtime or convergence speed of DAV.\n\n[1] Zhu et al. \"DSPO: Direct Score Preference Optimization for Diffusion Model Alignment\", ICLR 2025.\n\n[2] Xue et al. \"DanceGRPO: Unleashing GRPO on Visual Generation\", arXiv: 2505.07818.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KCCBUntwA", "reviewer": "Reviewer_77LS", "summary": "The authors propose DAV, an alignment algorithm for diffusion models via variational expectation maximization. In the E-step, DAV employs test-time search algorithms to generate samples from the reward-weighted posterior distribution. In the M-step, the diffusion model is fine-tuned using the samples from the E-step. The authors demonstrate the effectiveness of DAV on both the continuous text-to-image generation task and the discrete DNA sequence design.", "questions": "Please refer to the **weaknesses** section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea to formulate diffusion alignment as a variational expectation-maximization problem is interesting.\n- The paper is well-written, and the theory and method are well-motivated.\n- Experiments showcase the effectiveness of DAV.", "weaknesses": "- The searching algorithms lead to computational overhead. Therefore, a fairer comparison with the baselines should also take the computational cost into account. For example, it would be helpful to compare model performance under the same computational budget and the performance scaling curve of TR2-D2 as the computation increases.\n- The value of 3-mer correlation for DNA sequence design is significantly lower than those reported by baselines, e.g., in DRAKES paper the value is 0.887, much higher than DAV (0.397), while in table 2 and figure 5, it is only 0.229. Also, the target and naturalness are two competing properties, and one can get a higher value of one property by sacrificing the other via hyperparam tuning or using different training epochs. Does DAV have Pareto optimal performance compared to baselines? \n- The E-step can lead to an inaccurate estimation of the posterior distribution, due to the limited sample size and the value estimation error in the test-time algorithms. How does this affect the M-step optimization, and is the model robust to a suboptimal posterior distribution (e.g., with fewer samples or inaccurate samples)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2yPV7MeCuC", "reviewer": "Reviewer_KMQx", "summary": "The paper introduces DAV, a novel framework for fine-tuning pre-trained diffusion models. The authors motivate the work by claiming to address reward over-optimization.", "questions": "1. For the results presented in Figure 3, do the images for each algorithm come from a single training run, or are they gathered over multiple runs? If from a single run, please report the standard deviation rather than only the mean.\n2. For discrete finetuning, usually it's straightforward to test both DNA and RNA sequences. Can the authors provide explanations on why only the DNA enhancer is tested?\n3. See other questions above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work formulates diffusion alignment as an iterative Variational Expectation-Maximization, which appears to be a new and interesting theoretical lens for diffusion fine-tuning.2. \n2. The proposed DAV framework enjoys broad applicability. It can accommodate both continuous and discrete settings.\n3. The presentation of this work is easy to follow.\n4. The empirical results show DAV and DAV-KL enjoy superiority over multiple strong baselines, such as DDPO, DRaFT.", "weaknesses": "1.  The main comparison in Figure 3, which plots aesthetic reward against diversity/naturalness, is confusing and potentially incomplete. The reported performance of the RL-based if they are properly trained with \"suitable\" KL penalty (which might be non-trivial to choose). This raises questions about the optimal tuning of these baselines. \n2. Furthermore, the analysis in Figure 3 omits purely inference-time methods, which are often competitive in image experiments.\n3. As noticed by the authors, this method has non-negligible computation costs. The E-step involves substantial \"additional test-time computation\" through gradient-guided search. In large-scale diffusion model finetuning, DDPO already takes much time to converge (compared to the fastest direct propagation). It is important to quantitatively present the added training-time overhead of DAV relative to DDPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dJyAgmWkUr", "reviewer": "Reviewer_xurq", "summary": "The paper develops an approach to “Diffusion Alignment as Variational Expectation-Maximization” (DAV) that alternates between two complementary phases: the E-step, which is essentially an exploration step that aims to discover diverse and high-reward samples from the variational posterior; and the M-step, “amortization”,  which refines the diffusion model using samples identified from the E-step.", "questions": "Can the author(s) shed some light on how to choose the value of the KL-coefficient \\lambda in (11), which is meant to control “the trade-off between aligning with the expert policy and pre-serving the pretrained model”? In particular, is the value \\lambda robust or not with respect to downstream applications?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The DAV approach is built on solid technical foundations as outlined in \\S4. For instance, the E-step uses gradient-based guidance and importance sample to enhance the exploration. The M-step minimizes a mode-covering objective that incentivizes the covering of all diverse modes generated in the E-step. \n\nThe combined E-M steps iteratively refine the model towards a multi-modal aligned distribution; and this overcomes problems like over-optimization and mode collapse that often arise in RL.", "weaknesses": "The M-step involves two variations, in addition to the standard DAV objective in (10), there’s a variation,  DAV-KL in (11). From the experimental results in Table 1, it’s not clear which one to use and when, except perhaps the ad hoc trial and error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)