David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training
Authors: Weijian Luo
colin zhang
Debing Zhang
Zhengyang Geng
Venue: ICML 2025 poster (2025)
Keywords: One-step Text-to-image Generative Models
Reinforcement Learning
Human Feedback
Diffusion Models
Primary area: applications->computer_vision
TL;DR: We introduced a novel general score-based RLHF method and a SoTA 1-step Text-to-image generative model.

Abstract: We propose Diff-Instruct*(DI*), a data-efficient post-training approach to one-step text-to-image generative models to improve its human preferences without requiring image data. Our method frames alignment as online reinforcement learning from human feedback (RLHF), which optimizes a human reward function while regularizing the generator to stay close to a reference diffusion process. Unlike traditional RLHF approaches, which rely on the KL divergence for regularization, we introduce a novel score-based divergence regularization that substantially improves performance. Although such a score-based RLHF objective seems intractable when optimizing, we derive a strictly equivalent tractable loss function in theory that can efficiently compute its gradient for optimizations. Building upon this framework, we train DI*-SDXL-1step, a 1-step text-to-image model based on Stable Diffusion-XL (2.6B parameters), capable of generating 1024x1024 resolution images in a single step. The 2.6B DI*-SDXL-1step model outperforms the 12B FLUX-dev model in ImageReward, PickScore, and CLIP score on the Parti prompts benchmark while using only 1.88% of the inference time. This result strongly supports the thought that with proper post-training, the small one-step model is capable of beating huge multi-step models. We will open-source our industry-ready model to the community.

David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training
Weijian Luo" Colin Zhang'
Debing Zhang'
Zhengyang Geng?
Abstract
1. Introductions
Generative models have made substantial progress in re- years, largely improving the creative content cre-
We propose Diff- Diff-Instrucc((III)) a data-cfficient post-training approach for one-step text-to-image generative models to improve its human prefer- ences without requiring image data. Our method frames alignment as online reinforcement learn- ing from human feedback (RLHF), which opti- mizes the one-step model to maximize human reward functions while being regularized to be kept close to reference diffusion process. Un- like traditional RLHF approaches, which rely on the Kullback-Leibler divergence as the regular- ization, we introduce a novel general score-based divergence regularization that substantially im- proves performance as well as post-training sta- bility. Although the general score- based RLHF objective is intractable to optimize, we derive a strictly equivalent tractable loss function in the- ory that can efficiently compute its gradient for optimizations. We introduce DI*-SDXL-Iste -step. which is 2.6B one-step text-to-image model resolution of 1024 x 1024, post-trained from DMD2 w.r. SDXL. Our 2.6B DI model outperforms the 50-step 12B FLUX-dev model in ImageReward, PickScore, and CLIP score on the Parti prompts benchmark while us- ing only 1.88% of the inference time. This result clearly shows that with proper post-training, the small one-step model capable of beating huge multi-step diffusion models. Our model is open- sourced at this link: https::/ pkulw 11 1994/diff inssruuc_sstae We hope our findings can contribute to human- centric machine learning techniques.
ation across various domains (Karras et al.
2020 Nichol
& Dhariwal 2021
Poole et al.] 2022:
Kim et al.
2022:
Tashiro et al. 20021
Meng et al., 2021 Couairon et al.
2022
Ramesh et al.
2022
Esser et al. 2024|
Oord et al.
2016
Ho et al.
2022
Brooks al.
2024
Zhang et al.
2023a
Xue al.
2023
Luo & Zhang 2024
Luo al.
2023b
Pokle et al.
2022
Zhang al.
2023b
Feng al.
2023
Deng et al.
2024
Luo et al.
2024d Geng et al.
2024
Wang et al.
2024).
Among these advancements, two types of generative mod- els have gained significant attention, diffusion models (DMs) and one-step generators. Diffusion models (Sohl-) Dickstein et al. 20155 Ho al. (2020), score-based generative models (Song et al., 2020), progressively cor- rupt data with diffusion processes and then train models to approximate the score functions of the noisy data dis- tributions across varying noise levels. The learned score functions can generate high-quality samples by denoising the noisy samples through reversed stochastic differential equations. While DMs can produce good outputs, they of- ten require large number of model evaluations, which lim- its their efficiency in applications.
In contrast, one-step generators (Luo et al. 2024c a Zheng & Yangl 2024 Kang et al. 2023 Sauer et al., 2023a Yin ct al. 2024 Zhou et al. 2024a) have emerged as a highly efficient alternative to multi-step diffusion models. Un- like DMs, one-step generative models map latent noises directly to data samples in single forward pass, making them highly efficient for real-time applications such as text- to-image and text-to-video generations on edge devices. Many existing works have demonstrated the leading per- formances of one-step eext-to-image generators (Luo et al.] 2024c Zhou et al. [024a] Yin et al.] 2024) by employ- ing diffusion distillation (Luo 2023) as well as GAN tech-
com/
'Hi Lab of Xiaohongshu Inc, Beijing, China. "School of Com- puter Science, Carnegic Mellon University, Pittsburgh, USA. Cor- respondence to: Woijian Luo <luoweijian xiaohongshu.com.
niques Goodffllow et al. 2014 Brock et al.) 2018 Karras
et al. 2020 Sauer et al.) 2023b
Yet, these works only focus on matching the one- step model's distributions with pre-trained diffusion
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
models or ground truth data, overlooking the critical

---

One-step Model Beats Large Diffusion with Score Post-training
challenge of aligning one-step text-to-image models with human preferences, which is one of the central requirements of human-centric Al.
process of diffusion model can transform initial distribu- tion zo towards some simple noise distribution,
dæ. = F t)dt + G(t)dw:,
(2.1)
To close this gap, we introduce Diff-Instruct* (DI*), a novel post-training approach that aligns one-step text-to- image generators with human preference while maintain- ing the ability to generate diverse and photo-realistic high- resolution images. We frame the human- suman-preference align- ment problem as reward maximization with basee di- vergence constraints. This yields generated samples that not only have improved human reward but also adhere to user prompts. Unlike traditional RLHF methods (Chris- tiano et al. 2017 Ouyang et al., 2022 Luo 2024), which rely on Kullback-Leible (KL) divergence for distribution regularization, we demonstrate that score-based regulariza- tion is important to preserve sample diversity and avoids reward hacking- - the model generates weird, painting-like outputs with high rewards but low realism.
where F is a pre-defined vector- valued drift function, G(t) is a pre-defined scalar-value diffusion coefficient, and w, denotes independent Wiener process. A continuous- indexed score network 8,( () is employed to approxi- mate marginal score functions of the forward diffusion pro- cess 1..). The learning of score networks is achieved by minimizing a weighted denoising score matching objec- tive (Vincent 2011
Song et al. 2020).
LDSM
= 1= my lmn Pe (m (ma)
X()
(30e (zee t)
- Va, log (ze (20) 112de.
The weighting function X(I) controls the importance of the learning at different time levels, and pr (2 denotes the conditional transition of the forward diffusion (2.1) J. After training, 8 () 2 Va, log gi (æx)) is a good approxima- tion of the marginal scores of the diffused data distribution.
Our experiments demonstrate three key advantages: 1) Efficcinny: The 2.6B D/* model generates 1024 1024 resolution images in a single step, using only 1.88% of the inference time and 29.3% of the GPU memory of the 50-step 12B FLUX-dev model. 2) Su- perior Performance: On Parti prompts, DI*-SDXL-Iste achieves higher PickScore, ImageReward, and CLIPScore than FLUX-dev-50stepy It also sets a new state-of-the-art HPSv2.1 score of 31.19 among open-source models. 3) Fidelity: The model maintains diversity and prompt ad- herence, matching FLUX-dev and SD3.5-large on COCO benchmarks while being orders of magnitude faster.
3. Score-based Post-training
Problem Setup. Our basic setting is that we have known human reward function r(æo,, c). which encodes the human preference for an image 2o and corresponding text description c. Besides, we also have a pre-trained dif- fusion model which will later act as a reference distribu- tion Pref(#a) = qu (zro). The reference diffusion model specified by the score function 8y, (3x) == Va, log qe (are), where gz,')'s is the underlying distribution diffused at time t according to (..I. The pre-trained diffusion model represents the ground- truth data distribution that is used to prevent the one-step model from seeking high human re- ward but losing the ability to generate realistic images.
We summarize our contributions as follows:
We introduce Diff-Instruct* for preference alignment of one-step text-to-image generative models with a strict theoretical guarantee based on a new score- based PPO paradigm;
We introduce two novel approaches to incorporate classifier-free guidance into human preference align- ment through explicit-implicit reward decoupling.
Our goal is to train a human- -refen-peeferred one-step text-to- image model go that generates images by directly mapping random noise z ~ P= to obtain Zo = go(2,c) con- ditioned on the input text C ~ C. We want the one-step model's output distribution, po po (ooolcc) (c), to maximize the pected human rewards while maintaining the ability to gen- erate realistic images. We force po (Zolc) to not diverge from Pref (-) by using a divergence regularization term. Let D(..-) be distribution divergence. For any fixed prompt c, the training objective is defined as:
With extensive evaluations, our best open-sourced 2.6B DIs-SDXL-Ister text-to-image model, a leading text-to-image model with a resolution of 1024 x 1024 that significantly outperforms the leading FLUX-dev- 50step model with 1.88% inference time.
0" argmin Ton Pag e) (2). Dipe. Pipee (..))
2. Preliminary
Diffusion Models. Here, we introduce preliminary knowledge and notations about diffusion models. Let (z)) = (z) be the data distribution. The goal of gener- ative modeling is to train models to generate new samples ~ (ar). Under mild conditions, the forward diffusion
Here a is a coefficient that balances reward influences and (-)) acts as regularization term. When using Kullback- Leibler divergence as the regularization term, the objective turns to the online Proximal Policy Gradient (PPO) algorithn(Schulman et al. 2017), which has much success
3..
2

---

One-step Model Beats Large Diffusion with Score Post-training
Figure 1: Generated 1024 x 1024 images from one-step 2.6B DI*-SDXL model. The DI-SSDXL-Istep model outperforms FLUX-dev-50step Lab 2024) and SD3.5-large-28stepy 2024) in human preference metrics the Parti prompt bench- mark and HPSv2.1 scores, Please zoom in to check the details.
in reinforcement learning from human feedback (RLHF) for large language models (Ouyang et al., 2022) and one- step text-to-image models alignment (Luo) 2024). Par- ticularly, recently (Luo 2024) has shown that traditional KL-PPO can fairly train human- preferree one-step text-to- image models. However, KI. divergence is notorious for mode-seeking behavior, which makes Luo (2024) very easy to collapse to some distribution modes such as
painting-like images - that have high human reward but lack diversity. To address this issue, we find that gen- eral score-based divergences have an appealing diversity- preserving property that has been proved in one-step dif- fusion distillation (Luo al. 2024c Zhou al. 2024b). Based on this intuition, we propose Inst-uuts novel post-training approach built upon new Score- online PPO paradigm that uses a general score-based divergence
3

---

One-step Model Beats Large Diffusion with Score Post-training
to regularize the RLHF process.
tion Prrete, (2rg) pointwise, which was also done in the lit- erature of diffusion distillations works such as Luo et al.
3.1. Score-based Divergences
(2024b c): Zhou et al. (2024a Yin et al. 2003 2024). Therefore, we can alternate between 1) updating the as- sistant diffusion 8y (2,,), 4) using one-step model generated samples (which are efficient) and 2) updating the one-step model by minimizing the tractable objective (3.4). We name our training method that minimizes the objective because it is inspired by Diff-Instruct (Luo et al. (024b) and Diff-InstructH (Luo) 2024) that involves an additional diffusion model and re- ward model to train one-step text-to-image models.
Inspired by Score-Implicit Matching (SIM (Luo et al.) 2024ch) theory that revealed the advantages of general score-based divergences over the traditional KL diver- gence, we define the regularization term D(pe, Pref) in (3.1) via the following general score- divergence. As- sume d Idd + R is a scalar-valued proper distance fune- tion (i.c., a non-negative function that satisfies Ya, d(x)) > and ((x) = if and only if æ = 0). Given a parameter- independent sampling distribution ny that has large distri- bution support, we can formally define time-integral score divergence
CD) in (3.4) the
3.3. Understanding Classifier-free Guidance
Classifier-free Guidance Secretly Induced an Implicit Reward. In previous sections, we have shown in theory that with explicitly available reward models, we can readily train the one-step models to align with human preference. In this section, we enhance the DI by incorporating the classifier-free reward that is implied by the classifier-free guidance of diffusion models.
[" D D Pres) = L-" d( - -s (3.2)
where Po.: and ge denote the marginal densities of the dif- fusion process (2.1) at time initialized with pe.0 pe and qu Pref respectively. w(t) is an integral weighting func- tion. Clearly, we have DI (po,Prec = 0 if and only if Po po(xu) Pref( (2o), a.8. Tu-
The classifier-free guidance (Ho & Salimans uses a modified score function of the form
2022 (CFG)
3.2. Theories for Tractable Objectives
ãce (me))tc) s ())) w (12, ( (()))
Recall that go is a one-step model, therefore, samples from Pa can be implemented through a direct and differentiable mapping Zy go (z, c). With the score-basee based regulariza- tion term (3.2). for each given text prompt c, we can for- mally write down our training objective to minimize as:
to replace the original conditional score function Sref (Zt, Using CFG for diffusion models empirically leads better sampling fidelity but with a cost of diversity.
As is first pointed out by Luo 2024), the classifier-free guidance is related to an implicit RLHF. In this part, we derive tractable loss function that minimizes the so-called classifier free reward, which we use together with the ex- plicit reward r(:: :) in DI
(0) = Rap ar(wo. c)] + D t0. (p6. 28.9)((.0)
(3.3)
Now we are ready to reveal the objective of Diff-Instruct* that we use to train prefereeeee one-step models ge- Notice that directly minimizing objective (3.3) is in- tractable because we do not know the relationship between / and corresponding Po,t- However, we show in Theorem
Theorem 3.2. Under mild conditions. if we set an implicit reward function as (3.6), the loss (3.5)
3.1 that an equivalent tractable loss (3.4) will have the same gradient as the intractable loss function
Letp = [."
E Ex mopoop(t w(() (ra)) (egle.]]).)) (3.5) as-y(eslep)
(DI- (0)
amps.
an(ma. c)
(3.4)
an (xn 2) m,
2D =gg(()
(1)BB mpeiie m0)
d' (
has the same gradient as the negative implicit reward func- tion (3.6)
==
( (ax) - Vo, Do /m0) (m))) joa)]
-r(mo c) = - 1. Bai-po, w((t log Pref (wx (t,
dr.
(3.6)
pref(m,(1) (t)
with y: == Spactor, (xx:) - Sqe (xx.).
Theorem 3.1. Under mild assumptions, if we take the sampling distribution in (3.2) as ,, Peglo then the gra- dient of (33 w.r.t 0 is the same as 3.4p:
The notation sg. sg[az]] means detaching the 0 gradient on 2;,-
Theorem 3.2 gives tractable loss function (3.5) aiming
to minimize the negative classifier-free reward function. Therefore, we can scale and add this loss (0) (3.5) to
ag Corig
ae (O).
the DI* loss 3.4) to balance the effects of explicit reward and implicit CFG reward.
In practice, we can use another assistant diffusion model 8yp (ax,,) to approximate the one-step model's 's score func-
Choice 1: Incorporating CFG via Implicit CFG Re- ward. Based on Theorem 3.2. we can incorporate
4

---

One-step Model Beats Large Diffusion with Score Post-training
Algorithm 1:
Pseudo Code.
Input:
prompt dataset C, generator go g0( c), prior distribution reward model r(æ, c). reward model scale arews CFG reward scale Gcfg, reference diffusion model f (m) cc, c), assistant diffusion s1 [t, c), forward diffusion p(az)(a)) (2.1), assistant diffusion updates rounds KTA, time distribution T()), diffusion model
weighting A(t). generator IKL loss weighting w()).
while not converge do freeze f, update vb for KA rounds by
1. sample prompt e C; sample time ~ xl); sample ~ Pz(z);
2. generate fake data: zo sgigo(z, c)]; sample noisy data: ~ pe(2)((ao)
3. update & by minimizing loss: [(() A(t)ll (xg)tt Va. log pe (2))
freeze v. update 9 using SGD:
1. sample prompt e C; sample time (t); sample p=(z);
2. generate fake data: xn = gz, c); sample noisy data: my ~ (=))))00))
3. explicit reward: Crew( (f) Frewr(aa C);
4. CFG reward: (0) aef9 w(t) (re))(see(ss//z)lll - seg 0))²;; zx;
sves 1, c))} (2rl))),, Va, (as 6. update 0 by minimizing DI loss: LDI (0) frew (0) Lefa (0) + (0);
5. score-regularization: Lreg (0) -
end
return 8, vh.
classifier-free guidance by adding the pseudo reward (3.6) to explicit human reward to get a mixed reward. The guid- ance scale Cefa and O'eew balances the strength of explicit human reward and implicit CFG reward.
such as PickScore (Kirstain et al. 2023). CLIP score (Rad- Ford et al., [202I), or rewards trained in-house with internal feedback data.
Pseudo-Huber Distance Functions. Various choices of distance function d(-) result in different training algo- rithms. For instance, d(y,) = || [| is a naive choice. This distance function has been studied in the pure diffusion dis- tillation literature (Luo et al.) 2024c Zhou al.] (004bbaa) a). In this paper, draw inspiration from (Luo al. 20240 and find that using the so-called pseudo-Huber distance leads to better performance. The distance writes d(y) =
Choice 2: Incorporate CFG via CFG-enhanced Ref- erence Diffusion. Besides the implicit CFG reward, we can also inject the CFG mechanism into Disf-Instruct* by replacing the naive reference diffusion with a CFG- enhanced reference diffusion which writes Fref ( (t,)) == 8ref (a:ll |4, e) + (efa((ref((t))) 2)). This approach is straightforward. In practice, we find empirically that the choice 2 is easier to tune and results in better performances. However, choice 1 also leads to solid one-step models. Please check out ablation studies in Table 2 for details.
/ /lll.ll 11 - c² and Spuzto (xx) Sqc (Ix)) then
D (o. (pa, Pref)
(3.7)
Vi
(0((0,))) Vz, (ogre(a;
VIlw
3.4. The Practical Algorithm
As Algorithm shows, the DI involves three models, with one one-step model go. one reference diffusion model sref and one assistant diffusion model sg. The reference diffu- sion is frozen, while the one-step model and the assistant diffusion are updated alternately. Two hyperparameters, the arew and aefg. control the strength of the explicit re- ward and the implicit CFG reward during training- The explicit rewards can either be off-the-ske the-shelf reward models,
4. Related Works
Diffusion Distillation Through Divergence Minimiza- tion.
is inspired by research on diffusion distillation (Luo 2023). which aims to minimize distribu- tion divergence to train one-step text-to-image models. Luo et al. (2024b) first studies the diffusion distillation by mini- mizing the Integral KL divergence. Yin et al. (2023) gener- alizes Inftrunst and adds a data regression loss for bet-
5

---

One-step Model Beats Large Diffusion with Score Post-training
ter performance. Zhou et al. (2044b study the distillation
high-quality off-the-shelf reward model trained using hu- man feedback data) as the explicit human reward. In Ta- ble we do an ablation study that compares one-step DI* models with different guidance strategies and reward scales and finds that using inner guidance with a scale of 7.5 and an explicit human reward with a scale of 100 results in the best performance.
by minimizing the Fisher divergence. Luo al. (2024c)
study the distillation using the general score-based diver-
gence. Many other works also introduced additional tech-
niques and improved the performance (Cieng et al.
2023
Kim et al. 2023 Song at. 2023 Song & Dhariwal
2024
Nguyen & Tran
2023
Song et al. 2024, Yin et al.
2024
Zhou et al.
2024a;
Heck et al., 2024;
Xie et al., 2024
Sal-
Quantitative Evaluations Metrics. We compare the DIs-SDXL-Istep model with other leading open-soureed models of resolution of 1024 x 1024 on the Parti prompt benchmark, COCO-2014-30K benchmark, and also Human preference scores. These models include diffusion- (or flow-matching) models such as FLUX-
imans al.
2024
Geng et al., 2024
Meng al..
2022
Sauer et al.
2023b
Luo et al.; 2023a;
Liu et al.
2023 Gu
et al.
2023
Xu et al.
2024:
Ren et
al.
2024
Lin et al.
2024
Zheng et al.
2024:
Li ct
al.
2024
Berthelot et al.
2023
Zheng et al.
2022).
dev (Lab) 2024), Stable-Diffusion- -.r-gge 2024).
Preference Alignment for Diffusion Models and One- step Models. In recent years, many works have emerged trying to align diffusion models with human preferences. There are three main lines of alignment methods for dif- fusion models. 1) The first kind of method fine-tunes the diffusion model over a specifically curated image-prompt dataset (Dai et al., 2023 Podell et al., 2023). 2) The second line of methods tries to maximize some reward functions either through the multi-step diffusion genera- tion output (Prabhudesai et al. 2023 Clark et al., 2023 Lee et al., 2023) or through policy gradient-based RL approaches (Fan et al. 2024 Black et al. 2023). For these methods, the backpropagation through the multi- step diffusion generation output is expensive and hard to scale. 3) The third line, such as Diffusion-DPO (Wal- lace et al. 2024). Diffusion-KTO (Yang et al.) 2024). and MaPO (Hong et al.) (024) tries to directly improve the diffusion model's human preference property with raw collected data instead of reward functions. Besides the human preference alignment of diffusion models, Diff- Instruct++(Luu 2024) recently arose as the first attempt to improve human preferences for one-step text-to-image models. Though we get inspiration from Disf--nstrruc++,, our Diff-Instruct** uses score-based divergences, which are technically different from the KL divergence used in DI++, Besides, we show in Section 5.1. Diff- outper- forms Disf-Instruct++.
and SIXL (Podell et al. 2023) with its DPO variant (Wal-
lace et al. 2024); 1024 resolution one-step models such as
DMD2 Yin et al., 2024). Diff-Instruct(Luu et al. 2024b),
Score Implicit Matching (SIM) (Luo et al. 2024c). and also Diff-Instruct++ (Luo, 2024). On Parti and COCO prompt datasets, we compute four standard scores in Ta- ble I the Image Reward (Xu et al.) 2023), the Aesthetic Score (Schuhmann) 2022). the PickScore (Kirstain et al. 2023), and the CLIP score (Radford et al. evaluate the 30K-FID (Heusel et al. 2017 on the COCO validation dataset. On Human Preference Score (Wu et al. 2023) benchmarks, we calculate HPSv2.1 scores.
2021). We also
5.1. Performances and Findings
2.6B DI*-SDXL-1ste Model Shows Very Strong Per- formances. The FLUX-dev-50stee model (Lab] 2024] is recognized as a leading open open-sourced diffusion model that is trained on millions of high-quality internal data samples. As shown in Table our best 2.6B DI*-SDXL-Istep model outperforms the 12B FLUX-dev- and 8B SD3.5-large-28ster using only 1.88% inference time and 29.3% GPU memory costs on the Parti prompt dataset which consists of challenging prompts with rich concepts. The 2.6B DJ*-SDXL-Istee is on par with FLUX and SD3.5 on COCO prompts in terms of preference scores and FIDs. As Table 3 shows, the 1 model has a record-breaking HPSv2.1 score of 31.19, outperform- ing the rest of the text-to-image models of 1024 x 1024 res- olution. These empirical results confirm that proper post- training enables small one-step models to outperform huge multi-step models, analogous to the story of David and Go- liath. See Figure 2 for visual comparisons with the 12B FLUX-Dev and the 8B Stable Diffusion 3.5-Large models.
diffusion model
5. Experimeets
Experiment Settings for SDXL Experiment. We use the open-sourced SDXL of 1024x 1024 resolution as our reference diffusion in Algorithm [ We construct the one- step model with the same architecture as the SDXL model and initialize the one-step model weights with the DMD2- SDXL- model (Yin et al. 2024), which is good one- step text-to-image model distilled by using a combination of Diff-Instrrut loss and GAN loss. We use the prompts of the LAION-AESTHETIC dataset with an aesthetic score larger than 6.25, resulting in a total of 2.2 million text prompts. We use the PickScore (Kirstain et al. 2023) (a
Ablation Study. In Table 2, we conduct comprehen- sive ablation study on the Parti prompt dataset to compare DI* (RLHF using score-based divergence) with different guidance strategies and reward scales (which recover previ- ous distillation approaches such as Diff-Instrrct(Lunn et al.] 2024b) and Score Implicit Matching et al. 2024c))
6

---

One-step Model Beats Large Diffusion with Score Post-training
Table 1: Quantitative comparisons of 1024 x 1024 resolution leading text-to-image models on Parti Yu et al., 2022 Prompts (the upper part) and MSCOCO-201 validation prompts (the under part). DI short for Diff- Instrucct SIM: short for Score Implicit Matching. 1: our implementation.
MODEL
STEPS TYPR PARAMSI IMAGET REWARD SCORE
AesT
PICKÍ SCORE
CLIPT SCORE
INFER TIME. PER 10 IMAGRS
PARTI PROMPTS
BDXE
AL.
202.3
.0
0.2274
32.72
111
SDXI--DPO ET AL.
2024
50
UNET
2.6B
1.102
5.77
0.2290
33.03
111 SEC
LD3.5-LARGHAAI 2024
28
DIT
1.133
5.70
0.2306
32.70
66.23 SEC
LEUX-DEY-LBB 2024
50
DIT
12B
1.132
5,90
0.2317
31.70
118.64 SEC
DMD2-SDXL VIN ET 2024)
UNET
2.6B
0.930
5.51
0.2249
32.97
2.22 SEC
INSTRINTTTTT' LUD
20248)
UNET
2.6B
1.058
5.60
0.2253
33.02
2.22 SFC
STM (Luo AL.. 2024c)
UNET
2.6B
1.049
5.66
0,2273
32.93
2.22 SEC
DIFF-NSTRRUC+++ SDXL LuO
2024
UNET
2.6B
1.061
5.58
0,2260
32.94
2.22 SEC
DD (OURS)
UNET
2.6B
1.067
5.74
0.2304
32.82
2.22 SEC
DDXL (OURS, LONG TRAIN)
UNET
2.6B
1.140
5.83
0.2331
32.75
SEC
DI--SDLL-DDPO (OURS, LONG TRAIN)
UNET
2.6B
1.210
5,90
0.2342
32.91
2.22 SEC
COCO-2017-VAL, 30K PROMPTS
COCO-FIDJ
SDXL-BASC PODELL ET AL.
2023
50
UNET
2.6B
0.825
5.55
0.2281
86
15.89
SOXL-DPO(WALLKCE ET
2024
50
UNET
2.6B
0.950
5.69
0.2295
32.17
21.48
SD3.5-LARGE(A 2024
28
DIT
8B
1.048
5.46
0.2305
31.85
17.69
FLUUX- (LAB 2024
50
DIT
12B
1.065
5.64
0.2333
30.88
24.77
DMD2-SDXL (IN ET AL 2024)
UNET
2.6B
0.822
5.42
0.2251
31.89
15.80
INSTRUCT- SDXL (Luo
2024h
UNET
2.6B
0.907
5.48
0,2264
31.85
21.62
SIM-SDXL' LUo BT 2024c)
UNET
2.6B
0.925
5.54
0.2277
31.90
18.84
MIN=INSTRUUTT++SSDDA LLo 2004
UNET
2.6B
0.908
5.53
0.2265
31.85
21.42
DI--DDXL (OuRS)
UNET
2.6B
0.933
5.57
0.2305
31.75
19.09
DD (OURS, LONGER TRAINING) DI--SDXL-DPO (OuRS, LONG TRAIN)
0,2335
31.57
21.15
UNET
2.6B
1.034
5.65
0.2342
31.58
22.12
and DI++ Luo (RLH with KL. divergence).
DI is compatible with Pre-aligned Diffusion Models Beyond the ablation comparisons of different explicit re- ward strengths and CFG strengths, we also find that DJ# is compatible with pre-aligned reference diffusion models. As
The results show that DI* using score-based divergence outperforms the DI++ model under the same configurations and reward scales. Notably, with proper reward scales, DI* models demonstrate superior human preference scores, indicating its simple yet effective contribution to align- ing one-step text-to-image models with human feedback. Critically, DI+ with KL divergence tends to collapse to painting-like images with over-saturated color and lighting (Please refer to Figure for visualizations). which lack di- versity despite high rewards. Instead, score-based diver- gences empirically maintain diversity while steadily im- proving human preferences, suggesting that DI* can im- prove images' human preference scores regardless of their styles in paintings, photos, or animations.
Table [ and Table show, if we replace the reference dif- fusion model from SDXL open-sourced SDXL-DDPO (Stable Diffusion XL with Diffusion Direct Preference Op- timization (Wallace et al. (2024)), the preference alignment effects of the resulting one-step model are measured by PickScore, ImageReward, AestheticScore, and 1PSv2.1, and were further improved. This demonstrates the compat- ibility and robustness of Diff-Instruct* with different ref- erence diffusion models. In practice, we recommend that users pre-aligned diffusion models that
Qualitative Comparisons. Figure [ shows some 1024x 1024 images generated by DI*-SDXL-Ister mod- els. Figure 2 shows qualitative comparison of the DI*-SDXL--Istep model against FLUX and SD3.5 diffu- sion models. The DI*-SDXL-Istep model shows better aesthetic details, improved layouts, and reality lighting compared FLUX and SD3.5, attributed to our score post-training. Figure 4 shows the visual comparison of DI* (with different ablative reward scales) against our initial model (the DMD2-Istep model(Yin et al., (024)), other one-step models, and SDXL with and without DPO. While SIM (Luo et al. 2024c). DI++{Luo 2024), and Diff-Instruct(L... et al.
In addition, we acknowledge the trade-off between pref- erence scores and CLIP scores: increasing explicit human reward scales degrades CLIP scores. This phenomenon re- veals natural contradiction between huwan--reeeene preferreee and objective data distributions. RLHF encourages the one- step model to generate human-preferce images, which in- evitably shifts the underlying distribution from the ground- truth data distribution. This aligns with observations that FLUX, despite strong preference scores, underperforms in CL.PPSoore and FID compared to smaller models.
2024b) can output high-quality
7

---

One-step Model Beats Large Diffusion with Score Post-training
Table 2: Ablation study on Parti Yu et al., 2022) Prompts of 1024 resolution DI*-SDXL-Istep models with different reward scales. DI:: short for Diff-Instructt SIM: short for Score Implicit Matching, DIs-Out: incorporating CFG with implicit CFG reward; DI incorporating CFG with enhanced reference diffusion. f; our implementation.
MODEL
STEPS
PARAMS
IMAGET REWARD
Aest SCORE
PICKt SCORE
CLIPT SCORE
(arc
RMD2-SDXL(CU MODEL)
2.6B
0.938
5.51
0.2249
32.97
DI++-SDXL (ALIGNED USING KL)
2.6B
0.846
5.50
0.2243
32.66
(0,
DI++-SDXL (EQU TO DUFE-NNTTRUCT
2.6B
1.058
5.60
0.2253
33.02
(0, 7.5)
DI++-SDXL' (ALIGNED USING KL)
1.061
5.58
0.2260
32.94
(100,
DI'-OUT-SDXL (OUT CFG)
2.6B
1.082
5.63
0.2263
33.03
(100,7.5)
DI--N--DXLL (BASELINE, No REWARD) (EQU
2.6B
0.782
5.74
0.2256
32.16
(0,0)
CFG)
1.049
(0,7.5)
DI--IN-DXXL (HUMAN REWARD CFG))
2.6B
1.031
5.69
0.2274
32.87
(1,7.5)
DI'-IN-SDXL
2.6B
1.048
5.66
0.2278
32.91
(10.7.0)
DIS-
2.6B
1,020
5.68
0.2278
32.82
(100,
DIS-IN-SDXL
2.6B
1.067
5.74
0.2304
32.82
(100, 7.5)
DI'- (BEST,
1.140
0.2331
32.75
(100,7.5)
Table 3: Quantitative evaluations of models HPSv2.1 scores. We compare open-sourced models regardless of their base
model and architecture. indicates our implementation.
MODEL
ANIMATION?
CONCEPT-ARTY
PAINTING1
PHOTOT
AVERAGET
COSTEP-SDXL-BAR PODELL PODELL ET AL..
2023
30.85
29.30
28.98
27.05
29.05
DUSTE--DDXL--PO WALLACE ET AL.
2024
32.01
30.75
30.70
28.24
30.42
28STER-SDD.5-LARGE
31.89
30.19
30.39
28.01
30.12
50STEE-FL.UX-DEV
32.09
30.44
31.17
29.09
30.70
ISTEP-DMD2-SDXI YIN
2024
29.72
27.96
27.64
26.55
27.97
ISTEr- SSTE--DHH-INSTRUCT SUXL
AL..
2024B
31.15
29.71
29.72
28.20
29.70
1STEP-SIM-SDXL LUO ET AL. 2024C
31.97
30.46
30.13
28.08
30.16
1 Luo 2024
31.19
29.88
29.61
28.21
29.72
LSTEP-DI
32.26
30.57
30.10
27.95
30.22
ISTDD-DI-SDXI..OORRR TRAINING)
33.22
31.67
31.25
28.62
31.19
1STE--DE-SDXLL-DDOOOOO LONGER TRAINING)
33.92
32.80
32.71
29.62
32.26
samples, they suffer from oversaturation issues, meaning that the images have over-saturated colors and lighting that might make users feel uncomfortable. On the contrary, the DIt-lstep model shows very gentle lighting and aesthetic colors, which align better with human preferences. We empirically conclude that DI prevents the model from collapsing into high-reward painting-like images, in which DI++ collapsed. Besides, the SDXL and SDXL-DPO are easy to generate painting-like images instead of photo-realistic images.
ically found that flaws commonly observed in other one- step models also appeared in Diff-Instruct*. For exam- plc, the one-step model sometimes generates inaccurate human faces and hands. We believe that these bad cases are caused by the limitations of the one-step generation ap- proach, which poses challenges for the model in generating correct content in single pass. We believe that general- izing Diff- Instructt to multi-step (few-step) models could help combine the advantages of both efficiency and gen- cration correctness. Our results suggest that consistently improving the one-step model architecture, scaling its pa- rameters. improving reference diffusions, and enhancing the reward models lead to better models.
6. Conclusion and Limitations
In this paper, we present Diff-Instructts novel approach for preference alignment of one-step text-to-image gener- ative models. We introduce two novel approaches to in- corporate traditional classifier-free guidance into alignment through the lens of explicit-implicit reward decoupling. Using DI*, we introduce the 2.6B DI*-SDXL--Iste model that outperforms current leading models such as the 12B FLUX-dey- 50step model with 1.88% inference cost.
Finally, we identify directions for further research for score post-training. For instance, while DJa needs pre- treitrained reward model, methods like DPO (Rafailov et al., 2024 Wallace al., 2024) have introduced aligning generative models directly using human feedback data. It would be valuable explore efficient score post-training for one-step generative models by directly incorporating feedback data and eliminating the pre-trained reward models.
Nonetheless, Diff-Innstucct" has its limitations. We empir-
8

---

One-step Model Beats Large Diffusion with Score Post-training
DIx-SDXL-1stte (2.6B)
FLUX-Dev-50step (12B)
SD3.5-large 28ste (8B)
Figure 2: Comparison on Parti prompts of our 2.6B DI*-SDXXI-Istep model against 12B LuL---dde--Osteee and 8B SD3.5- 12ste-28step diffusion models, Prompts used are listed the appendix.

---

One-step Model Beats Large Diffusion with Score Post-training
Acknowledgement
Couairon, G., Verbeek, J., Schwenk, H., and Cord, M. Diffedit: Diffusion-based semantic image editing with mask guidance. ArXiv, abs/2210. 11427, 2022.
We acknowledge constructive suggestions and feedback from reviewers and ACs/SACs/PCS of NeurIPS 2024. We thank the authors of DMD2 for their open-weight DMD2 models, which we use the base model before post- training.
Dai, X., Hou, J., Ma, C.-Y., Tsai, S., Wang, J., Wang, R., Zhang, P., Vandenhende, S., Wang. X., Dubey, A., al. Emu: Enhancing image generation models ing photogenic needles in haystack. arXiv preprint arXiv:2309. 15807, 2023.
Impact Statement
We propose Diff-Instruct* (DI*) that can effectively post- train one-step text-to-image generative models be aligned with human preferences. The method and corre- sponding aligned models lie in the domain of Reinforce- ment Learning using Human Feedback, which involves the participation of humans for training reward models. Though the Diff-Instruct* algorithm itself does not gener- ate harms to society, any misuse of the algorithm could po- tentially lead to misleading, fake, or biased generated con- tent. We conduct experiments whose resulting models are less likely to be misused. Fur- ther experiments are needed to better understand these con- sistency model limitations and propose solutions to address them
Deng, W., Luo, W., Tan, Y., Biloš, M., Chen, Y., Nevmy- vaka, Y., and Chen, R. T. Variational schr\," odinger dif- fusion models. arXiv preprint arXiv:2405.04790 2024.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high- resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
academic benchmarks,
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C.,, Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Re- inforcement learning for fine-tuning text-to-image diffu- sion models. Advances in Neural Information Process- ing Systems, 36, 2024.
Feng. Y.. Luo, W., Huang, Y., and Wang. T. A lipschitz ban- dits approach for continuous hyperparameter optimiza- 0339, 2023.
References
AI, S. Stable diffusion 3.5 large.
https:
tion. arXiv preprini
whhuggingface.co//ttaae
Geng, Z., Pokle, A., and Kolter, J. Z. One-step diffu- sion distillation via deep equilibrium models. In Thirty- seventh Conference Neural Information Processing Systems, 2023. URL https:://operrevees net/ forum?id=b6xvk
stable dtable-diffusionn3..333 large 2024.
Berthelot, D., Autef, A., Lin, J., Yap. D. A., Zhai, S., Hu, S., Zheng. D., Talbott, W., and Gu, E. Tract: Denoising diffusion models with transitive closure time-disilllation. 2023.
arXiv preprint arXiv:2303.0422
Geng, Z. Pokle, A., Luo, W., Lin, J. and Kolter, J. 7. Consistency models made easy. arXiv preprint 24l1::200.145446 2024.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. arXiv preprint arXiv: 230..33331.331,, 2023.
Goodfellow, L., Pouget-Abadic, J., Mirza, M., Xu, B., Warde- Farley, D.,, Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014.
Brock, A., Donahue, J., and Simonyan, K. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 11096, 2018
Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng. C., Wang, R., and Ramesh, A. Video generation models as world simulators. 2024.
Gu, J., Zhai, S., Zhang, Y., Liu, L., and Susskind, J. Boot: Data-free distillation of denoising diffusion models with bootstrapping. arXiv preprint arXiv:2306. 05544, 2023.
Heek, J., Hoogeboom, E., and Salimans, T. Multistep consistency models. arXiv preprint xxXiv:2403.06807, 2024.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg. S., and Amodei, D. Deep reinforcement learning from hu- man preferences. Advances in neural information pro- cessing systems, 30, 2017.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Clark, K., Vicol, P., Swersky, K., and Fleet, J. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.174 17400, 2023.
10

---

One-step Model Beats Large Diffusion with Score Post-training
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint wrXiv2220.112599 2022.
Lin, S., Wang. A., and Yang. X. Sdxl-lightning: Pro- gressive adversarial diffusion distillation. arXiv preprint arXiv:2402. 13929, 2024.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. Advances in Neural Information Pro- cessing Systems, 33:6840-683 6851, 2020.
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitmick, C. L.,, and Dollár, Microsoft coco: Common objects in con- text, 2014. URLhttp:: arriv. org aa///bbs/44000 0312 cite arxiv: 1405.03 1000mmeCmmtt 1) updated an- notation pipeline description and figures; 2) added new section describing datasets splits; 3) updated author list.
Ho, J.. Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. J. Video diffusion models. arXiv preprint
arXiv:2204.03455, 2022.
Hong, J., Paul, S., Lee, N., Rasul, K., Thorne, J., and Jeong, J. Margin-aware preference optimization for aligning diffusion models without reference. In First Work- shop on Scalable Optinization for Efficient and Adaptive Foundation Models, 2024.
Liu, X., Zhang, X., Ma. J., Peng, J., et al. Instaflow: One step is enough for high-quality diffusion- text-to- image generation. In The Twelfth International Confer- ence on Learning Representations, 2023.
Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to- image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Luo, S., Tan, Y., Huang, I.,, Li, J., and Zhao H. La- tent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint 23X1::2310.04378, 2023a.
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8110- 8119, 2020.
Luo, W. A comprehensive survey on knowledge distillation of diffusion models. arXiv preprint arXiv:2304. 04262, 2023.
Luo, W. Diff-instruct++: Training one-step text-to-image generator model to align with human preferences. arXiv preprint (tti::2110.1888)) 2024.
Kim, D., Lai, C.-.., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consistency trajectory models: Learning probabil- ity flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279 2023.
Luo, W. and Zhang, Z. Data prediction denoising models:
The pupil outdoes the master, 2024. URL https:// openreview net / forum ? www:didwwwwwwwucc
Luo, W., Jiang. H., Hu, T., Sun, J., Li, Z., and Zhang, Z. Training energy-based models with diffusion contrastive divergences. arXiv preprint arXiv:2307. 2307.0 0/0.0/668, 2023b.
Kim, H., Kim, S., and Yoon, S. Guided-tts: A diffusion model for text- text-to-speech via classifier guidance. In Inter- national Conference on Machine Learning, pp- 11119 11133. PMLR, 2022.
Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, Z. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems. 36. 2024a.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 2014.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J. and Levy, O. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.00156, 2023.
Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, Z. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024b.
Lab, B. F. Flux.
black- black-forest wow.storrttttee labs/ flux 2024.
Luo, W., Huang. Z., Geng, Z., Kolter, J. Z., and Qi, G.- J. One-step diffusion distillation through score implicit matching. arXiv preprint arXiv:2410. 16794, 2024c.
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., A.bbeel, P., Ghavamzadeh, M., and Gu, S. S. Align- ing text-to-image models using human feedback. arXiv preprint ar ArXiv:2302. 12/92, 2023.
Luo, W., Zhang, B., and Zhang, Z. Entropy-based training methods for scalable neural implicit samplers. Advances in Neural Information Processing Systems, 36, 2024d.
Li, J., Feng, W., Chen, W., and Wang, W. Y. Reward guided latent consistency distillation. arXiv preprint arXiv:2403. / 1027, 2024.
Meng, C., Song, Y., Song, J.. Wu, J., Zhu, Y., and Ermon, S. Sdedit: Image synthesis and editing with stochastic

---

One-step Model Beats Large Diffusion with Score Post-training
differential equations. arXiv preprini arXiv:2108.01073 2021.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 2022.
Meng, C., Gao, R., Kingma, D. P., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffusion models. arXiv preprint arXiv:2210.031-22 2022.
Ren, Y., Xia, X., Lu, Y., Zhang, J., Wu, J., Xie, P., Wang. X., and Xiao, X. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprint MoXivv2404.1 13686, 2024.
Nguyen, T. H. and Tran, A. Swiftbrush: One-step text-to- image diffusion model with variational score distillation. arXiv preprint arXiv:2312.05233 2023.
Salimans, T., Mensink, T., Heek, J., and Hoogehoom, E. Multistep distillation of diffusion models via moment matching. arXiv preprint arXiv:2406.0411
Nichol, A. and Dhariwal, P. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
2024.
Sauer, A., Karras, T., Laine, S., Geiger, A., and Aila, T. Stylegan-t: Unlocking the power of gans for fast large- scale text-to-image synthesis. International confer- ence on machine learning, pp. 30105-30118, PMLR, 2023a.
Oord, A. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 03499, 2016.
Sauer, A., Lorenz, D., Blattmann, A., and Rombach, Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042, 2023b.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 11-27730-27744 2022.
Schuhmann, C. Laion-aesthetics. https://laion
ai/blog/laion- http:/aaiioeeettttccsssoss/ 2022. Accessed: 2023 11- 10.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634/ 2017.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning, pp. 2256-2265. PMLR, 2015.
Pokle, A., Geng, Z., and Kolter, J. Z. Deep equilibrium approaches to diffusion models. Advances in Neural In- formation Processing Systems, 35:37975-37990, 2022.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Song, Y. and Dhariwal, P. Improved techniques for training consistency models. In The Twelfth International Con- ference on Learning Representations, 2024.
Prabhudesai, M., Goyal, A., Pathak, D., and Fragkiadaki, K. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
Song, Y., Sohl-Dicksted J., Kingma, D. P., Kumar, A.,
Ermon, S., and Poole, B. Score-based generative model- ing through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, Clark, J.. et al. Learning transferable visual models from natu- ral language supervision. In International conference machine learning, pp. 8748-8763, PMLR, 2021.
Song. Y., Dhariwal, P., Chen, M., and Sutskever, I. Consis- tency models. arXiv preprint arXiv:2303.01469 2023.
Song, Y., Sun, Z., and Yin, X. Sdxs: Real-time one-step latent diffusion models with image conditions. arXiv preprint arXiv:2403.16627 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36. 2024.
Tashiro, Y., Song, J., Song. Y., and Ermon, S. Csdi: Con- ditional score-basee diffusion models for probabilistic time series imputation. Advances in Neural Informa- tion Processing Systems (NeurIPS), 2021. 2021.
12

---

One-step Model Beats Large Diffusion with Score Post-training
Vincent, P. A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7): 1661-1674, 2011.
Yu, J., Xu, Y., Koh, J. Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A., Yang, Y., Ayan, B. K., et al. Scaling autoregressive models for content-rich text-to- image generation. arXiv preprint 2XX1:2206.11789, 2 (3):5, 2022.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8228 8238, 2024.
Zhang, B., Luo, W., and Zhang, 7. Enhancing adversar- ial robustness via score-based optimization. In Thirty- seventh Conference on Neural Information Process- ing Systems, 2023a. URL htt.httss//// /opp:///ev/eeeeeee.. net forum? www.ummmid---aaaaa MOAHXRZHthm
Wang, Y., Bai, W. Luo, W., Chen, W. and Sun, H. In- tegrating amortized inference with diffusion models for learning clean distribution from corrupted images. arXiv preprint arXiv:2407. 162 2024.
Zhang, B., Luo, W., and Zhang, Z. Purify++: Im- proving diffusion-puriie -urrifianptiiiiatiion with advanced diffusion models and control of randomness. arXiv preprint arXiv:2310.1876. 2023b.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image syn- thesis. arXiv preprint arXiv:2306.0933
Zheng, B. and Yang. T. Diffusion models are innate one- 2024.
step generators. arXiv preprint
2023.
Zheng, H., Nie, W., Vahdat, A.. Azizzadenesheli, K., and Anandkumar, A. Fast sampling of diffusion models via operator learning. arXiv preprint arXhv:2211.1342 2022.
Xie, S., Xiao, Z., Kingma, D. P., Hou. T. Wu, Y. N., Mur- phy, K. P., Salimans, T., Poole, B., and Gao, R. Em distillation for one-step diffusion models, arXiv preprint arXiv:2405.16852 2024.
Zheng, J., Hu, Fan, Z., Wang, C., Ding, C., Tao, D., and Cham, T.-J. Trajectory consistency distillation. arXiv preprint 1911::2402.19159, 2024.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
Zhou, M., Wang, Z., Zheng, H., and Huang, H. Long and short guidance in score identity distillation for one-step text-to-image generation. arXiv preprint arXiv:2406.01561, 2024a.
Xu, Y. Zhao, Y. Xiao, Z., and Hou, T. Ufogen: You for- ward once large scale text-to-image generation via diffu- sion gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8196- 8206, 2024.
Zhou, M., Zheng, H., Wang. Z., Yin, M., and Huang, H. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. arXiv preprint arXiv:2404. 04057, 2024b.
Xuc, S.. Yi, M., Luo, W., Zhang, S., Sun, J., Li, Z., and Ma, Z.M SA-solver: Stochastic adams solver for fast sampling of diffusion models. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023. URL httpp:://opereeee///.///
net rum??
id=f6a9XVFYIo
Yang, K., Tao, J., Lyu, J., Ge, C., Chen, J., Shen, W., Zhu, X., and Li, X. Using human feedback to fine-tune diffu- sion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024.
Yin, T. Gharbi, M., Zhang, R., Shechtman, E., Durand, F. Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. arXiv preprint arXiv:2311. 2023.
Yin, T., Gharbi, M., Park. T., Zhang, R., Shechtman, E., Durand, F., and Freeman, W. T. Improved distribution matching distillation for fast image synthesis. arXiv preprint 2005 14867, 2024.
13

---

One-step Model Beats Large Diffusion with Score Post-training
Pretrain Teacher
Collect Human Feedback Data
16
Diffusion Models
teacher
Use teacher, TA, and the reward to align 1-step student model. (Diff- Infff-nnttrcct)
Image-Captinn Data
teacher
step Distillation
Reward Modeling
Distill get 1-step Student Model (with liff-Instruct family or other distillation methods)
Train Human Reward Model with Human Feedback Data
TA
student
reward
student
Stage Post-training (Preference Alignment)
Stage Pre-training
Stage Reward Modeling
Figure 3: Workflow developing one-step text-to-image generative models using Diff-Insti Instrucct... The workflow includes stages pre-traine pre-training, reward modeling, and preference alignment.
A. Theoretical Results
A.1. Proof of Theorem 3
Proof. Recall that (-) is induced by the one-step model go()). therefore the sample obtained wo = go(zlc); z pa- The term contains parameter through 2o ge (=(cc), Ps. To demonstrate the parameter dependence, use the notation po (-). Note that (-) (-) is the reference distribution. The alignment objective writes
Lorig (8)
empp. (s
[- ar( c)] [u.21 Pref)))))
(A.1)
The first loss term of I ar(#0. c) is easy to compute by directly pushing the generated sample 2o and the text prompt e into the reward model r(,,),, However, the second loss term (A.2) is intractable because we do not explicitly know the relation between and (-)
f" Bmin) fa(spp.. (2xe) 8q. (are)) fal,
(po.Pes) = 10 Pres)
(A.2)
===
We turn to derive the equivalent loss for DIO.T] Pres)) First we take the gradient of A.2p. show
Lr w())
D [. (po. Pref) = 00
(d(spa. (xx) - sqc (xx1) j
(A.3)
00
r=
T. 0 = Er E (otemmuut(t/t u (((v)))
00 Spo. (x+)
(A.4)
Notice that p.. is induced by first generating samples with one-step one-step model then adding noise with diffusion process (2.1). we do not know the term 36 Sp. (æx). Therefore the gradient formula (A.4) is intractable. However, we will show that a tractable loss function can recover the intractable gradient (A.4). and therefore can be used for minimizing
A.2 Our proof is inspired by the theory from Vincent (2011), Zhou et al. (0024)) and Luo et al. (2024c).
Luo
We first present so-called Score-projection identity (Theorem A. which has been studied in Zhou et al. 2024b) and Vincent (201T):
Theorem A.1. Let (-) be a 0-free vector-valued function under mild conditions, the identity holds:
u(xx)) 8po. (x.: V æe log q (x) O, VO.
(A.5)
Mn [eg (ell
We give short proof of Theorem A.I as clarification. Readers can also refer to Vincent 2011] or Zhou et al.
(024bb as
a reference.
14

---

One-step Model Beats Large Diffusion with Score Post-training
Di--SDXL1sttep (Longer Training)
SIM RM--SDXL-- 1step
Diff-instruct++ -ttpinsrrrtt---------------
-DXL-1sttep
Diff-instruct-SDue SDXL 1step
DI--SDXL-1stem
DMD2- MMD2--DXL--O -ittppsttepiiit Model)
SDXL 50step
SDXL- D-D---DOO-S------ 50step
Prompt: style WN--SDXL-1step (Longer Training)
style of glass, light pink, solarpunk, vivler,
SIM SDX-SDX 1step
Diff-Instruct++ +-DXX-11stes
Piff-Insstuct-SDXX -1step
1stepp(ini Model)
50step
SDXL-DPO-BBE 50stap
Prompt: A ejpeсs made of colorful building blooks, cyberpunk
Dn--SDXL-1step (Langer Training)
Diff-Instruct++ -step RMM-SXXL-20 SSxt-
InstructSSDXL-Lsee
DI-SDDXL-sstep
DMD2-SDXI -stt--SDX--sste((nit Model)
SDXL-500 -5xtep
SOXL-DPO--0step
Prompt: armchair
Figure 4: Ablation comparisons of 1step models and other SDXXL-based models in Table 2
15

---

One-step Model Beats Large Diffusion with Score Post-training
Recall the relation between &po, (x,) and V log ye (2)),
know
8ps,1 (x:)
Va, log po.0 (2
J po. (zo) V a, qi ( z )dzo
(0,)
V m, log qt (3 (oo))o (200 ) (272 dxo
po.
We have
200P6,00 u(xx,)" "' 8po. (2t) = Eximpo, (xx))" 8po (241) (0 uag)
po.r(2e))ua(i " 8po (20+)dac
[ (o.c(x)))((00)) / Van log qe (2,, (2o (2+ (x0) dxgda
(x,)(
I u(a=)? |Va, Va, log qe (2) (ate darodar
u(xx)) VV, log qe (ox 2o polooopp...o (a) dx
u(xx) " Vay log lr (ot (zp)
Ro0-r (mng)
If we take the 0 gradient on both sides of (A.5). we have
Ox
0 = E
u(xx))
( (spo.. (xx) - Va, log 2e (2x moo}
10:000-0000 -0000- lmg)
00
Oac
uxx))²
T
V m.
log 9: (=1//20))
+ Bieerpu, u(xx)²
Spu, (2ee
(A.6)
30
00
So we have an identity
Em--po u(xx))
ppe (xt)
E
momPo.o
u((x)) (u(az))8prel1) (2:)) Va, log ge (20, (200
00
30
Mo (mg)
Notice that the left-hand side of equation (A.7) can be interpreted as the gradient the loss function when the parameter dependency of the sampling distribution is cut off, i.c
Ex po. u (2x))
00 spa, (2x))
Bree pagioa
8pu, (3x))
(A.7)
00
Therefore we have the final equation
u (xx)))) spu. (1:2)
i
uu(ars) (8puntos, (xre) - Va, logg((r)(ao))) }
F P
IF
#0mPp.0-
(A.8)
08
00
welwo (dre) isg)
which holds for arbitrary function ul-) and parameter If we set
u(x:) = d'(y)) yt = Spur1o), (xxs) Sq. (xx.)
Then we formally have
0
(d((vi) T
E. P
Spm. (axs)
00
d'(xr))
Spa,e (x0) (p0)
Vzz log qe (20)
(A.9)
00
=y m (aey (mm)
16

---

One-step Model Beats Large Diffusion with Score Post-training
This means that the 0 gradient of tractable loss:
ce(t){ - dl' (wve) )"{()))() (po. (x:) Var, log ai (at (20) }
E
(A.10)
to replace the wanted gradient A.3 which can minimize the regularization loss A.2
Combining r(zo, e) and A. we have the practical loss
(D)
=I
s-ps #0
-rr(xoo c)
(A.11)
~e 2e (mg) w(t) { -off-ddin d((ov)) }
E
t,y ag
Speatel. (x1) Va, log qu (a, (zo) j
Remark A.2. In practice, most commonly used forward diffusion processes can be expressed as a form of scale and noise addition:
a(()=z 3(1e,, e~ (e,
(A.12)
So the term #t in equation (A.11) can be instantiated as 2 ~ Pz, e Wee I), a. =(1)x0 + B(1)e.
A.2. Proof of Theorem 3.2
Proof. Recall the definition of the classifier-free reward (36). The negative reward writes
Pre/ (ee/(w Pref (t)
-r(wo: c) -E,,, æ. pp. w(t) log
This reward will put a higher reward on those samples that have higher class-conditional probability than unconditional probability, therefore encouraging class-conditional sampling. is clear that
Ø 00
r(xo, c)) Es, xpo w())
Ox:
Va, log Pref Pref( Va, log Pre log (x)(t)
00
0x:
-- ap wt))
8 (rgg(s// c) (rer
x 00
(A.13)
Therefore, we can see that the equivalent loss
T Lesg = E mo (tt) Spes (sg(x,]ltt, c) - Sres (sgl2, 0)
x,
(A.14)
recovers the gradient formula A.13
B. Additional Discussions
B.1. Meanings of Hyper-paramettrr
As in Algorithm 1. the overall algorithms consist of two alternative updating steps. The first step to update vi of assistant diffusion model by fine-tuning it with student-generated data. Therefore the assistant diffusion sy (x, c) can approximate the score function of student one-step model distribution. This step means that the assistant diffusion needs to communicate with the student to know the student's status. The second step updates the one-step by minimizing the tractable loss (C.T) using SGD-based optimization algorithms such Adam (Kingma & Ba 2014). This step means that the teacher and the assistant diffusion discuss and incorporate the student's interests to instruct the student one-step model.
Meanings of Hyper-parameters
As we can see in Algorithm I (as well as Algorithm 2 Each hyperparameter has its intuitive meaning. The reward scale parameter Orew controls the strength of human preference alignment. The larger the arew is, the stronger the one-step
17

---

One-step Model Beats Large Diffusion with Score Post-training
model is aligned with human preferences. However, the drawback for large Grew might be the loss of diversity and reality. Besides, we empirically find that larger Orre leads richer generation details and better generation layouts. But very large Crew results in unrealistic and painting-like images.
The CFG reward scale controls the strength of using CFG rewards when training. We empirically find that the best CFG scale for Diff-Instruct is the same as the best CFG scale for sampling from the reference diffusion model. However, Ocef may conflicts with Orew the Stable Diffusion 1.5 experiment, we find that using a large CFG reward scale leads to worse human preferences. Therefore, the proper combination of (Arew: Oefs) asks for careful tuning.
The diffusion model weighting X(t) and the one-step model loss weighting w(t) controls the strengths put on each time level of updating assistant diffusion and the student generator. We empirically find that it is decent to set X(t) to be the same as the default training weighting function for the reference diffusion. And it is decent to set the w=(() = for all time levels in practice. In the following section, we give more discussions on Diff-Instruct"
B.2. Discussions on Diff-Instruct*
Flexible Choices of Divergences. Various choices of distance function d(.) result in different training algorithms. In this part, we discuss two instances. The first choice distance function is simple squared distance, i.c. d(y:) l The corresponding derivative term writes d' (y:) = 2yr. In fact, such a distance function recovers the practical diffusion distillation loss studied in Zhou et al. 2024b a). The second distance is the pseudo- distance, which shows more robust performances than the simple squared distance. The pseudo Huber distance is defined with d(y) Vlg + c²- where is a pre-defined positive constant. The corresponding regularization loss (3.4) writes
T
D (po. Pref)
y
(p((t,)) Va, log (x/ }.
(B.1)
v
+
Here yt == S (2++) - -.q (axt).
DI* Does Not Need Image Data for Training. One appealing advantage of DI# the image data-free property. which means that DI* requires neither the image datasets nor synthetic images that are generated by reference diffusion models. This strength distinguishes DI* from previous fine-tuning methods such as generative adversarial training (Goodfellow et al. 2014) which require training additional neural classifiers over image data, well those fine-tuning methods over large-scale synthetic or curated datasets.
C. Qualitative Results
Evaluation Details To quantitatively evaluate the performances of the one-step model trained with different alignment settings, we compare our generators elaborately with other 1024x 1024 resolution open-sourced models that are either based SDXL diffusion models larger models such as FLUX-dev and SD 3.5 large. For all models, compute four standard scores: the Image Reward (Xu et al. 2023). the Aesthetic Score (Schuhmann) 2022), the PickScore(Kirstain let al. (2023), and the CLIP
et al. 2021). Since most existing literature tests the human preference scores with different prompts which are possibly not available, to make a fair comparison, in our experiment, we use 30k prompts from the COCO-2014 (Lin et al., 2014) validation dataset and intensively evaluate a wide range of existing open open-sourced models. All models are tested with the same prompts and the same computing devices.
Besides the COCO prompts, also evaluate one-step models with open-sourced models with Human Preference Score v2.0 (HPSv2.0) (Wu et al.] (023) over their benchmark prompts. The HPS is widely used standard benchmark that evaluates models' capability of generating images of 4 styles: Animation, concept art, Painting, and Photo. The score reflects the prompt following and the human preference strength of text-to-image models. We use the HPSv2's T default protocols for evaluations.
18

---

One-step Model Beats Large Diffusion with Score Post-training
Algorithm 2: Diff-Instruct*.
Input:
prompt dataset C, one-step model ge (002 c), prior distribution Ps. reward model r(æ, c). reward model scale Orews CFG reward scale Gcfy- reference diffusion model Sref (æ; (c, c), assistant diffusion (x, forward diffusion pr (r, (o) (2.1) assistant diffusion updates rounds KTA, time distribution π (l), diffusion
model weighting ((t), student loss time weighting w(t).
while not converge do freeze 0, update vb for KTA rounds using SGD by minimizing
(
E
(war()) mg ** m
Var (2,
freeze vA update 0 using SGD by minimizing loss
(0) =E onc, aupps =0-u0(s.c)
Orrea rr(20, c) IE
tma(())
welmo-velielen
wol)) w()))d((sw(zaall c) B (2)) & (2oel, - Van log (x)
+ aefg (sgg/e c) seef (sghx,lt Ø)}² xt
(C.1)
end
return 0, tb.
C.1. Pytorch style pseudo-code of Score Implicit Matching
In this section, we provide PyTorch-style pseudo-code for the algorithm below.
import torch
import torch.m. nn as nn import torch.eptim as optim import copy
use_efg True reward True
Initialize generator G Generator
## load teacher DM
Drf DiffusicnModel1 () (o to_ so__ckpt)> .eval www.uires_grrrr.e... serddrees_rrd______ (Fai_(Faise) (False)
pta copy deepcepy (Drf) II initialize online DM with teacher DM
RexardModel () use_reward else None
Define optimizers opt_G = optim.Adameee
Adam (G. paramttetrss 1r=0.000 0001, betas= 0.999) Sta__ta optim Adam (Dta (arameters 1r=0.001, betas= (0.0, 0. 999)
Training loop
while True:
# update Dta
uta.train () requires_grad (True)
() reauirrssrrssggr____
## update assistant diffusion
prompt (orcch'ppromp'"
torch. (1024, 4, 64, 64), device=G.device) with torch.no_grad fakc_x0 = (2, prompt)
sigma terch. exp O*torch. randn [1,1,1,11, serrice-fakexx..dee 2.0)
noise randn like (fake_x0)
19

---

One-step Model Beats Large Diffusion with Score Post-training
fake_xt fake_x0 sigma+ se
pred_x0 Dta (fake_ext, sigma, prompt)
weight dinfutiioi weiitiiioiisiiiiiiiihtt weight
(sigma)
batch_loss lcss weight * (pred_ x0 fake fake_x0) xx2
batch_loss batch_loss. sum ([1, 2, 3]) mean ()
optimizer__ta..exr___
batch_ (
aptimizer_Dta step
# update
Dta.eval () reeuuieeseegerr.... (False)
G.train() www.uiies_grereeeeee grad_ (True)
prompt batchh'prompt'
torch. (1024, 4, 64, 64), devico=G.devloo) fake_x0 ( prompt)
sigma terch. (2. 12t.:/.0000.....e.... randn ([1,1,1,2] device=fake_xt ) 2.00 noise torch. like (fake_x0 x0)
fake iake_ sigma+noise
with torch. ne_grad if use_cfg:
efg_vector (frkeeet__ sigma, prompt) Drf (fake_xt, sigma, None)
else:
cfg_vector None
x0_r_ (fake_xt sigma, prompt) prod_x0_ta Dta (fake_xt,, sigma, prompt)
denoise_diff = pred_x0_ta pred_x0_rf adp_wgt terch. sqrt (denoise_dis weight compute___weight weeightt (sigma, denoise_diff))
sum ([1, keepdims-True) phuber_c**2
compute score regularization loss batch_ lcss weight denoise diff (fake D D_yn) /adp_we
compute explicit reward loss if needed if use_reward:
reward_loss loss = -reward_scalll r (fake_x0, prompt)
batch_loss += reward loss
compute cfg reward loss needed if use_cfg:
cfg_reward_loss = efg_scale + efg_vector fake_x0
bbatch_loss +- cfq_reward_less 1oss
batch_loss- batch_ Bescc__osss.eee sum ([1, 31) .eea ()
optimizer__..o. Beaizzr_G..zee zero grad ()
batch lasswaack
optimizer_G. ()
Listing 1: Pytorch Style Pseudo-code of D
20


Reviews:
[{"review_id": "7QZrQVL9XO", "reviewer": "Reviewer_KEDK", "summary": "The paper proposes Diff-Instruct* (DI*), a new post-training framework for one-step text-to-image generative models, aiming to align their outputs with human preferences without requiring image data. The method leverages score-based reinforcement learning from human feedback (RLHF), optimizing a human reward function while maintaining closeness to a reference diffusion model. Instead of conventional KL divergence for regularization, the authors introduce a score-based divergence, which is theoretically justified and empirically shown to improve model performance. The authors trained a one-step generator trained via Diff-Instruct*, called DI*-SDXL-1step model, capable of generating 1024×1024 images in a single step. It outperforms the 12B FLUX-dev model on key human preference metrics while running much faster.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we are delighted that you like our score-based reinforcement learning post-training of the one-step diffusion model. We appreciate your valuable suggestions. In the following paragraphs, we will address your concerns one by one.\n\n**Q1**. Clarify the differences between Diff-Instruct-star, DMD, and other approaches.\n\n**A1**. We are sorry for the confusion. In one-step diffusion distillation/post-training literature, there are typically two types of training criteria. \n\n+ The first type is trajectory-based methods, such as Progressive distillation, consistency distillation, and other variants. \n\n+ The second type is **distribution matching based methods**, which aim to minimize some probability divergence between one-step model distribution and teacher diffusion model distribution. Among them, **Diff-Instruct** first proposes one-step diffusion distillation by minimizing the integral KL divergence. **DMD** and **DMD2** generalize the KL divergence minimization by incorporating a regression loss and a GAN loss. **SiD**[4] studies the minimize the one-step distillation by minimizing Fisher divergence instead of KL. Later, **SIM** unifies the one-step diffusion distillation by minimizing general score-based divergences (including Fisher divergence) with detailed theories that prove the gradient equivalence between SIM loss and the underlying divergence. \n\n+ **Post-training**: The **reward-guided LCM**  first combines a reward model with consistency loss to train few-step diffusion models. **DI++** then studied the post-training of the one-step model by introducing a KL divergence-PPO framework and achieved a very strong human preference quality. Our DIstar is inspired by DI++, by differs in (1) theoretical and empirical study the post-training via score-based-PPO instead of KL-based PPO; (2) introducing novel techniques that successfully scales the post-training of one-step diffusion model at 1024x1024 resolution, and outperforms current SoTA diffusion model (FLUX-dev) with only 1-step (1.8\\%) inference costs. This surprisingly strong performance also gives us the inspiration for our title **Small one-step model beats large diffusion with score-based post-training**. \n\nIn conclusion, DIstar follows the Occam's Razor principle: achieving better human-preference quality than open-sourced SoTA FLUX-dev with minimal losses (reward and general score-divergence). \n\n**Q2**. Discussions on why the score-based divergence is potentially better in diversity-preserving than KL.\n\n**A2**. We appreciate your good question! We will answer your question from both theoretical and empirical perspectives. \n\n**Theoretical perspective**: If we use $p_ r$ to represent reference distribution and $p_{g}$ the one-step generator distribution. The KL divergence between $p_ g$ and $p_ r$ is defined as:\n$D_ {KL}(p_ g||p_ r) = \\mathbb{E}_ {x\\sim p_g}\\log \\frac{p_ g (x)}{p_ r (x)}$\nwhile the general score-based divergence with a form $\\mathcal{D}_ {SD}(p_ g,p_ d) = \\mathbb{E}_ {x\\sim \\pi} d(\\nabla_ x \\log p_ g (x) - \\nabla_ x \\log p_ d (x))$.\nKL divergence is long believed to be bothered by mode-collapse issues because the likelihood ratio $\\frac{p_ g (x)}{p_ r (x)}$ will be ill-defined if the  $p_ g$ and $p_ d$ have misaligned density support. However, the score-based divergence does not have such a \"ratio\" that will be ill-posed and, therefore, is safe in the case when $p_ g$ and $p_ r$ have misaligned support. Besides, the distance function $d()$ of general score-based divergence also brings flexibility in defining divergences to properly measure the difference between two distributions in a robust magnitude. For instance, our used Pseudo-Huber distance (line 259)  has a self-normalization form, which also helps stabilize the gradient of score-divergence to a unit scale, potentially further helping address the mode-collapse issue.\n\n**Empirical perspective**: We conduct a new experiment on one-step diffusion distillation on the CIFAR10 unconditional. The FID and Recall of KL and score-divergence model are (3.70,0.52) and (2.01,0.60). This shows that distilling using KL divergence leads to both worse FID and Recall. \n\n**Q3**. It would be good if the authors could present more qualitative examples comparing to KL-based models, such as DI++-SDXL, to verify the motivation of the method.\n\n**A3**. In **Fig. 4** in the appendix of the paper, we qualitatively compare the DIstar model with DI++, SIM, Diff-Instruct, DMD2, SDXL, and SDXL-DPO models. We clearly find that images generated by the DIstar model show better aesthetic quality, with gentle lights and colors. We also put more qualitative comparisons between DIstar-SDXL-DPO and DMD-SDXL-DPO in this anonymous link: https://anonymous.4open.science/r/distar_anonymous-E007/distar_dmd.png.\n\n**We appreciate your valuable review. We sincerely hope our response has resolved your concerns. If you still have any concerns, please let us know.**", "claims_and_evidence": "The paper claims that score-based divergence regularization is superior to KL-divergence for RLHF in generative models, which is supported by the ablation studies shown in Table 2. However, in Lines 375–378, the authors refer to Figure 1 for visualizations of DI++ with KL divergence to support their claim that DI++ tends to collapse into painting-like images with oversaturated colors and lighting, which lack diversity despite high rewards. However, Figure 1 only displays examples of DI*.", "methods_and_evaluation_criteria": "The proposed method is reasonable, and the evaluation experiments on SDXL using the Parti prompt benchmark, the COCO-2014-30K benchmark, and Human preference scores are also well-justified.", "theoretical_claims": "The paper introduces a tractable reformulation of score-based divergence for RLHF. The derivation is plausible and well-structured, but some steps could benefit from more intuition from a theoretical perspective, such as why the score-based divergence has a better diversity-preserving property than KL.", "experimental_designs_or_analyses": "Ablation studies are comprehensive and quantitative results are strong and convincing, with DI* showing consistent improvements across multiple reward configurations. It would be better if the authors could present more qualitative examples comparing to KL-based models, such as DI++-SDXL, to verify the motivation of the method.", "relation_to_broader_scientific_literature": "The paper falls within the area of diffusion distillation and RLHF for diffusion models. The related works are generally well-cited.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- The proposed method demonstrates performance improvements over competing approaches across diverse benchmarks, supported by both quantitative and qualitative evaluations.\n\n- The paper provides thorough ablation studies, which effectively isolate and quantify the contribution of each component to the observed performance gains, offering strong empirical evidence for the method's design.\n\nWeaknesses:\n\n- The application of score-based distribution matching, while effective, lacks significant novelty, as it has been previously explored in papers like DMD.\n\n- While quantitative results suggest the superiority of score-based PPO over KL-based methods, the qualitative analysis requires expansion. Specifically, more illustrative examples are needed to substantiate the claim of improved density preservation.", "other_comments_or_suggestions": "See Claims And Evidence.", "questions_for_authors": "See Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H7CriabM86", "reviewer": "Reviewer_NM9A", "summary": "This paper proposes Diff-Instruct*, a post-training method to align one-step text-to-image generative models with human preferences. The work is an evolution of “Diff-Instruct” and “Diff-Instruct++,” replacing the KL-based divergence (as in standard PPO) to a score-based divergence for regularization. Empirical results show superior performance compared to KL-based method.\n\n## Update after rebuttal\nThe authors' rebuttal has addressed my concerns regarding the motivation for replacing KL-divergence and the necessity of aligning a distilled one-step diffusion model. I am raising my score to weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we are glad that you like our novelty of regularizing the one-step diffusion model during post-training. In the following paragraphs, we will answer your questions one by one.\n\n**Q1**. Experimental or theoretical evidence that shows mode-collapse issues of one-step diffusion distilling using KL-divergences. \n\n**A1**. We appreciate your good question! We will answer your question from both theoretical and empirical perspectives. \n\n**Theoretical perspective**: If we use $p_ r$ to represent reference distribution and $p_{g}$ the one-step generator distribution. The KL divergence between $p_ g$ and $p_ r$ is defined as:\n$D_ {KL}(p_ g||p_ r) = \\mathbb{E}_ {x\\sim p_g}\\log \\frac{p_ g (x)}{p_ r (x)}$\nwhile the general score-based divergence with a form $\\mathcal{D}_ {SD}(p_ g,p_ d) = \\mathbb{E}_ {x\\sim \\pi} d(\\nabla_ x \\log p_ g (x) - \\nabla_ x \\log p_ d (x))$.\nKL divergence is long believed to be bothered by mode-collapse issues because the likelihood ratio $\\frac{p_ g (x)}{p_ r (x)}$ will be ill-defined if the  $p_ g$ and $p_ d$ have misaligned density support. However, the score-based divergence does not have such a \"ratio\" that will be ill-posed and, therefore, is safe in the case when $p_ g$ and $p_ r$ have misaligned support. Besides, the distance function $d()$ of general score-based divergence also brings flexibility in defining divergences to properly measure the difference between two distributions in a robust magnitude. For instance, our used Pseudo-Huber distance (line 259)  has a self-normalization form, which also helps stabilize the gradient of score-divergence to a unit scale, potentially further helping address the mode-collapse issue.\n\n**Empirical perspective**: We conduct a new experiment on one-step diffusion distillation on the CIFAR10 unconditional. The FID and Recall of KL and score-divergence model are (3.70,0.52) and (2.01,0.60). This shows that distilling using KL divergence leads to both worse FID and Recall. \n\n**Q2**. Is aligning a distilled one-step diffusion model more effective than distilling from a pre-aligned multi-step diffusion model? If so or not so, why or why not? \n\n**A2**. This is a very good question! We will answer via three perspectives: \n\n+ First of all, distillation takes far more than compute than post-training: roughly 8:1 GPU hours. Therefore, in this paper, we focus on post-training that improves existing models with minimal costs. \n\n+ Second, **we appreciate your good intuition** and find that if we **replace the naive reference diffusion with an aligned diffusion** (like SDXL-DPO) in DI-star, we can even accelerate the post-training process with better final results. In the rebuttal phase, **we conduct a new experiment** (in **Table 2**) that compares DI-star w/o pre-aligned reference diffusion, as well as other DMD2-like models. \n\n**Experiment Setup.** We replace SDXL with SDXL-DPO as the reference diffusion model. Then, we compare models after post-training using DI-star, DMD, and DI++. \n\nAs **Table 2** shows, **in DI-star (Score-based PPO), both reward models and pre-aligned reference diffusion contribute to the performance improvements of one-step models in post-training**. Besides, we find that DI++(KL-PPO) will harm the contribution of the reward model and pre-aligned diffusion.\n\n**Table 2.** Quantitative comparisons of Preference Scores after post-training using SDXL-DPO on **Parti prompts (left part)** and **HPSv2.1 score (right part)**. \n| Model                               | Steps | Type | Params | Image Reward | Aes Score | Pick Score | CLIP Score | HPSv2.0 | HPSv2.1 |\n|--|--|--|--|--|--|--|--|--|--|\n| SDXL-DPO     | 50    | UNet  | 2.6B   | 1.102     | 5.77      | 0.2290      | 33.03      | | 30.42 |\n| DIstar-SDXL-DPO    | 1    | UNet  | 2.6B   | 1.160     |  5.84      | 0.2324      | 32.85      | 28.53 | 31.39 |\n| SIM-SDXL-DPO     | 1    | UNet  | 2.6B   | 1.063    |  5.79      | 0.2270      | 32.83      | 28.42 | 30.49|\n| DI++-SDXL-DPO     | 1    | UNet  | 2.6B   | 0.897     |  5.41      | 0.2225      | 33.06     | 27.86 | 28.07 |\n| DMD-SDXL-DPO     | 1    | UNet  | 2.6B   | 0.974     |  5.57      | 0.2244      | 33.07      | 28.15 | 29.29 |\n| DIstar-SDXL-DPO (Long, Best)    | 1    | UNet  | 2.6B   |  **1.210**    |   **5.90**     |   **0.2342**    |   32.91    | **28.81** | **32.25** |\n\n**Q3**. More clarifications of the trade-offs between the three loss signals\n\n**A3**. \n\n+ We find that CFG-implicit reward easily causes over-saturated colors. On the contrary, CFG-enhanced reference diffusion will not cause issues like over-saturated colors. \n\n+ ImageReward is conflict to CFG-rewards and CFG-enhanced divergences. However, PickScore is not conflicted with CFG-enhanced divergence loss, which makes both losses improve the one-step model during post-training.\n\n**We appreciate your valuable review. We sincerely hope our response has resolved your concerns. If you still have any concerns, please let us know.**", "claims_and_evidence": "The claim of KL-divergence being \"notorious for mode-seeking behavior\" is not supported by solid experimental or theoretical evidence.\nWhile building upon Diff-Instruct++, there could be more insights into why aligning a distilled one-step diffusion model is more effective than distilling from a pre-aligned multi-step diffusion model. Further clarification is needed to understand whether aligning a one-step model is easier and offers better performance.", "methods_and_evaluation_criteria": "The idea of using score-based divergence to regularize one-step diffusion model is novel. The design choices and discussion of incorporating CFG are interesting. The derivation  of the proposed method is reasonable and clear. \n\nThe evaluation criteria appear appropriate.", "theoretical_claims": "The theoretical derivation seems plausible, but more insight into the core motivation (why score-based divergence is better than KL-divergence) would strengthen the work.\n\nMinor issues: The paper should be more self-contained, explicitly stating new conclusions (e.g., “As first pointed out by Luo (2024), classifier-free guidance is related to an implicit reward function”) for readers less familiar with the background.", "experimental_designs_or_analyses": "1. The comparison with SDXL-DPO is not entirely fair, particularly regarding inference time improvements. The evaluation should also compare with a DMD-distilled version of SDXL-DPO.\n\n2. The ablation study requires further detail; e.g., clarify the trade-offs between the three loss signals (main human reward, CFG reward, and regularization loss).", "relation_to_broader_scientific_literature": "The paper is an evolution of “Diff-Instruct++” (Luo 2024). The paper also builds upon one-step diffusion models like DMD-v2 (Yin et al., 2024) and draw inspiration from score-based divergences from 1-step diffusion distillation (Luo et al., 2024c; Zhou et al., 2024b).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths\n\nThe idea of using score-based divergence to regularize one-step diffusion model is novel. The design choices and discussion of incorporating CFG are interesting. The derivation  of the proposed method is reasonable and clear. Empirical results show superior performance compared to KL-based method.\n\n\nWeaknesses\n\nThe paper could better study the trade-off between the three loss signals (human reward, CFG reward, and regularization). For instance, if the optimization directions of these components conflict, what is the impact on text-image alignment, human preference, and image quality?", "other_comments_or_suggestions": "1. In Fig. 3 (supplementary), please clarify what “TA” stands for.\n2. In Algorithm 1, although the notation “sg” appears earlier, it should still be clearly defined so that the algorithm is fully self-contained.", "questions_for_authors": "1. Why is aligning a distilled one-step diffusion model better than distilling from a pre-aligned multi-step diffusion model? Is aligning a one-step model inherently easier, and what evidence or insight supports this claim?\n\n2. If the optimization directions of the three loss signals (human reward, CFG reward, and regularization) conflict, what is the impact on text-image alignment, human preference, and image quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ZpetTVjxW", "reviewer": "Reviewer_zH6y", "summary": "The paper presents a novel online method for alignment fine-tuning of a one-step diffusion-based text-to-image generation model.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we are delighted that you like the novelty of DIstar for the one-step diffusion model post-training. We appreciate your valuable suggestions. In the following paragraphs, we will address your concerns one by one.\n\n**Q1**. Clarifications on image-data-free property and post-training setups.\n\n**A1**. We will polish the setup as image-data free and post-training of one-step diffusion models.\n\n**Q2**. (1) Broader evaluations, such as evaluations on challenging prompts like Parti and HPSv-benchs. (2) How does DIstar perform on models with the ability to take complex prompts, such as Pixart-alpha models? (3) Explorations on other rewards. \n\n**A2**. Thanks for your useful feedback. **In Table 1 and Table 2** in our paper, we have compared DIstar with other models on Parti prompts and HPSv2.1 benchmark. In the rebuttal period, we conduct two new experiments: **(1)** DIstar with SDXL-DPO (a pre-aligned SDXL model) as the reference diffusion; **(2)** DIstars with Pixel-art-alpha model that uses ImageReward as explicit reward models. We present more quantitative results in **Table 2** (in rebuttal to **Reviewer NM9A**), and **Table 1**.\n\n**Table 1.** Quantitative comparisons of one-step SD1.5 model and one-step Pixel-art-alpha (**PAA**) models and others in Preference Score on **COCO-2017-validation** prompt dataset.\n| Model                               | Steps | Type | Params | **Image Reward** | **Aes Score** | **Pick Score** | **CLIP Score** | **HPSv2.0** |\n|--|--|--|--|--|--|--|--|--|\n| PAA-512                     | 25    | DiT  | 0.6B   | 0.82         | 6.01      | **0.227**      | **31.20**      | 28.25|\n| PAA-DI++      | 1     | DiT  | 0.6B   | 1.24     | 6.19      | 0.225      | 30.80      | 28.48 |\n| **PAA-DIstar**      | 1     | DiT  | 0.6B   | **1.31**     | **6.30**      | 0.225      | 30.84      | **28.70** |\n\nThe Pixel-art-alpha (PAA) diffusion model uses the T5 text encoder, which has a much longer context length. Besides, we also explore the use of ImageReward to see if DIstar is consistent across different reward models. So, we compare DIstar on the PAA model with other open-sourced models. As **Table 1** shows, the PAA-DIstar model achieves a leading ImageReward, Aesthetic score, and HPSv2.0 score. It goes on par with the best PickScore as PAA-diffusion-25step shows.\n\n**Q3**. Discussion on the mismatch setups introduced in MaPO.\n\n**A3**. Thanks for the valuable comment. After a careful reading of the MaPO paper, we find it technically very interesting. MaPO introduced a novel approach to post-train diffusion models in cases of preference mismatch. We acknowledge that in our paper, we did not consider cases of preference mismatches, which is potentially important to study in our future. However, the DIstar and MaPO have essentially different targets in \n\n(1) MaPO targets in diffusion models, while DIstar targets in post-training of one-step models;\n\n(2) MaPO assumes access to the preference dataset, while DIstar assumes we have high-quality and diverse reward models;\n\nHowever, we really appreciate the solid contributions of MaPO. We will add an additional discussion on MaPO in our revision. We are glad to explore DIstar in cases of preference mismatches introduced by MaPO. \n\n**Q4**. Discuss DIstar with other related approaches. \n\n**A4**. Please see **A1** in rebuttal to **Reviewer KEDK**.\n\n**Q5**. Are there any criteria the assistant model needs to satisfy?\n\n**A5**. In general, arbitrary diffusion models can serve as an assistant model. But in practice, we initialize the assistant diffusion model with the preference diffusion model and find it works well. \n\n**Q6**. Clarify more on three losses. \n\n**A6**. Please see **A3** in the rebuttal to **Reviewer NM9A**.\n\n**Q7**. What are the memory requirements of this method? Is DIstar compatible with LoRA?\n\n**A7**. We acknowledge that DIstar needs an additional assistant diffusion model, which brings more memory costs. However, in practice, we find that the additional cost is acceptable if we use Pytorch techniques like BF16 training, Distributed data-parallel (DDP), and gradient accumulations. We did not use LoRA in order to get the best performances.\n\n**Q8**. Criteria for the assistant model. DIstar for few-step models? \n\n**A8**.  In general, arbitrary diffusion models can serve as an assistant model. DIstar for a few steps, possibly in our future work.\n\n\n**Q9**. Other comments on paper writing.\n\n**A9**. We will remove the blue colors. $p_\\theta$ represents the distribution of the one-step model. $d'(y)$ in equation 3.4 represents the derivative of distance $d(y)$. We appreciate your valuable comments on the paper writing, and we will revise the paper writing in the revision. **We will add experiment details in our revision.**\n\n**We appreciate your valuable review. We sincerely hope our response has resolved your concerns. If you still have any concerns, please let us know.**", "claims_and_evidence": "* The paper claims to require no data in the abstract. But in reality, it does use a dataset of prompts. I think this is misleading.\n* The paper abstract seems to suggest the paper trains a one-step generation model and then fine-tunes it for preferences. But in practice, the authors start from a pre-trained one-step model. It could be made clear.", "methods_and_evaluation_criteria": "Proposed methods make sense. I have several points to make here. \n\n* The proposed method is shown to improve metrics like ClipScore, PickScore, etc. However, did the authors obtain other important metrics like GenEval [1], T2I-Compbench [2]? This could provide even better insights into understanding if the method disturbs the spatial capabilities of the pre-trained model. \n* It is not clear how this method performs on complex prompts. One drawback of using SDXL as the base architecture could be that it uses CLIP which restricts the prompt length only at 77. This is quite small compared to other more recent models like Flux.\n* The authors use a reference model to enforce regularization during he preference alignment step. What happens when there is a reference mismatch as investigated in MaPO [3]? \n* It would be better if the authors could distinguish between Diff-Instruct and Diff-Instruct* as the proposed method seems to draw a lot of parallels from Diff-Instruct. \n* Are there any criteria the assistant model needs to satisfy? \n* The paper has some insights on the different loss scales. I am interested to know if changing one aspect of the final loss function leads to any interesting properties (for example, does increasing the CFG influence as an implicit reward help with anything?).\n*  Did the authors explore other implicit rewards and explicit rewards in a controlled setup? I think examining different external rewards could be beneficial to understand the general trends when choosing one.\n* Instead of using prompts from the COCO dataset during evaluation, using the test sets of benchmark datasets like PartiPrompt [4], DrawBench [5], HPSv2-test set [6] could also be beneficial. \n\nReferences\n\n[1] GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment; Ghosh et al.; 2023; https://arxiv.org/abs/2310.11513. \n\n[2] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation; Huang et al.; 2023; https://arxiv.org/abs/2307.06350.\n\n[3] Margin-aware Preference Optimization for Aligning Diffusion Models without Reference; Hong et al.; 2024; https://arxiv.org/abs/2406.06424. \n\n[4] Scaling Autoregressive Models for Content-Rich Text-to-Image Generation; Yu et al.; 2022; https://arxiv.org/abs/2206.10789. \n\n[5] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding; Saharia et al.; 2022; https://arxiv.org/abs/2205.11487. \n\n[6] Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis; Wu et al.; 2023; https://arxiv.org/abs/2306.09341.", "theoretical_claims": "Theorem 3.1 has been substantiated in the supplementary.", "experimental_designs_or_analyses": "* How general is this method? Could the authors apply it to a smaller model like PixArt-Alpha [1] and demonstrate similar results? \n* What are the memory requirements of this method? The method seems to need three models in memory, which leads me to worry about its memory-intensive nature. \n* Could the method be implemented with LoRA [2]? For example, in C.1,  `Drf` could be the base reference model and `Dta` could be injected with the LoRA layers. Then during forwarding with `Drf`, the LoRA layers could be disabled and enabled when forwarding with `Dta`. If this works well, this would substantially reduce the memory requirements. An implementation of this is available in [3].\n* What is `G` in C.1.? What is its architecture? \n* Can DI* be applied to non-1-step models?\n\nReferences\n\n[1] PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis; Chen et al.; 2023; https://arxiv.org/abs/2310.00426. \n\n[2] LoRA: Low-Rank Adaptation of Large Language Models; Hu et al.; 2021; https://arxiv.org/abs/2106.09685. \n\n[3] StackLLaMA: A hands-on guide to train LLaMA with RLHF; Beeching et al.; 2023; https://huggingface.co/blog/stackllama.", "relation_to_broader_scientific_literature": "Having means to generate images speedily that follow human preferences better is beneficial.", "essential_references_not_discussed": "MaPO [1] in the preference alignment literature.\n\nImagine-Flash [2], Hyper-SD [3] in the one-step model literature.", "other_strengths_and_weaknesses": "I have covered them in the other sections.", "other_comments_or_suggestions": "* Why is the blue color present in abundance?\n* Why does the first equation start from 2.1.?\n* In Figure 1, the second sentence seems redundant.\n* In the introduction section, advancements and contributions seem to have a very redundant overlap. Consider revisiting. \n* It is not clear in what is $p_\\theta$ and how it is different from $g_\\theta$.\n* $r$ is undefined in the context of equation 3.1. \n* What is $\\mathbf{d}^{\\prime}\\left(\\boldsymbol{y}_t\\right)$ in equation 3.4?\n* The exact checkpoints used for the models could be provided as footnotes.\n* The training related details are missing.", "questions_for_authors": "Have noted in other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper has received all acceptances in the final recommendations. The authors introduce Diff-Instruct*, a novel post-training method designed to align one-step text-to-image generative models with human preferences. While the initial reviews highlighted concerns regarding unclear explanations and the absence of qualitative analysis, the authors' rebuttal effectively addressed several of these points. This response led to a consensus among the reviewers, ultimately resulting in the Area Chair's decision to accept the paper.
Decision: Accept (poster)