Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences
Authors: Weijian Luo
Venue: Accepted by TMLR ()

Abstract: One-step text-to-image generator models offer advantages such as swift inference efficiency, flexible architectures, and state-of-the-art generation performance. In this paper, we study the problem of aligning one-step generator models with human preferences for the first time. Inspired by the success of reinforcement learning using human feedback (RLHF), we formulate the alignment problem as maximizing expected human reward functions while adding an Integral Kullback-Leibler divergence term to prevent the generator from diverging. By overcoming technical challenges, we introduce Diff-Instruct++ (DI++), the first, fast-converging and image data-free human preference alignment method for one-step text-to-image generators. We also introduce novel theoretical insights, showing that using CFG for diffusion distillation is secretly doing RLHF with DI++. Such an interesting finding brings understanding and potential contributions to future research involving CFG. In the experiment sections, we align both UNet-based and DiT-based one-step generators using DI++, which use the Stable Diffusion 1.5 and the PixelArt-$\alpha$ as the reference diffusion processes. The resulting DiT-based one-step text-to-image model achieves a strong Aesthetic Score of 6.19 and an Image Reward of 1.24 on the COCO validation prompt dataset. It also achieves a leading Human preference Score (HPSv2.0) of 28.48, outperforming other open-sourced models such as Stable Diffusion XL, DMD2, SD-Turbo, as well as PixelArt-$\alpha$. Both theoretical contributions and empirical evidence indicate that DI++ is a strong human-preference alignment approach for one-step text-to-image models.

Published in Transactions on Machine Learning Research (11/2024)
Diff-Instruct++: Training One-step Text-to-image Genera- tor Model to Align with Human Preferences
Weijian Luo"
luowefjion@sttu.pii edu.
Peking University
Reviewed on OpenReview: https: // openreview. net/ forum? id= SebN
Abstract
One-step text-to-image generator models offer advantages such as swift inference efficiency, flexible architectures, and state-of-the-an generation performance. In this paper, we study the problem of aligning one-step generator models with human preferences for the first time. Inspired by the success of reinforcement learning using human feedback (RLHF), we formulate the alignment problem maximizing expected human reward functions while adding an Integral Kullback-Leib/e divergence term to prevent the generator from diverging. By overcoming technical challenges, we introduce Diff-nnstruct++ (DI++), the first, fast- converging and image data-free human preference alignment method for one-step text-to- image generators. We also introduce novel thcoretical insights, showing that using CFG for diffusion distillation secretly doing RLHF with DI++. Such an interesting finding brings understanding and potential contributions to future research involving CFG. In the experiment sections, align both UNet-based and DiT-based one-step generators using DI++, which use the Stable Diffusion 1.5 and the PixelArt-a as the reference diffusion processes. The resulting DiT-based one-step text-to-image model achieves a strong Aesthetic Score of 6.19 and an Image Reward of 1.24 on the COCO validation prompt dataset. It also achieves a lcading Human preference Score (HPSv2.0) of 28.48, outperforming other open-sourced models such as Stable Diffusion XL, DMD2, SD-Turbo, as well as PixelArt- a. Both theoretical contributions and empirical evidence indicate that DI++ is strong human- cumman-preference aligmment approach for one-step text-to-image models. The homepage of the paper is: https::/gittan cww/coomm//////ecccc htm/pkuuxj1994//rt difff instruct__p
1 Introductions
In recent years, deep generative models have achieved remarkable successes across various data generation
and manipulation applications [Karras al.] 2020
2022
Nichol & Dhariwal 2021
Oord et al.
2016
Ho
et al.
2022
Poole et al.]
2022
Hoogeboom et al.
2022
Kim al.] 2022
Tashiro et al. 2021
Kingma &
Dhariwal 2018
Chen et al.
2019
Meng et al.)
2021
Couairon et al. 2022 Zhang et al.
2023a Luo &
Zhang 2024
Xue et a..
2023
Lao et al. 2023c
Zhang et al.] 2023b Feng et al. 2023 Deng et al. 2024 Luo
et al. 2024c Geng et al.
2024
Wang et al. 2024 Pokle et a.. 2022). These models have notably excelled
producing high-resolution
lexl-conditional models such as images (Rombach et al., 2022 Saharia et al. 2022
Ramesh al. 2022; 2021 Luo al. 2024bb and other modalities with different applications
applications(Brooks (Brooks al, 2024 Kondratyuk et al. 2023 Evans et al. 2024 Luo al.) 2024c pushing the boundaries of Artificial
Inteligence Generated Content.
Among the spectrum of deep generative models, one-step generators have emerged as a particularly efficient and possibly best performing [Zheng & Yang 2024 Kim et al., 2024 Kang at al., 2028 Sauer et al.) 2023a generative model. Briefly speaking, a one-step generator uses a neural network to directly transport some latent variable to generate an output sample. Recently, there have been many fruitful successes training one-step generator models by distilling from pre-trained diffusion models (aka, Diffusion Distillation (Luo.
Alternative email: pkulwj 19.4iccoond.comm

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 1: Images generated by one-step text-to-image generator that has been aligned with human prefer- using ++ff-Instrucl+++. We put the prompt in Appendix D.1
2023
in domains of image generations (Salimans & Ho 2022 Luo el al.
2024a
Geng al. 2023 Song
et al.
2023
Kim et al.
2023
Song & Dhariwal 2023 Zhou et al. 2024b].
text-to-image synthesis Meng
et al.
2022
Gu et al.
2023
Nguyen & Tran 2023 Luo et al. 2024b
2023a
Song et al. 2024 Liu et al.
2024b
Yin et al.
2023
data manipulation Paararr et al. 2024
etc.
However, current one-step text-to-image generators face several limitations, including insufficient adherence
to user prompts, suboptimal aesthetic quality, and even the generation of toxic content. These issues arise because the generator models are not aligned with human preferences. In this paper, we study the problem of aligning one-step generator models with human preferences for the first time. We achieve substantial progress by training generator models to maximize human preference reward. Inspired by the succoss of reinforcement learning using human feedback (RLHF) (Ouyang et. al. 2022) in aligning large language models, we formulate the alignment problem as maximization of the expected human reward function with an additional regularization term to some reference diffusion model. By addressing tochnical challenges, we obtain effective loss functions and formally introduce Disf-nnstruct-an ++, an image data-free method to train one-step text-to-image generators follow human preferences.
In the experiment part, we demonstrate the strong compatibility of Diff-Instruct+++ on both the UNet-basce diffusion model and one-step generator such as Stable Diffusion 1.5, and the DiT-based (Peebles & Xie 2022
diffusion and generator such as PixelArt-a (Chen et al. 2023). We first pre-train a one-step generator model using Dift-Instruct (Luo et al. 2024a), initialized with Stable Diffusion 1.5 and PixelArt-a. We name this model an unaligned base generator model (base model for short). Next, we align the base model using DI++ with an off-the-shelf Image Reward (Xu et al. (2023a) model using different configurations, resulting in various prrfn-pd onc-step text-to-image models.
The alignment process with DI++ significantly improves the generation quality of the one-step generator model with minimum computational cost. To evaluate our models from different perspectives, we conduct both qualitative and quantitative evaluations of aligned models with different alignment sottings. In the quantitative evaluation, we evaluate the model with several commonly used quality metrics such as Huma- preference Score (HPSv2.0) WW et al. 2023). the Aesthetic score (Schuhmann) 2022), the Image Reward (Xul ct al. 2023a), and the PickScore (Kirstain al. (2023) and the CLIP score on the same 1k prompts from MSCOCO-200 2017 validation datasets. Our main findings are: 1) the best DaT-basod one-step text-to-image model achieves leading HPSy2.0 score of 28.48; 2) The aligned models outperform the unaligned ones
2

---

Published in Transactions on Machine Learning Research (11/2024)
Pre-train the teacher diffusion model
Collect the Human Feedback Data
Use teacher, TA, and reward model to align student (with Diff-Insttruct+++)
teacher
Image-Caption
Data
reward
Diff-Instruct
Reward Modeling
teacher
Distill to get the One-step Student Generator [Optional] (with Inftrunnstuct or others)
Train Human Reward Model with Human Feedback Data
TA
student
reward
Diff-Instruct+
student
Stage 1: Pre-training
Stage 2: Reward Modeling
Stage 3: Alignment
Figure A demonstration of three stages for training a one-step text-to-image generator model that is aligned with human preference. The pre-training stage (the leftmost column) pre-trains the reference diffusion model well as the one-step generator. The reward modeling stage (the middle column) trains the reward model using human preference data. The alignment stage (the rightmost column) uses pre-trained reference diffusion model, the reward model, and TA diffusion model to align the one-step generator with human preference.
with significant margins in terms human preference metries; 3) The alignment cost is cheap, and the convergence Tast; 4) The aligned model still makes simple mistakes.
We will discuss these findings in Section 5.3 in detail. We also establish the theoretical connections of DI++ with diffusion distillation methods, as well as the classifier-free guidance (Ho & Salimans 2022). In Theorem 3.4 in Section 3.1 we show an interesting finding that the well-known classifier-free guidance is secrelly doing RLHF according to an sesiicitll-definee reward function. Therefore, diffuusion distillation with CFG can be unified within DI++ These theoretical findings not only help understand the behavior of classifier-free guidance but also bring new tools for human preference alignment for text-to-image models.
In Soction 5.4 we discuss the potential shortcomings of DI++, showing that even aligned with human reward, the one-step model still makes simple mistakes sometimes. Imperfect human reward and insufficient hyperparameter tunings possibly cause this issue.
2 Preliminary
Diffusion Models. In this section, we introduce preliminary knowledge and notations about diffusion models. Assume we observe data from the underlying distribution qa(æ). The goal of generative modeling is to train models to generate new samples æ qz(x). Under mild conditions, the forward diffusion process of a diffusion model can transform any initial distribution go qa towards some simple noise distribution,
dæ; F(x,, + G(t)dw:,
(2.1)
where is a pro-defined vector-valued drift function, G(t) is a pre-defined scalar-value diffusion coefficient, and w; denotes an independent Wiener process. A contimuous-indexet score network s (x, t) is employed approximate marginal score functions of the forward diffusion process (2.1) The learning of score networks
3

---

Published in Transactions on Machine Learning Research (11/2024)
is achieved by minimizing a weighted denoising score matching objective
(Vincent 2011
Song al. 2020
LDSM = L'
X(t)E
[1,,(((()) V sa log (0 2m) 1l
(2.2)
L=
meg (Myy ing)
Here the weighting function X(t) controls the importance of the learning at different time levels and (2, ( denotes the conditional transition of the forward diffusion [2.1]. After training, the score network (p t)
Va, log ge ((x)) is a good approximation of the marginal score function of the diffused data distribution.
Reinforcement Learning using Human Feedback. Reinforcement learning using human feedback (Christiano et al. 2017 Ouyang et al. 2022) (RLHF) is originally proposed to incorporate human feedback knowledge to improve large language models (LLMs). Let po(c)c) be a large language model's output distribution, where the input prompt that randomly sampled from a prompt dataset C, and the æ is the generated responses. Let r(z,c) e) be scalar reward model that probably has been trained with human feedback data and thus measure the human preference an image-prompt pair (æt, c). Let Pre (z)ee be some reference LLM model. The RLHF method trains the LLM to maximize the human reward with Kullback-Leibler divergence regularization, which is equivalent to minimizing:
(( =
enc. (elg(e
[- (-r(z, [x] 3PKLilpollle
(2.3)
The KL divergence regularization term lets the model be close to the reference model thus preventing it from diverging, while the reward term encourages the model to generate outputs with high rewards. After the RLHF, the model will be aligned with human preference.
One-step Text-to-image Generator Model.
A one-step Gooddellow et al. 2014 Luo el al. 2024a Yin
et al. 2023 Zhou et al. 2024b; Luo et al., 2024b) text-to-image generator model is a neural network go go(-):) that can turn an input latent variable ~ (z))) and an input prompt e to a generate image x (or some latent vector before decoding as the case of latent diffusion models (Rombach et al. 2022) with a single
neural network forward inference: x (=)c). Compared with diffusion models, the one-step generator model has advantages such as fast inference speed and flexible neural architectures.
3 Humann-peffreene Alignment of One-step Text-to-image Models
In this section, we introduce how to align one-step Lext-lo-image generalor models and human preferences using hiff-Instruct+++. In Section 3.1 we introduce the formulation of the alignment problem and then identify the alignment objective. After that, we address technical challengos and propose the Theorem 3.1 and Theorem 3.2] which set the theoretical foundation of DI++ through the lens of reward maximization. Interestingly, we also find that the diffusion distillation using classifier-free guidance is secretly doing RLHF with DI++. Based the theoretical arguments in Section 3.1 we formally introduce the practical algorithm of DI++ in Section 3.3 and give an intuitive understanding of the algorithm as an education process that includes a teacher diffusion model, a teaching assistant diffusion model, and a student one-step generation.
3.1 The Alignment Objective
The Problem Formulation. We consider the text-to-image generation task. Other conditional generation applications share a similar spirit. Assume æ is an image and e is a text prompt that is sampled from some prompt dataset C. The basic setting is that we have one-step generator ge(-I-), which can transport prior latent vector ~ P= to generate an image based on input text prompt c: æ = co(z/c). We use the notation po(x]c) to denote the distribution induced by the generator. We also have reward model r(æ, e) which represents the human preference on a given image-prompt pair (ær
Inspired by the succoss of reinforcement learning using human foodback (Ouyang et al. 2022) fino-tuning large language models such ChatGPT (Achiam et al., 2023),, we first set our aligument objective maximize the expected human reward function with additional Kullback-Leibler divergence regularization
4

---

Published in Transactions on Machine Learning Research (11/2024)
w.r.t some reference distribution Pref (tc)) which is equivalent to minimize the following objective:
[()) E
r(z, c)] + 3D (Dxr
(alp(())))
E e-cpa-ppi =mag(sle)
r(x, c)] + 3D
(3.1)
The KL divergence regularization to the reference distribution Pres(-) guarantees the generator distribution po(-) to stay similar to (te ) in order to prevent from diverging.
Though objective 3.1b appealing, for the one-step model, we do not know the explicit form of distribu- tion po(2]c). Inspired by Diff-Instruct (Luo et al.] 2024a] and other distribution matching- based based diffusion
distillation approaches, we can effectively estimate the 0-gradient of objective 3.1 with equation 3.2 in
Theorem 31
Theorem 3.1. The 0 gradient of the objective 3.1 is
Ox 30
Grad(O) E Bo-c amag(ale)
Var(w, e) + 3 logno(zlc)
Va log Pres (1/cc)]
(3.2)
We will give the proof in Appendix B.1 For gradient formula 3.2 we can see that the a gradient of r(æ, c) is easy to obtain. If we can approximate the score function of both generator and reference distribution, i.e. Va log po (x) (c) and v logPref (arle, we can directly compute the 0 gradient and use gradient descent algorithms to update the parameters 0. However, since the generator distribution is defined directly in image space, where the distributions are assumed to lic some low dimensional manifold (Song & Ermon 2019 Therefore, approzimating the score function and minimizing KL divergence is difficult in practice.
Diffusion Modcls are Reference Processes. Instead of minimizing the negative reward with KL regu- larization in (3.1, we turn to (.3 by generalizing the KL divergence regularization to the Integral Kullback- Leibler divergence proposed in Luo et al. (2024a) w.r.t to some reference diffusion process Pref (ax. |t, c). This novel change of regularization divergence distinguishes our approach from RLHF methods for large language model alignments. Besides, such a change from KL divergence IKL divergence makes it possible to use pre-tmained diffusion models reference processes, we will show in the following paragraphs.
Let D; be noisy data that is diffused by the forward diffusion (2.1) starting from zo. We use Pref (2t and ( It, c) to denote the densities and score functions of the reference diffusion process (the score functions can be replaced with pre-trained off-the-shelf diffusion models). Tet p=(x])t, and (t(x,)1, c) be the marginal distribution and score functions of the generator output after forward diffusion process 2.1 We propose to minimize the negative reward function with an Integral KL divergence regularization:
1 1" w(t))ku(po(0x)tt Pref (2r c))dt 1=0
[())= = E aepp po [- r(wo, c)] + pompp())
(3.3)
Different from vanilla RLHF objective 3.. 1, the objective with IKL regularization 3.3 assigned a regu- larization between generator's noisy distributions and reference diffusion process Pref (-14, c). Following similar spirits of Theorem 3.1 we have gradient formula in Theorem 3.2 which takes the IKL divergence into account.
Theorem 3.2. The 0 gradient of the objective
33.3 is
{
dz; V 20 r (oo c) + Bwi(t [so(xilt, c) (zef c)]
(f)
Eo,t, aap on (pmpoti))) mn (mg (mgep((s//
(3.4)
0e
We will give the proof in Appendix replaced with off-the-shelf text-to-image diffusion models.
B.2
We can clearly see that the reference score functions can be
Remark 3.3. If we view the generator as a student, the reference diffusion model acts like a teacher. We can use another diffusion model s-(-) to act like teaching assistant (TA), which is initialized from the teacher and fine-tuned using student-generatell data to approximate the student, generator score functions.
5

---

Published in Transactions on Machine Learning Research (11/2024)
ie. (p(axft, c) = so(a))t, c). The reward function r(:,)) can be viewed the student's personal preference. With this, we can readily estimate the gradient (3.4) and update the generator model to maximize students' interests (aka, to maximize the reward) while still referring to the teacher's and the TA's advice. From this perspective, the Diff- +n algorithm is similar to an education procedure that encourages the student maximize personal interests, while still taking advice from teachers and leaching assistants. Since the use of IKL. divergence for regularization in RLHF is inspired by Diff-Instruct (Luo al. 2024a), we name our alignment approach the +iff-Instruct+++t
Though minimizing objective (3.3) is solid in theory, it would be beneficial to properly understand and use a classifier-free guidance mechanism in the alignment process. In the following paragraphs, we give an interesting theoretical finding, showing that classifier free guidance is secretly doing RLHF with an implicitly- defined reward function. With this, we can properly incorporate both the human reward and the CFG reward for alignment with fiff-Instruct++
3.2 Classifier- free Guidance is Secretly Doing RLHF.
In previous sections, we have shown in theory that with available reward models, we can readily do RLHF for one-step generators. However, in this part, we additionally find that the classifier-free free guidance secretly doing RLHE, and therefore will show that using CFG for reference diffusion models when distilling them using Diff- Inftruntt is secretly doing Diff-Instruct-i with an implicitly defined reward.
The classifier-free guidance
(Ho & Salimans 2022 (CFG) uses a score function with a form
(2 f|c) = Sref (z t () + w/ (Sre t|c) $reʃ (x,,(
to replace the original conditions score function Sref ( t/c). This empirically leads to better sampling quality for diffusion models. In this part, we show how diffusion distillation using Diff-Instruct with CFG is secretly doing RLHF with +I++, If we consider a reward function as:
r(ao, c) = f w(() log Pref( dt.
(3.5)
Pref (ref((xe())
This reward function will put higher reward to those samples that have higher conditional probability than unconditional probability. Therefore, it can encourage samples with high conditional probability. We show that the gradient formula (3.4) in Theorem 3.2 will have an explicit solution:
Theorem 3.4. Under mild conditions, if we set an implicit reward function as 3.4) in Theorem 3.2 with have an explicit expression:
3.5 the gradient formula
dx. 00
Grad(O) =R e. sg Bwr (1) 80 (0(ee)), 3 (arl( me moooppin (mg)
(3.6)
(2: + (1 1 (are - Sref (x:
res (well. c)
We will give the proof in Appendix B.4 This gradient formula 3.6 recovers the case that uses the CFG for diffusion distillation using the Inst-unt algorithm to train a one-step generator. The parameter (1 (1+1) is the so-called classifier-fres guidance scale. In our Algorithm D the CFG scale. In the following section, we formally introduce the practical algorithm of Diff-Instruct+++
and 2 we use the coefficient aefg to represent
Remark 3.5. Thcorem 3.4 has revealed new perspective that understands classificr-free guidance as a training-free free and inference-time RLHF. This helps to understand why humans prefer samples generated by using CFG. Besides, Theorem 3.4 also shows that using bnff-Instruct with CFG to distill text-to- toxt-t diffusion models is secretly doing DI++. Therefore we can use both CFG and the human reward to strengthen the one-step generator models.
3.3 A Practical Algorithm
Though the gradient formula 3.2) gives an intuitive way to compute the parameter gradient to update the generator, it would be better to have easy-to-implement pseudo loss function instead of the gradient

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 1: Diff-Instructt+++ for aligning generator model with human feedback reward.
Input:
prompt dataset C, generator go & c), prior distribution Pz: reward model r(æ; c), reward scale Orew: CFG scale Okfs. reference diffusion model $ref (). c), TA diffusion sy (y c), forward diffusion p(
[2.1] TA diffusson updates rounds KTA, time distribution =t),
diffusion model weighting A(t), generator IKL loss weighting w(t).
while not converge do
fix 0, update & for KTA rounds by minimizing
((b)
(0)1))) - Va, (xr
e ea (1) (e
mm))
update 0 using StaD with the gradient
Gr) =B ennC.ans awpp. =0 =oma6(z.e)
(-rrew Va, m r(0o, ci]
(3.7)
w(t)B
{ (2t It, c) - (x, c)} 00
dt.
(3.8)
C...C.appp snupp (emee m0=0 (x. (mm
1=0
(t ( it, c) (eel( Ø) acfg Seee (xrlt,), 8res( Ø)
(3.9)
end return v.
estimations for executing the algorithm. To address such an issue, we present a pseudo loss function defined as formula B.13), which we show has the same gradient as (3.4h in Appendix.
With the pseudo loss function B.13 we formally introduce the DI++ algorithm which is presented in Algorithm (and more exccutable version in Algorithm 2 As in Algorithm 1 the overall algorithms consist of two alternative updating steps. The first step update vo of the TA diffusion model by fine-luning it with coudent-generatee of student generator distribution. This step means that the TA needs to communicate with the student to know the student's status. The second step estimates the parameter gradient of equation (3.2) and uses this parameter gradient for gradient descent optimization algorithms such Adam (Kingma & Ba 2014) to update the generator parameter 0. This step means that the teacher and the TA discuss and incorporate the student's s interests to instruct the student generator. Due to page limitations, we put a discussion about the meanings of The hyper-paramelers in Appendix A.2
data. Therefore the TA diffusion sy (xzlt,, c) can approximate the score function
4 Related Works
RLHF for Large Language Models. Reinforcement learning using human feedback (RLHF) has won great success in aligning large language models (LLMs). Ouyang et al. (022) formulates the LLM alignment problem as maximization of human reward with a KL regularization to some reference LLM, resulting in the InstructGPT model. The Diff-nnstruct++ draws inspiration from Ouyang et al. (2022). However, DI++ differs from RLHF for LLMs in several aspocts: the introduction of IKL regularization, the novel gradient formula, and the overall algorithms. Many variants of RLHF for LLMs have also been intensively studied in
Tian et al.
2023 Christiano et al. 2017]; Rafailov al. (2024): Ethayarajh al. 2024h, etc.
Diffusion Distillation Through Divergence Minimization. Diffusion distillation Luoo 2023 is research area that aims to reduce generation costs using teacher diffusion models. Among all existing methods, one important line is to distill onc-step generator model by minimizing certain divergences between one-step generator models and some pre-trained diffusion models. (Luo el al. 2024a) first si study
the diffusion distillation by minimizing the Integral KL divergence. (Yin et al. (2023) generalize such a
concept and add a data regression loss to distill pre-trained Stable Diffusion Models. Many other works
have introduced additional techniques and improved the distillation performance (Geng et al. 2023 Kim
et al. 2023 Song et al. 2023 Song & Dhariwal Nguyen & Tran 2023 Song et. al. 2024 in et. al. 2024

---

Published in Transactions on Machine Learning Research (11/2024)
Zhou et al.
2024a Heek et al. 2024 Xie et al. 2024 Salimans et al. 2024 Different from IKL divergence,
[202bb study the distillation through variant of Fisher divergence. Other methods, such 2021]; Xu et al. (2024). have used the generative adversarial training [Goodfellow et al. 2014) techniques in order to minimize certain divergences. The + is motivated by Diff-Instruct. However, to the best our knowledge, we are the first to study the problem of aligning one-step generalor models with human preferences.
Zhou et al.
Xiao et al.
Preference Alignment for Diffusion Models. In recent years, many works have emerged trying to align diffusion models with human preferences. There are three main lines of alignment methods for diffusion models. 1) The first kind of method fine-tunes the diffusion model over specifically curated imago-prompt
dataset (Dai et al.
2023 Podell et al. 2023 2) the second line of methods tries to maxinize some reward
functions either through the multi-step diffusion generation outpui (Prabhudesai et al. 2023 Clark et al.
2023 Lee et al.
2023) or through policy gradient- based RL approaches (Fan et al. 2024 Black et al,
2023 ). Though this approach shares goals similar to those of DI++ and RLHF for LLMs, the problems and
challenges are essentially different. Our Diff-Instruct+++ is the first work to study the alignment of one-step generator models, instead of diffusion models. Besides, backpropagating through the multi-step diffusion generation output is expensive and hard to scale. 3) the third line, such as Diffusion-DPO (Wallace et al., 2024), Diffusion-KTO (Yang et al., 2024). tries to directly improve the diffusion model's human preference property with raw collected data instead of reward functions.
5 Experiments
In previous sections, we have established the theoretical foundations for DI++. In this section, consider one-step text-to-image models with two kinds of neural network architectures: the UNet-based one-step text-to-image model with the well-known Stable Diffusion 1.5 (SD1.5) (Rombach et al. 2022) as the reference diffusion model, and the diffusionntransformeee based one-step model with the PixelArt-a as the reference diffusion. For both models, we use DI++ to align the one-step models with human preferences using ImageReward(Xu et al. 2023a
5.1 The Experiment Setup
As have shown in Figure our experiment workflow consists of three modeling stages: the pre-training stage, the reward modeling stage (not necessary), and the alignment stage.
The Pre-training Stage. As the leftmost column of Figure shows, the pre-training stage pre-trains the reference diffusion model, and base one-step generator that is not necessarily aligned with human preference. The rescarcher can either train the diffusion model in-house or just use publicly available ohf-the- shelf diffusion models. With the pre-trained leacher diffusion model, we can pre-train our one-step genera Lor model using diffusion distillation methods (Luo 2023), generative adversarial training (Sauer et al. 2023a Zheng& & Yang 2024), or a combination of them [Kim et al.] 2023 Yin et al. 2024 Xu et al. 2024). Notice that in the pre-training stage, we do not necessarily use human reward to instruct the one-step model, therefore the generator can only learn to match the reference distribution, leading to decent generation results without strictly following human preference.
The Reward Modeling Stage. As the middle column of Figure 2 shows, in the second stage, the re- searcher is supposed to collect human feedback data and train reward model that reflects human preferences for images and corresponding captions. Notice that the image and caption data for this stage can cither be real image data or those images and prompls generated by users using the one-step generators in the first stage. For instance, if researchers want to enhance the generation quality of their users' commonly used prompts, they can collect the most used prompts from their users' activity and generate images using the one-step generator pre-trained the first stage. They can send the imago-prompt pair to users for feedback their preferences. The reward modeling method is quite flexible. Researchers either train the reward
8

---

Published in Transactions on Machine Learning Research (11/2024)
CFG + (large) Reward
CFG + Reward
CFG Reward
w/o CFG Reward
Figure 3: A qualitative comparison of one-step generator models aligned using Diff-Instruel++ with different configurations. The bottom row is the weakest setting with no human preference alignment. Upper rows are models that are aligned stronger in a progressive way. We put the prompts to generate images in Appendix D.2 The generated images are more and more aesthetic with stronger human preference alignments.
model in-house using different neural networks and data (Oyyang et al. 2022) or using off-the-shelf reward models such as Image Reward
[Xu et al. 2023a), or human preference scores (Wu et al. 2023
The Alignment Stage. As the rightmost column of Figure shows, the final stage is the alignment stage, which is the major stage that DI++ considers. In this stage, researchers can use the reference diffusion model from the first stage as a teacher, and the reward model from the second stage to align the one-step generator with the DI++ algorithm (Algorithm or After the aligument stage, the one-step generator can generate images that are not only realistic but also match human preferences.
http:://github..cmm//ttb///////

---

Published in Transactions on Machine Learning Research (11/2024)
The Models and the Dataset. For the SD1.5 experiment, refer to the high-quality codebase of SiD-LSG (Zhou et al. 2024a when constructing a 0.86B UNet-based one-step text-to-image model. For the PixelArt-a experiment, we use the 0.6B PixelArt-a model (Chen et al.) 2023) of 512x512 resolution to initialize and construct an 0.6B onc-step model. We put more experiment details in Appendix A.3
For the SD1.5 experiment, we use prompts from the Laion-Aesthetic dataset with an Aesthetic score higher than 6.25 (i.e., Laion-Aesthetic 625+ dataset) following the SiD-LSG. For the PixelArt-a experiment, we use the prompts from the SAM-LLaVA-Captinn 10M dataset our prompt dataset. The SAM-LLaVA- Caption-10k 10M dataset contains the images collected by Kirillov et al. 2023 together with text descriptions that are captioned by LLaVA model (Liu et al. 2024a). The AAM-LLaVA-Caption- training the PixelArt-a model. Since the PixelArt-a diffuusin model uses a T5-XXL, which is memory and computationally expensive. To speed up the alignment training, we pre-encode the text prompts using the T5-XXL text encoder and save the encoded embodding vectors.
datasct is used for
5.2 Quantitative Evaluations with Standard Scores
In this section, we quantitatively evaluate one-step models aligned with DI++ together with other text-to- image generative models.
Evaluation Metrics. We evaluate five widely used human preference scores: the human preference score (HPSv2.0) (Wu et al., 2023) the ImageReward (Xu et al.] 2023a), the Aesthetic Score (Schuhmann 2022) the PickScore Kirstain PickScoreKKirstain et al. 2023). and the CLIPScore CLIPScore[Racford Radford et al.) 2015).. Among them, the HPSv2.0
has standard prompt datasct for ovaluations, therefore we follow its tradition evaluate model scores. For the other four scores, we pick 1k text prompts from the MSCOCO-2017 validation dataset and evaluate all models on these prompts.
DiT-based DI++ Model Shows the Best Performance. As Table | shows, the DiT-DI++ model shows a leading HPSv2.0 of 28.48 with only 0.6B parameters, outperforming other open-sourced models which have sizes varying from 0.6B to 10+B. In Table 2 the DiT-DI++ model achieves an ImageReward of 1.24 and an Aesthetic score of 6.19. Besides, in both Table D and 2 the SDL.5-DI++ models show improved scores than SD1.5 diffusion model, Hyper-SD, and 3D1.5 diffusion DPO(Walle et al. 2024).
The Zero-shot Generalization Ability of Aligned Models. Another interesting finding of the DI++ algorithm, as well as the human preference alignment of the one-step generator model, is its zero-shot generalization ability. As Table[ann@2shoxx models aligned with Image Reward not only show dominating Image Reward metric but also show strong aesthetic scores and HPSv2.0. Such zero-shot generalization ability indicates that if the models are aligned with a sufficiently good reward function with DI++, they can readily generalize well to other human preference scores.
Comparing with Other Few-step Generator Models. Table and 2 show the superior advantage of DI++ aligned one-step model over other few-step models. In this paragraph, give Figure 4 for qualitative comparison of our models with other few-step models in Table 2 When compared with other few-step generative models, the aligned model shows better aesthetic quality.
Ablation Comparison Table shows an ablation comparison of the effects of Diff-Insttrut++ with different explicit reward scales or and classifier-free guidance reward scales ae- For instance, for the SDL.5 model, we find that larger explicit reward scale always leads to higher Image Reward and Aesthetic Score, while it may trade off the CLIPScore which represents the image-text semantic alignment. Such an observation indicates a tradc-off between the image-text alignment and human-preference alignment if we solely use human- prefeeenreeeeecceeeeee reward as the implicit reward. One possible solution is to also add the image-text alignment scores such as the CLIPScore as an explicit reward. Besides, we also find that CFG reward scale that is too large might hurt the human-preference alignment. For instance, the SD1.5-DI++ one-step model with (a, 100, a. = 1.5) shows better score than models with (a, 100, a. 4.5). This indicates that though CFG is practically useful in dilfusion sampling, it is not perfectly aligned with human preferences. Therefore, finding proper CFG reward scale is important for optimal human prefeeecee aligument with fiffInnstruct++..
10

---

Published in Transactions on Machine Learning Research (11/2024)
Table HPS v2 benchmark. We compare open-sourced models regardless their base model and architecture.f indicates our implementation. Models with bold formals are our models. Numbers with bold formats are the highest scores.
MODEL
ANIMATION
CONCEPT-ART
PAINTING
PHоTO
AVERAGE
GLIDE NICHOL
2021
23.34
23.08
23.27
24.50
23.55
LAFITE ZHOU ET AL.
2022
24.63
24.38
24.43
25.81
24.81
VQ-DIPFUSION
2022
24.97
24.70
25.01
25.71
25.10
FUSEDREAM LIU ET AL.
2021
25.26
25.15
25.13
25.57
25.28
LATENT DIFFUSION ROMBACH
2022
25.73
25.15
25.25
26.97
25.78
VIEW DING ET AL.
(2022
26.50
26.59
26.33
26.44
26.47
DALL E MINI
26.10
25.56
25.56
26.12
25.83
VERSATILE DIFFUSION XU
20238
26.59
26.28
26.43
27.05
26.59
VQGAN + CLIP ESSER ET Al..
2021
26.44
26.53
26.47
26.12
26.39
DALL RAMESH ROMBACH ET AL.
2022
27.34
26.54
26.68
27.24
26.95
STABLE DIFFUSION v1.4
2022
27.26
26.61
26.66
27.27
26.95
STABLE DIFFUSION v2.0
ROMBACH
2022
27.48
26.89
26.86
27.46
27.17
EPIC DIFFUSION
27.57
26.96
27.03
27.49
27.26
DEEPFLOYDDXT.
27.64
26.83
26.86
27.75
27.27
OPENJOURNEY
27.85
27.18
27.25
27.53
27.45
МAлСMIX REALISTIC
27.88
27.19
27.22
27.64
27.48
CHILLOUTMIX
27.92
27.29
27.32
27.61
27.54
DELIBERATE
28.13
27.46
27.45
27.62
27.67
REALISTIC VISION
28.22
27.53
27.56
27.75
27.77
SDXL-BASE PODELL BT AL.. 2023
28.42
27.63
27.60
27.29
27.73
DAXL-REFINER AL.
2023
28.45
27.66
27.67
27.46
27.80
DREAMLIKE PHOTOREAL 2.0
28.24
27.60
27.59
27.99
27.86
SD15-15STER
ROMBACH
2022
26.76
26.37
26.41
27.12
26.66
SD15-25STEN
ROMBACH ET AL.
2022
27.04
26.57
26.61
27.30
26.88
SD15-DPO-IE 15STPP WALLACE
2024
27.11
26.75
26.70
27.30
26.97
SD15-DPO-25S15
WALLACE ET AL.
2024
27.54
26.97
26.99
27.49
27.25
SD15-LCM-
LUO BT AL..
2023A
23.35
23.53
23.81
23.52
SD15-LCM-4STEP
LUO ET AL. 2023
26.42
25.79
25.95
26.91
26.27
SD15-TCD-ISTEN
ZHENG ET AL.
2024
23.37
23.16
23.26
23.88
23.42
SD15-TCD-4STEP
ZHRNG EVT Al..
2024
26.67
26.25
26.26
27.19
26.59
SD15-HyPRR- ISTEP
REN ET A...
2024
27.76
27.36
27.41
27.63
27.54
SD15-HYYER--STER REN ET
2024
28.04
27.39
27.42
27.89
27.69
SD13-INSTAFPLWWIIITTE
LIU ET AL.
2023
26.07
25.80
25.89
26.32
26.02
SDI5-PEEREFLLOW-IIITER YAN
2024
25.70
25.45
25.57
25.96
25.67
SD15-BOOT-ISTEE Gu ET AL.
2023
25.29
24.40
24.61
25.16
24.86
SD2I-SWIFTBRUSH- ISTEP NGUVEN TRAN
2023
26.91
26.32
26.37
27.21
26.70
SD15-SSDD..G-1SmTe ZHou RT AT...
2024x
27.39
26.65
26.58
27.30
26.98
SD21-SHDLSG-ISTER ZHOU ET AL.
2024A
27.42
26.81
26.79
27.31
27.08
SD21-TUREO-1STER SAUER ET AL.
2023B
27.48
26.86
27.46
26.89
27.71
SDXI-DMD2-1STES
YIN ET AL.
2024
27.67
27.02
27.01
26.94
27.16
SDXL-DMD2-4STee- 1024
YIN ET AL.
2024
28.97
27.99
27.90
28.28
28.29
SDXI-DDMD2-1Smm--511
YIN ET AL..
2024
27.70
27.07
27.02
26.94
27.18
SDXL-DMDD-4STEP-B0
YIN
2024
27.22
26.65
26.62
26.57
26.76
SD15-DMD2-111H--112
YIN ET AL.
2024
26.31
25.75
25.78
26.59
26.11
D
2023
28.77
27.92
27.96
28.37
28.25
ExHLARRT---33STPPPP
CHEN RT A1..
2023
28.68
27.85
27.87
28.29
28.17
SD15-DI4+4-SSepl... 100, a.
1.5)
28.42
27.84
28.01
28.19
28.12
DiT-DI++-1Stopce. 10, a. 4.5)
28.91
28.25
28.28
28.50
28.48
11

---

Published in Transactions on Machine Learning Research (11/2024)
Table 2: Quantitative comparisons text-to-image models on 1k MSCOCO-2011 validation prompts. DI++ is short for Diff-Instruct++i a, and a are short for Qrew and 0.ff in Algorithm | means our imple- mentation. Data means the model needs image data for training. Sampling means the model needs draw samples from reference diffusion models. Reward means the model needs human reward model for training.
Image Reward
Aes
Pick CLIP
Model
Steps
Type Params
Score
Score Score
SD15-Base Rombach R et al.
2022
15
UNet
0.86B
0.08
5.25
0.212
30.99
SD15-Base Bwa.BBaaeRoooooa Rombach et al
2022
25
UNet
0.86B
0.22
5.32
0.216
31.13
SD15-DPO Wallace al.
2024
15
UNet
0.86B
0.20
5.29
0.214
31.07
SD15-DPO SD15-DPOWalllace Wallace al.
2024
UNet
0.86B
0.28
5.37
0.218
31.25
SD15-LCM Luo et al.
2023a
UNet
0.86B
-1.58
5.04
0.194
27.20
SD15-LCM
Luo et al.
2023a
4
UNet
0.86B
-0.23
5.40
0.214
30.11
SD15-TCD Zheng al.
2024
UNet
0.86B
-1.49
5.10
0.196
28.30
SD15-TCD
Zheng et al.
2024
UNet
0.86B
-0.04
5.28
0.212
30.43
PeRFlow Yan et al. 2024)
4
UNet
0.86B
-0.00
5.51
0.211
29.54
SD15-Hyper(HE) RRen et al.
2024
UNet
0.86B
0.28
5.49
0.214
30.82
SD15-Hyper Ren al.
2024
4
UNet
0.86B
0.42
5.41
0.217
31.03
SD15-InstaFlow al.
2023
UNet
0.86B
-0.16
5.03
0.207
30.68
SD15-SIDDLSG/2hou et al.
2024a
UNet
0,86B
-0.18
5.16
0.210
30.04
JDXL-Bass(Roubaach et al.
2022
25
UNet
2.6B
0.74
5.57
0.226
31.83
SDXL-DMID2-1024 Yin al.
2024
UNet
2.6B
0.82
5.45
0.224
31.78
SDXL-DMID2-1024) Yin al.
2024
UNet
2.6B
0.87
5.52
0.231
31.50
SDXL-DMD225112 Yin et al. 2024
UNet
2.6B
0.36
5.03
0.215
31.54
SDXL-DMDD2-512 Yin et al.
2024
4
UNet
2.6B
-0.18
5.17
0.206
29.28
SD15-DMD2--11 Yin et al. 2024
UNet
2.6B
-0.12
5.24
0.211
30.00
SD2I-1uubbfSauer al.
2023b
UNet
0.86B
0.56
5.47
0.225
31.50
PixelArt-o-511((hen et al.
2023
25
DiT
0.6B
0.82
6.01
0.227
31.20
FixelArt-a-512(Chea et al. 2023
15
DIT
0.6B
0.82
6.03
0.226
31.16
SD15-DI++ (a,=0, 1.5)
UNet
0.86B
0.29
5.26
0.216
30.64
SD15-DI++ 1000=00, a. -.5)
UNet
0.86B
0.46
5.44
0.218
30.33
SD15-DI++ (a,=100, a.54.5
UNet
0.86B
0.45
5.30
0.217
31.00
SD15-DI++ (a.-1000) -15)
UNet
0.86B
0.82
5.78
0.219
30.30
DiT-DI++ (o,=0, o.54.5)
DiT
0.6B
0.74
5.91
0.225
31.04
DIT-DI++ (a,=1, =.55.)
DIT
0.6B
0.85
6.03
0.224
30.76
DIT-DI++ (a,=10, 0.54.5)
DiT
0.6B
1.24
6.19
0.225
30.80
5.3 Qualitative Evaluations
In this section, we evaluate all one-step text-to-image generator models, showing that the + model shows improved human preference performancos. Before the quantitative evaluations, we first give a qualitative comparison of the models with and without alignment.
The Effects of Human Reward and CFG Reward. Figure 3 shows a visualization comparison of based one-step models trained with DI++ with different image reward scales and CFG reward scales. We will give some intuition to better understand the effects of two rewards.
From the bottom to the top, the first row of Figure 3* is the generated results from models trained with no image reward and no CFG reward (i.e., scales are set 0). These images are pretty realistic (such the coast image), but are of poor details and aesthetic appearance;
12

---

Published in Transactions on Machine Learning Research (11/2024)
5000 4:1
Mron
-gg 2010
Figure 4: Qualitative comparison of our our Diff-Instruc++ aligned models against other few-step text-to- image models in Table 2 The left three columns are randomly placed, with one generated by PixelArt-a model with 30 steps, one generated by a one-step model aligned with Diff-Instruct-*4 with a CFG scale of 4.5 and reward scale of 1.0, and another generated by one-step model aligned with 4.5 CFG and 10.0 reward. Please zoom in to check details, lighting, and aesthetic performances. Could you please tell which one you like the best? We put the answer for each image and prompts for three rows in Appendix D.4
2.
The second row is the model trained only with unit scale of image reward. These images much better than DI++ models with both rewards. However, we can see that solely using image rewards results in vivid colors. This is possible because of the fact that human beings would prefer colorful images with great details. However, only using image rewards results in the objects in the generated images being small, which is possibly not wanted;
3. The third row is the generated images from models trained only with CFG reward. We can see the resulting images are pretty aesthetic. However, these images have too big objects and worse background details;
4. The fourth row is the generated images from models trained with unit image reward together with CFG reward. These images show improved image layout, vivid colors, as well as rich details in both foreground and background. We think these models are of the best results. The quantitative HPS score in Table also confirms our observations;
The fifth row is the images from models trained with large image rewards and standard CFG rewards. We find that these images tend to be more like paintings instead of realistic photos. The colors are also much brighter than other models. This shows that too-large image reward scale may lead to the model generaling painting-like images instead of realistic photo-like images. Therefore, researchers are encouraged to carefully choose the image reward scales and the CFG scales to train one-step models with Diff-Instruct++++ for their own purposes.
Figure to Figure 6 in the supplementary material shows more comprehensive qualitative results which may bring deeper understanding of DI++. Due to page limitations, we put more discussions qualitative findings in Appendix A.4
5.4 Limitations and Failure Cases of Diff-Instruct++
Despite the alignment training stage, the generator model still makes simple mistakes occasionally. Figure
shows some bad generation cases that we picked with multiple generation trails. (1) The Aligned Model
Still Misunderstands the Input Prompt. As the leftmost three images of show, the generated images
13

---

Published in Transactions on Machine Learning Research (11/2024)
Figure 5: Bad generation cases by aligned one-step generator model (4.5 CFG 1.0 reward).
ignore the concept of pear earrings, the battling in coffee cup, and the car playing football. However, we find such a mistake happens only occasionally. (2) The Generator Sometimes Generates Bad Human Faces and Hands. Please see the fourth and fifth images of Figure [ The face of the generated lady in the fourth image is not satisfying with blurred eyes and mouth. In the fifth image, the generated Iron Man character has multiple hands. (3) Sometimes The aligned model Still Can not Count Correctly. For instance, in the rightmost image, the prompt asks the model to generate birthday cake, however, the model generates two cakes with one near and another lying lar away. Besides, as Figure shows, very strong explicit reward scales lead the image to be very colorful with vivid details. This may cause generated images to look more like paintings instead of realistic photos. Therefore, we suggest researchers carefully choose the explicit reward scales according to their alignment purpose when using Diff-Ine Instruct++.
6 Conclusion and Future Works
In this paper, we have presented the Diff-Instruct+++ method, the first attempt to align one-step text-to- image generator models with human preference. By formulating the problem as a maximization expected human reward functions with an IKL divergence regularization, we have developed practical loss functions and fast-converging yet image data-free alignment algorithm. We also establish theoretical connections of Diff- + with previous methods, pointing out that the commonly used classifier freef guidance is secretly doing Diff-Instruct-+t Besides, we also introduce three-stage workflow to develop one-step text-to-image generator models: the pre-training, the reward modeling, and the alignment stage. We train one-step generator models with different alignment configurations and demonstrate the superior advantage of Diff- Instruct++ with a human reward that improves the sample quality and better prompt alignment.
While +iff-Instruct++ does not completely climinate the occurrence of simple mistakes image genera- tion, findings strongly suggest that this approach represents a promising direction for aligning one-step generators with human preferences. We think our work can shed light future research in improving the responsiveness and accuracy of text-lo-image generalion models, bringing us closer to AGI systems that more faithfully interpret and execute human intentions in visual content creation.
Acknowledgments
First, would like to acknowledge Zhengyang Geng for helpful discussions experiment settings and the overall writing of the paper. We also appreciate the authors of Diff- Instruct(Luo et al. 2023b Score-implicit Matching[Luo et al.] 2024b) and Long-short Scoro-identity Distillation (Zhou et al. 2024a for their high- quality codebases on diffusion distillation. Second, we would also like to acknowledge Xiaohongshu Inc. for its support computing on ablation experiments. Finally, we would like to thank the reviewers and editors of the Diff-Instruut paper for their professional opinions and wonderful efforts organizing the reviewing process.
(Zeillatiinnhhuuuuu
14

---

Published in Transactions on Machine Learning Research (11/2024)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tgee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint WrXii:2303.08777 2023.
Kevin Black, Michael Janner, Yilun Du, Πly Kostrikov, and Sergey Levine. Training diffusion models with
reinforcement learning. Xiv preprinl or Xiw:2305.13300 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL stttp:://oeena..com/eeee
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James T. Kwok, Ping Luo, Huchuan Lu, and Zhonguo Li. Pixart-a: Fast training of diffusion transformer for photoreal- istic text-to-image synthesis. ArXiv, (es/2300..0426, 2023. URL https://api semanticscholar org/
CorpusID:2633330
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Vörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Newral Information Processing Systems, pp. 9916 (9166999,,, 2019.
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Kevin Clark, Paul Vicol, Kevin Swersky, and David J Fleet. Directly fine-tuning diffusion models on differ- entiable rewards. XrXiv preprint arXiv:2309. 17400, 2023.
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. ArXiv, abs/ 2ttp/2210.11122. 2022.
Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Van- denhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a hayslack. arxiv preprint arXiv:2309.15800 2023.
Wei Deng, Weijian Luo, Yixin Tan, Marin Bilos, Yu Chen, Yuriy Nevmyvaka, and Ricky TQ Chen. Varia- tional schr\'' odinger diffusion models. arXiv preprint arXiv:2405.0 04..0...7799 0479,, 2024.
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. Advances in Neural Information Processing Systems, 35:16890-1699 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
2021.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint 4rXio:2407.14358 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Yasong Feng, Weijian Luo, Yimin Huang, and Tianyu Wang. A lipschitz bandits approach for continuous hyperparameter optimization. Xrxio preprint Sd::2302.01530 2023.
Zhengyang Geng, Ashwini Pokle, and Zico Kolter. One-step diffusion distillation via deep equilibrium models. In Shirth- Conference on Neural Information Processing Systems, 2023. URL https:
15

---

Published in Transactions on Machine Learning Research (11/2024)
Zhengyang Geng, Ashwini Pokle, William Luo, Justin Lin, and J Zico Kolter. Consistency models made easy. arxiv preprint orXin:2406. 14548, 2024.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672 2680, 2014.
Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Josh Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping. arXiv preprint arXiv:2306.00 20::2300.00544, 2023.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on compuler vision and pollern recognilion, pp. 10696 10706, 2022.
Jonathan Heek, Emiel Hoogeboom, and Tim Salimans. Multistep consistency models.
orXiv preprint
arXiv:2403.0688 068007 2024.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1550 2022.
Jonathan Ho, Tim Salimans, Alexcy Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXfo:2204.08458, 2022.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling, Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867-8887. PMLR, 2022.
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVP conference on computer vision and pallern recognilion, pp. 8110-81 8119, 2020.
Tero Karras, Mika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- basedd generative models. In Proc. NewrIPS, 2022.
Dongiun Kim, Chieh-Hsin Lai, Wei-Hsiiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXie preprint arXiv:2310. 02279, 2023.
Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Pagoda: Progressive growing of a one-step generator from a low-resolution diffusion teacher. arXiv preprint 3rXiv:2405.1448 2024.
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-Uss: diffusion model for text-to-speech via classifier guidance. In International Conference on Machine Learning, pp. 11119 11133. PMLR, 2022.
Dicderik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arxiv:1412. 6980, 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215-10224. 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IRFE/CVP International Conference Computer Vision, pp. 4015-4026, 2023.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprinl ar axXw:2305.01560, 2023.
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hormung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, al. Videopoct: A large language model for zero-shot video generation. arXiv preprint Xxxx:2312.14122 2023.
Kimin Loe, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Picter Abbeel, Moham- mad Ghavanzadeh, and Shixiang Shane Gu. Aligning texl-to-image models using human feedback. arXiv preprint Xitu:230211219 2023.
Haotian Tiu, Chunyuan Li, Qingyang Wu, and Yong Jae Tee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Hongjian Liu, Qingsong Xie, Zhijic Deng, Chen Chen, Shixiang Tang, Fucyang Fu, Zheng-jun Zha, and
Haonan Lu. Scott: Accelerating diffusion models with stochastic consistency distillation. arXiv preprint 2rXiv:2403. 01505, 2024b.
Xingchao Liu, Chengyue Gong, Lemeng Wu, Shujian Zhang, Hao Su, and Qiang Liu. Fusedream: Training-free text-to-image image generation with improved clip+ gan space optimization. arXiv preprint arXiv:2112.0157% 2021.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based lext-to-image generation. In The Twelfth International Conference on Learning Represen- tations, 2023.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.0433 04378, 2023a.
Weijian Luo. A comprchensive survey on knowledge distillation of diffusion models. ar preprint arXiv:2304 0tt.22304.4222, 2023.
Weijian Luo and Zhihua Zhang. Data prediction denoising models: The pupil outdocs the master, 2024. URL https::/openreevix: wtwpe/rrriiieeeeefoouun.... met/foorumiddwwicccurrrr
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre-trained diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https://openreviers http://penreviee..etefefooooee net/forum?
1d-MLIs5iRq4w
Weijian Luo, Hao Jiang, Tianyang Hu, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Training energy-based models with diffusion contrastive divergences. arXiv preprint arX:::2307.0100 2023c.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff-instruct: A universal approach for transferring knowledge from pre trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024a.
Weijian Luo, Zemin Huang, Zhengyang Geng, J Zico Kolter, and Guo-Jun Qi. One-step diffusson distil- lation through score implicit matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL /ttttt:///peeetet http://www....eeeoooooooooogoooa
Weijian Luo, Boya Zhang, and Zhihua Zhang. Entropy-based training methods for scalable neural implicit samplers. Advances in Neural Information Processing Systems, 36, 2024c.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yay Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0100 2021.
Chenlin Meng, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. Xiv preprint arXio:2210.0314 2022.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with variational score distillation. orXiv preprint arXiv:2312.05255 2023.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. ar preprint WiXiu2212.09672 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXim21 12. 10741, 2021.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio. arXiv preprinl ar veXii:1609.000
2016.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances in neural information processing systems, 35:27730- 27744,, 2022.
Gaurav Parmar, Taesung Park, Srinivasa Narasimhan, and Jun-Yan Zhu. One-step image translation with text-to-image models. arXiv preprint orXie:2403.12003 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. ar preprint
WMXivv2212.097/5 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXfe:2307.0159. 2023.
Ashwini Pokle, Zhengyang Geng, and J Zico Kolter. Deep equilibrium approaches to diffusion models. Advances in Newral Information Processing Systems, 35:37975 37990, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprinl Xic::22009 14988, 2022.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning pest-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.0373: 2023.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolu- tional generative adversarial networks. arXio preprint arXiv:1511.06433 2015.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821 8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint or Xixv:2204.0611 2022.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprinl orXim:2404.13668 136866 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Essor, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to- image diffusion models with deep language understanding. arXiv preprint 2eXic:2205.14889 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022. URL http:::/ppperreeeee..... http:///www....eeeooooooooo//a/a
Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep distillation of diffusion models via moment matching. arXiv preprint 0rx::22406.04 103, 2024.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis. In International conference machine learning, pp. 50105-30118, PMLR, 2023a.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprinl arXi::331.110442 2023b.
Christoph Schuhmann. Laion-aesthetics. pttpe///aaaaaa//bbb///aaaortteae 2022. Accessed: 2023 11- 10.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. orXiv preprint arXiv:2310.141 2023.
Yang Song and Stefano Ermon. Generative Modcling by Estimating Gradients of the Data Distribution. In Advances in Neural Information Processing Systems, pp. 11918-11930, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Tly Sutskever. Consistency models. arXiv preprinl arXiv:2308.01469 2023.
Yuda Song, Zehao Sun, and Xuanwu Yin. Sdxs: Real-time one-step latent diffusion models with image conditions. arxi preprint 2rX 16627, 2024.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score based diffusion models for probabilistic time series imputation. In Advances in Neural Information Processing Systems (NeurIPS). 2021.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn, Fine-tuning language models for factuality. arXiv preprint arXiv:2311.08.00 2023.
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Newral Computation, 23(7):1661 1674, 2011.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pottern Recognition, pp. 8228 8238, 2024.
Yifei Wang, Weimin Bai, Weijian Luo, Wenzheng Chen, and He Sun. Integrating amortized inference with diffusion models for learning clean distribution from corrupted images. arXiv preprint arXiw:2407.11102 2024.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint ar Xiv:23306. 09341, 2023.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2021.
Sirui Xie, Zhisheng Xiao, Diederik P Kingma, Tingbo Hou, Ying Nian Wu, Kevin Patrick Murphy, Tim Salimans, Ben Poole, and Ruiqi Gao. Em distillation for one-step diffusion models. arXiv preprint wxXiv:2405116872 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023a.
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 7754 7754-7765, 2023b.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Uoogen: You forward once large scale text-lo- image generation via diffusion gans, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8196-8206, 2024.
Shuchen Xue, Mingyang Yi, Weijian Luo, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhi-Ming Ma. SA-solver: Stochastic adams solver for fast sampling of diffusion modols. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Hanshu Yan, Xingchao Liu, Jiachun Pan, Jun Hao Liew, Qiang Liu, and Jiashi Feng. Perflow: Piecewise rectified flow as universal plug-and-play accelerator. arXiv preprint ar arXin:2405.0055 2024.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. Tn Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8941 8951, 2024.
Tianwei Yin, Michaöl Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint or arXiv:2311.1888 2023.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint arXiv:2405. 144667677.1 2024.
Boya Zhang, Weijian Luo, and Zhihua Zhang, Enhancing adversarial robustness via score-based op- timization. In Thirth-seventhh Conference on Neural Information Processing Systems, 2023a. URL http://pppeeneiirree.eoeoorrrraa het:/forrumid
https://openrevieet net
Boya Zhang, Weijian Luo, and Zhihua Zhang. Purify++: Improving diffusiinn-prrfiicttin with advanced diffusion models and control of randomness. arXiv preprint 2xXiv:2310.18700 2023b.
Bowen Zheng and Tiamming Yang. Diffussio models are innate one-step generators. arXio preprint arXiv:2405.20730 20750, 2024.
Jianbin Zheng, Minghui Hu, Zhongyi Fan, Chaoyue Wang, Changxing Ding, Dacheng Tao, and Tat-Jen Cham. Trajectory consistency distillation. arXiv preprint (rXiv:2402.19155 19159, 2024.
Mingyuan Zhou, Zhendong Wang, Huangjie Zheng, and Hai Huang. Long and short guidance in score identity distillation for one-step text-to-image generation. arXie preprint arXiv:2406.0150 2024a.
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang, Mingzhang Yin, and Hai Huang. Score identity distilla- tion: Exponentially fast distillation of pretrained diffusion models for one-step generation. arXiv preprint 0000::2444.04077, 2024b.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-fre training for text-to-image generation. In Proceedings of the IEFE/CVF conference on compuler vision and pattern recognition, pp. 17907-17917, 2022.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Algorithm 2: Diff-Instruct-+++ Pseudo Code.
Input:
prompt dataset C, generator [0(2o]=, 2, c), prior distribution Pz. reward model r(æ, c), reward scale Orew; CFG scale Oefs: reference diffusion model 8ref c), TA diffusion sy (pe c), forward diffusion (())) [2.1]. 'AA diffusson updates rounds KTA, time distribution (t),
diffusion model weighting A(t), generator IKL loss weighting w(t).
while not converge do
fix 0 update & for KAA rounds by
1. sample prompt C; sample time & π π(); sample z pz(2);
2. generate fake data: Zo = sglgo(z, c)]; sample noisy data: as ~ pa (as mn);
3. update & by minimizing loss: [(t) (z)l))) Va, log (zo))
fix v, update 0 using StaD:
1. sample prompt e~ C: sample time L ~~ πt);; sample z Pz(2);
generate fake data: ag = go (z, c); sample noisy data: az ~ pt pr (20);
3. compute CFG score: Krep it, c) Sre/ ( Ø) aefo Sre (au (xer( Ø)]:
4. compute score difference: yt sy(sg|ze]]l, (sg
5. update 0 by minimizing loss: [()) = { arew c) + w/ w() x1);
end
return 8, v.
Broader Impact Statement
This work is motivated by our aim to increase the positive impact of one-step text-to-image generative models by training them to follow human preferences. By default, one-step generators are either trained over large-scale image-caption pair datasets or distilled from pre-trained diffussio models, which convey only subjectivv knowledge without human instructions.
Our results indicate that the proposed approach is promising for making onc-step generative models more aesthetic, and more preferred by human users. In the longer term, alignment failures could lead to more severe consequences, particularly if these models are deployed in safety-critical situations. For instance, if aligmment failures occur, the one-step text-to-image model may generate toxic images with misleading information, and horrible images that can potentially be scary to users. We strongly recommend using our human preference aligument techniques together with AI safety checkers for text-lo-image generation to prevent undesirable negative impacts.
A Important Materials for Main Content
A.1 Pseudo-code of Diff-Instruct++
A.2 Meanings of Hyper-parameters.
Meanings of Hyper-parameters. As we perparameter has its intuitive meaning. The reward scale parameter Orew controls the strength of human preference alignment. The larger the Orew is, the ronnge the generalor is aligned with human preferences. However, the drawback for a too large Qrew might be the loss diversity and reality. Besides, we empiri- cally find that larger rrew leads richer generation details and better generation layouts. The CFG scale controls the strength of CFG when computing score functions for the reference diffusion model. As we have shown in Theorem 3.4 the Ceff represents the strength of the implicit reward function 3.5 We empirically
in Algorithm (as well Algorithm 2 Each hy-
21

---

Published in Transactions on Machine Learning Research (11/2024)
find that the best CFG scale for Diff-Instruct++ is the same as the best CFG scale for sampling from the reference diffusion model. The diffusion model weighting X(1) and the generator loss wni w(() controls the strengths put on each time level of updating 'A diffusion and the student generator. We empirically find that is decent to set X(t) to be the same as the default training weighting function for the reference diffusion. And it is decent to set the wol)) = for all time levels in practice. In the following section, we give more discussions on Diff-Instruct+
A.3 Experiment Details for Pre-training and Alignment
Experiment Details for Pre-training and Alignment We follow the setting of Diff-Instruct Luo ct al., 2024a) to use the same neural network architecture as the reference diffusion model for the one-step generator. The PixelArt-a model trained using so-called VP diffusion (Sonu et al. 2020), which first scales the data in the latent space, then adds noise the scaled latent data. We reformulate the VP diffusion as the form of so-called data-prediction proposed in EDM paper (Krrras et al. 2022) by re-scaling the noisy data with the inverse of the scale that has been applied to data with VP diffusion. Under the data-predicn prediction formulation, we select a fixed noise dint level to be dmit = 2.5 following the Diff-Instruc and SiD (Zhou let al. 2024bp. For generation, we first generate & Gaussian vector 2 ~ Ps = NOO,, o) Then we input z into the generator to generate the latent. The latent vector can then be decoded by the VAE decoder to turn into an image if needed.
The Training Setup and Costs. We train the model with the PyTorch framework. In the pre-training stage, we use the official checkpoint of off-the-shelf PixelArt-a-512x512 model [ as weights of the reference diffusion. We initialize the TA diffusion model with the same weights the reference diffusion model. We use Diff- Instruct to pre-train the generator. We use the Adam optimizer for both TA diffusion and generation at all stages. For the reference diffusion model, we use a fixed classifier-free guidance scale of 4.5, while for TA diffusion, we do not use classifier-free guidance (i.e., the CFG scale is set to 1.0). We set the Adam optimizer's beta parameters to be $1 = 0.0 and 32 = 0.999 for both the pre-training and alignment stages. We use a learning rate of 5e - for both TA diffusion and the student one-step generator. For the one-step generator model, we use the adaptive exponential moving average technique by referring to the implementation of the EDM (Karras et al., 2022). We pre-train the one-step model on 4 Nvidia A100 GPUs for two days (4 x 48- = 192 GPU hours), with batch size of 1024. We find that the Diff- Inftrunst algorithm converges fast, and after the pre-training stage, the generator can generate images with decent quality.
In the alignment. stage, we aim to inspect the one-step generator's behavior with different aligmment con- figurations. Notice that as we have shown in Theorem 3.4 using classifier- guidance is secretly doing RLHF with ln therefore we add both CFG and human reward with different scales to thor- oughly study the human preference alignment. More specifically, we align the generator model with five configurations with different CFG scales and reward scales:
1. no CFG and no reward: use a 1.0 CFG scale and a 0.0 reward scale; This is the weakest setting that we regard the model as a baseline with no human preference alignment;
2. no CFG and weak reward: use 1.0 CFG scale and 1.0 reward scale;
3. strong CFG and no reward: 4.5 CFG scale and 0.0 reward scale;
4. strong CFG and weak reward: 4.5 CFG scale and 1.0 reward scale;
5. strong CFG and strong reward: 4.5 CFG scale and 10.0 reward scale.
For all alignment training, we initialize the generator with the same weights that we obtained in the pre- training stage. We put the details of how to construct the one-step generator in Appendix C.I We also initialize the TA diffusion model with the same weight as the reference diffusion. We use the Image Reward as the human preference reward and use the Dift-Instruct algorithm | (or equivalently the algorithm 2
22

---

Published in Transactions on Machine Learning Research (11/2024)
to fine-tune the generator. We also used the Adam optimizer with the parameter (31, (0.0, 0.999) for both the generator and the TA diffuuson with batch size of 256. For the alignment stage, we use fixed exponential moving average decay (EMA) rate of 0.95 for all training trials. After the alignment, the gen- crator aligned with both strong CFG and reward model shows significantly improved acsthetic appearance, better generation layout, and richer image details. Figure shows demonstration of the generated images using our aligned onc-step generator with CFG scale Gefg of 4.5 and a reward scale Orew of 1.0. We will analyze these models in detail in Section 5.3
A.4 More Discussions on Findings of Qualitative Evaluations
There are some other interesting findings when qualitatively evaluate different models. First, we find that the images generated by the aligned model show better composition when organizing the contents presented in the image. For instance, the main objects of the generated image are smaller and show more natural layout than other models, with the objects and the background iteraling aesthetically. This in turn reveals the human presence: human beings would prefer that the object of an image does not take up all spaces of an image. Second, we find that the aligned model has richer details than the unaligned model. The stronger we align the model, the richer details the model will generate. Sometimes these rich details come as hint the readers about the input prompts. Sometimes they just come improve the aesthetic performance. We think this phenomenon may be caused by the fact that human prefers images with rich details. Another finding is that as the reward scale for alignment becomes stronger, the generated image from the alignment model becomes more colorful and more similar to paintings. Somet this leads to loss of reality to some degree. Therefore, we think that users should choose different aligned one-step models with a trade-off between acsthetic performance and image reality according to the use case.
B Theory
B.1 Proof of Theorem 3.1
Proof. Recall that po(-) is induced by the generator go(-), therefore the sample is obtained by æ = go (z(x))),
Ps. The term æ contains parameter through æ = g0(2)c), z ~ Pa- To demonstrate the parameter dependence,
we use the notation p-).). Pref()) is the reference distribution. The alignment objective writes
r(z,, c) (lgppo(())c)) log Pres Pog [(0) EE e.zopp spp0tlle)
(clc))
(B.1)
Ec { ((g0(2))). (og logpre/ logprr
(B.2)
Therefore, the 0 gradient of C()) writes:
0 ②
00 L(0)
Ee, xmps r(g0(z|c). $ ([0o)))((((=(c)cc) - log Pre (
00
dx
B θ
E Conpeeaups
Va Varr(a ( (x)e) log Prey Pog
E
log
(o)l)))))) (B.3)
00
mipo
00
23

---

Published in Transactions on Machine Learning Research (11/2024)
We can see, that the first term of equation B.3 is the result of Theorem 3.2
Next, we turn to show that
the second term of B.3 will vanish.
log (ogle( E
log po (x:cc)
aepptt c) 09
plee 00
Ee-pe
30 Po(oll)))
po (x/c)dd
po(a)
Eewpe
00 Po dz
(B.4)
0
/ (o((zlc)) )
Be-pe 00
Eewpo 00
1
(B.5)
(B.6)
The equality B.4 holds if function po(a)c) satisfics the conditions (1). po(x)(c is LebeStane integrable for with each 0: (2). For almost all R the partial derivative (pa(x)c)/00 exists for all 0 O. (3) there exists an integrable function h(.) RDD + R, such that po(cec) I h(x) for all a in its domain. Then the derivative w.r.t 0 can be exchanged with the integral over æ, i.e.
/ po(x)c)de. / oo po 00
a
B.2 Proof of Theorem 3.2
Proof. The proof of Theorem 3.2 is a direct gencralization of the proof of Theorem 3.1 as we put in Appendix
{
dx; V 20 r(oo c) + Bwo(t) (s((x)tt,) stef( c)] 00
Gfa Ee,t ta w (p-ooteio) my [ng pp))
Recall the definition of po c), the sample is obtained by 2o go(z|c), Ps, and 21 Pr (x (p)) according to forward SDE (2.1 Since the solution of forward, SDE is uniquely determined by the initial point and a trajectory of Wiener process Wre[o,T), we slightly abuse the notation and let Fz)e(z))), w, t) to represent the solution of æ, generated by zg and w. We let W ~ Pw to demonstrate a trajectory from the Wiener process where P., represents the path measure of Weiner process on E [0, T]. There are two terms that contain the generator's parameter The term æ, contains parameter through an = (2(c))) z P=- The marginal density po e) also contains parameter implicitly since p*(-It, e) is initialized with the generator output distribution po(-It 0)
The (- c) is defined through the pre-trained diffusion models with score functions (-ef()((,)) c). The alignment objective between pa(-It, c) and Pres(-It, c) is defined with,
L(0)
E Bett, -eetaasp.mpoopeel (x=o((s)e)) my p0)
r(wo, c) Bwr(t) Bwo() (x:ltt, -log Plog (xelt,))
(B.7)
24

---

Published in Transactions on Machine Learning Research (11/2024)
Therefore, the 0 gradient of (()) writes:
(()) =
a
r(90(2)c), +
w, t))lt, c) - Pog w,
f) cl]
00
0B
wooPww
E Be-persmpiie mppo)())))
ran c)
+ B [VV. logpo(x:lc) Va, log Pog (20e (20]lc)
(B.8)
00
00
experempE (
log logpo(zs(t.c)
(B.9)
00
The first term B.8 is what we want in the Theorem vanish under mild conditions. The term B.9 writes
3.2
We will show that the second term B.9 will
0
a
E copp.amps....................... (eele() 09
log (o c)
E e-pe
log po (c)
{ (ogpo(trlt.cc) e) po (2 it, c)dæ
Eewpe
polx,it,c)
Eewpo
/
0g popooolt fdz
(B.10)
/ poo c) fd do
Ecr-pe 00
0
Ee-pe 08
0
(B.11)
(B.12)
The cquality (B.10) holds if function po(x;, c) satisfies the conditions (1). Po (axelt, is LebeStanc integrable for æ with each 0; (2). For almost all RD, the partial derivative @pa(a; (0)0 exists for all 0 E O. (3) there exists an integrable function h(.) IR + I, such that po px c) I h(w,) for all in its domain. Then the derivative w.r.t 0 can be exchanged with the integral over æt, ie.
2 I ag Po (zilt, c)dæ
00 / po (=(l))ddz
B.3 Proof of the Pseudo Loss
Lemma B.1 (Pseudo Loss Function). The pseudo loss function B.13 has the same 0 gradient as
3.4
(0) = E E.t..spp mompo(az))) ae m0 (mg)
( e) + w(()))) at
(B.13)
5re f (z, It, c) = sre/ (z, tt Ø) + acsol (rres (2i c) - (x) Y: = l (sg( (xe c) Sref (sg/z;]lt,c). c).
Here the operator sg|:] means cutting off all dependence on this variable.
Proof. Recall the pseudo loss B.13
{
(0) = Ree. pec.. we pig-eeeei
r(wo, e) + Sw(())l an
y: == (sglax Sref (sg(zo|lt,
25

---

Published in Transactions on Machine Learning Research (11/2024)
Since all dependence of y cut out, Ye can be regarded constant tensor. Taking the 0 gradient of B.13
leads to
dxo
(0)
E Ee a. mo as (10gpp(()/)))))
V 2o
+ (w
30
00
00
doo
E e, Po my (900p(oe
V Vx 120
+ Bwi()
(sglo tt (sgg( c)
00
00
dxo
Bwi(t sa(xxlt, 8res(asft, ct)
17 0xx
E Po (e0o-plyyloo
r0zo c)
00
00
This is the exact gradient term of +
B.4 Proof of Theorem 3.4
Proof. Recall the definition of the reward behind classifier-free guidance 3.5 The reward writes
f" w(t) log Pres c) dt 1
r( c)
Pref (xx(()
This reward will put a higher reward on those samples that have higher class-conditional probability than unconditional probability, therefore encouraging class-conditional sampling. To make the derivation
clear, consider single time level t, and corresponding
r(x,,t, = w(t) log Pref (at tt
(B.14)
Pres (x,
The final result would be an integral of all single time-level t. With the single time level, we consider the alignment problem by minimizing
[()) = E (aamnoo(mmmill
r( t, c)] + (w(tt) it, c). Pref
pa ct)
Be..arpoo r(x,,( c)]
log Pref (xelt,
(B.15)
The optimal distribution pa- (x, c) that minimize the objective B.15 will satisfy the equation B.16
Pe- (2, Pre (x,
t( Bw((t log
+ CCc)
(B.16)
This means
Pref(zeltt log po- (zelt,c) c) C(c)
log
(B.17)
Prey (x,(t)
Pres (ee
The C(c) in equation [B.17 unknown constant that independent from æt and Then we can have the formula for the optimal distribution
log pa- (zelt, e) log pa- (x, c) + 1 log CogPres(ane log c) log (xe)() c(c) C(c)
1
3
3
log (x.(t) + (1 + } log log (2,lt)
1
(B.18)
Besides, we can see that the score function of the optimal distribution writes
1 }
V a, log pa- (z. (t,e) c) Va, log po- (2),() + (1
V.n log Pogere c) Va, log Pref (
(B.19)
26

---

Published in Transactions on Machine Learning Research (11/2024)
The equation [B19 recovers the so-called classifier-free guided score function. The final result is just integral of the [..19. Our results show that when using the classifier-free guided score for diffusion distillation using Inft-unstrut (i.e. the equation (3.6)) is secretly doing RLHFF (i.e. the Diff-Instruct+++) by using the reward [B14]. Besides, our results also bring new perspective: when sampling from the diffusion model using CFG, the user secretly doing an inference-time RLHF, and the so-called CFG scale is the RLFH strength.
C Experiments and Results
C.1 More Experiment details on Text-to-lmage Distillation
We follow the experiment setting of Diff-Instruct (Luo et al. 2024a), generalizing its CIFAR10 experiment to text-to-image generation. Notice that the Diff-Instruct uses the EDM model (Karras et al. 2022) to formulate the diffusion model, as well as the one-step generator. We start with a brief introduction to the EDM model.
The EDM model depends the diffusion process
dæ; tdw,, t [0,
(C.1)
Samples from the forward process C.1 can be generated by adding random noise to the output of the generator function, i.e., Zo + tc where ~ I)O,T) is Gaussian vector. The EDM model also reformulates the diffusion model's score matching objective as denoising regression objective, which writes,
((b) = /" X(t) (x pm (m)mm)) ll t) zu lt..
(C.2)
==
Where (-) is denoiser network that trics to predict the clean sample by taking noisy samples as inputs. Minimizing the loss (C.2) leads to trained denoiser, which has simple relation the marginal score functions as:
sy(x,, do( t)
(C.3)
Under such formulation, actually have pre-trained denoiser models for experiments. Therefore, the EDM notations in later parts.
Construction of the one-step generator. Let do(-) be pretrained EDM denoiser models. Owing to the denoiser formulation the EDM model, we construct the generator to have the same architecture as the pre-trained EDM denoiser with a pre-selected index t", which writes
Fp = go (z) = d(z,t", NO, (1)
(C.4)
We initialize the generator with the parameter as the teacher EDM denoiser model.
Time index distribution. When training both the EDM diffusion model and the generator, we need to randomly select a time / in order to approximate the integral of the loss function C.2 The EDM model has a default choice of t distribution as log-nor log-normal when training the diffusion (denoiser) model, i.e.
~ (1) : 1= exp(ss ~ N(Pmean P2aa), Pmean = 2200, Pard = 2.0.
(C.5)
(C.6)
And a weighting function
(ED (t2 02 o
(C.7)
(t x dddaaaa
In our algorithm, we follow the same setting as the EDM model when updating the online diffusion (denoiser) model.
27

---

Published in Transactions on Machine Learning Research (11/2024)
Weighting function. For the TA diffusion updates in both pre-training and alignment, we use the same (L)M(() (C.7 weighting function as EDM when updating the denoiser model. When updating the generator, we use a specially designed weighting function, which writes:
1
WGen (t) (sg(2 dy (sg(x)),t)llz Dt 20 + te, 6 (O.
(C.8)
(C.9)
The notation sg[:] means stop-gradient of parameter. Such a weighting function helps to stabilize the training. In the Text-to-Image distillation part, in order to align our experiment with that on CIFAR10, we rewrite the PixelArt-a model in EDM formulation:
o)(x;))xx--
(C.10)
Here, following the iDDPM+DDIM preconditioning in EDM, PixelArt-a is denoted by Fe. x is the image data plus noise with a standard deviation of 0, for the remaining parameters such as C1 and C2, we kept them unchanged to match those defined in EDM. Unlike the original model, we only retained the image channels for the output of this model. Since we employed the preconditioning of iDDPM+DDIM in the EDM, cach value is rounded to the nearest 1000 bins after being passed into the model. For the actual values used in PixelArt-a, beta start is set to 0.0001, and beta_ end is set to 0.02. Therefore, according to the formulation of EDM, the range of our noise distribution is [0.01, 156.6155), which will be used to truncate our sampled a. For our one-step generator, it is formulated as:
gx( Intt)) = -init
(C.11)
Here following Diff-Instruct to use mnit = 2.5 and x ~ N (m I), we observed in practice that larger values of Jini lead to faster convergence of the model, but the difference in convergence speed is nogligible for the complete model training process and has minimal impact on the final results.
We utilized the SAM-LLaVA-CO MAM-LLaVA-CappioonOOM dataset, which comprises prompts generated by the LLaVA model the SAM dataset. These prompts provide detailed descriptions for the images, thereby offering challenging set of samples for our distillation experiments.
All experiments in this section were conducted with bfloat16 precision, using the PixeLArt-XL-2251I2xx model version, employing the same hyperparametors. For both optimizers, we utilized Adam with a learning rale of 5e-6 and betas=(0, 0.999). Finally, regarding the training noise distribution, instead of adhering to the original iDDPM schedule, we sample the a from a log-normal distribution with mean of -2.0 and a standard deviation of 2.0, we use the same noise distribution for both optimization steps and set the two loss weighting to constant 1. Our best model was trained on the SAM Caption dataset for approximately 16k iterations, which is equivalent to less than 2 epochs. This training process took about 2 days on 4 A A100-40G GPUs.
With the optimal setting and EDM formulation, we can rewrite our algorithm an EDM style in Algorithm
C.2 More Discussions on Experiment Results
Low Alignment Costs. Besides the top performance, the training cost with D1++ is surprssingly cheap, Our best model is pro-trained with 4 A100-80G GPUs for 2 days and aligned using the same computation costs. while other industry models in Table 2 require hundreds of A100 GPU days. We summarize the dis- tillation costs in Table marking that DI++ is an efficient yet powerful alignment method with astonishing scaling ability. We believe such efficiency comes from the image-data- free property of DI++. The +I++ does not require image data when aligning, this distinguishes the DI++ from other methods that fine-tune models on highly curated image datasets, which potentially is inefficient.
28

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Hyperparameters used for +iff-Instruct++ on Aligning One-step Generator Models
Hyperparameter
Pre- Tre-Trainnng (Diff-Instrue
Alignment (Diff-nssruut+++))
DM s.
Generator ge
DM s.
Generator ge
Learning rate
5e-6
5e-6
5e-6
5e-6
Batch size
1024
1024
256
256
((*)
2.5
2.5
2.5
2.5
Adam $0
0.0
0.0
Adam $1
0.999
0.999
0.999
0.999
Time Distribution
(t)
PEDM (f) C.5
PEDM (C.
CC.5
PEDM (+)
C.5
Weighting
λEDM (t)
C.7
XEDM (t) C.7
Number of GPUs
4xA100-40G
4xA100-40G
4xH800-80C
4x 4XH000-O0G
Algorithm 3: Diff I Pseudo Code under EDM formulation.
Input:
prompt dataset C, generator (e z, c), prior distribution Ps. reward model r(x, c), reward scale Orew, CFG scale Oefg: reference EDM denoiser model ( c). TA EDM denoiser dæ (atll,,)), c), forward diffusion p(æ,/ (2.1), TA EDM denoiser updates rounds KT time distribution (t), diffusion model weighting X(t), generator IKL loss weighting w(t).
while not converge do fix 0, update & for KTA rounds by
sample prompt e ~ C; sample time & ~~ π{);; sample z ~ pz(2);
2. generate fake data: Zo sglgo(z, c)}; sample noisy data: I (x
3. update & by minimizing loss: ((v) = X(t)lld c) - xo11:
fix v. update 0 using StaD:
sample prompt C: sample time t (t);; sample z pz(z);
2. generate fake data: zo = go (z, c); sample noisy data: 2s ~ pr(a; kz0);
3. compute CFG score: d (z, 1t c) = dres (x, Ø) + aefs [dref ct) dre Ø)];
4. compute score difference: = (sglaxx) c) - (sg/
update 0 by minimizing loss: L(0) = arew (2x, + (t) x
end
return 0,
C.3 Pytorch style pseudo-code of Score Implicit Matching
In this section, we give a Porch style pseudo-code for algorithm 3
import torch
import torch. nn as an import torch. optim as optin inport copy
Initialize generator G Generator
load teacher DX
Drf DiffusionModel () -ooad (aappddt_tt__ckaaa eval () requuieesgrrad (False)
copy deepcopy (Drf) initialize online DM teacher DM
RevardModel () if alignment_stage else
29

---

Published in Transactions on Machine Learning Research (11/2024)
Define optimizers
15 opt_G optim. Adam (G. parameters (), 1r-0.001, betas-(o.0,, 0.999) opt_Sta optim. Adam (Dta paraaaanttrs O) 1r-0.001, betas=(o.0, 0.999))
17
18 Training loop
19
while True:
20
## update Dta
21
Dta. Dta. train requires_grad_ (True)
22
() requ.rreeueirrdaeee (False)
28
as update TA diffusion prompt = batch (atcoh''ponpt'
torch ((1024, 4, 64, 64), Prrice=G.devico with torch. torch.no_grad(). ())
fake_x0 - G(z, prompt))p)
sigma torch exp www.0tttrrch.o.. randa randn ([1,1,1,1) device-fake deviec) 2.00
31
torch randar Likeeeee_eefaeee
32
fake_xt fake_x0 sigma+noise
33
pred_x0 Dta(fake_xt) sigma, prompt)
34
36
weight wrnpteeddifueiin.commmm weight (sigma)
38
37
batch_loss weight * (pred_ x0 fake_x0 (ake__0))+3 batch_loss batch_loss sun
38
mean()
39
40
Drcimizer_Dta.zzrro.re
41
batch_low loss (ackvard
aptimizer_Dta.ntt Staieer_DDaaeee step
48
au update
Dta. D () requires_gras (False)
G. Grarii .reuuin...ruuurrs__.e...... requires_grad_ (True)
48
49
prompt batch prommpt torch. ((1024, 64, 64), device=G.device) fake_x0 G(z, prompt)
50
51
52
58
sigma torch exp www.0*ttrcchhocmmmoo randn ([1,1, device=fake_x0.deet device) 2.0) noise torch, randa_ like (fkeee___xx
54
55
fake_xt fake_x0 + sigma*noise
56
57
with torch. no_grad
58
if use_cfg: pred_x0_rff Drf (fake_xt, signa, None) cfg_scale (Drf (fake_xt, signa, prompt ) Drf (fake_xt, signa, None))
else: pred_x0_rff Drf (fake_xt, signa, prompt)
tret_x0_ta- Dta (fake_xt, nigma, prompt)
65
denoise diff pred_zo_ta pred_x0_rf veight Compute_G_ueigee (sigma, denoise_diff)
68
67
68
batch_ weight denoise_diff fake_xt
60
70
## compute reward loss if needed
71
if alignment_stage reward_loss scevard_scale r(fake_x0, pronpt)
72
78
batch_loss += reward_loss
74
75
batch_loss batch loss sun ([1,2, . mean ()
76
()
batch_ loss backward ()
30

---

Published in Transactions on Machine Learning Research (11/2024)
79
optimizer_G.ster
Listing 1: Pytorch Style Pscudo-code of Diff-Instruct++
D Prompts
D.1 Prompts for Figure
The prompts are listed from the first row to the second row; from left to right.
A small cactus with happy face in the Sahara desert.
dog that has been meditating all the time.
A alpaca made of colorful building blocks, cyberpunk.
A dog is reading thick book.
delicate apple (upleuunveerse of stors inside the apple) made of opal hung on a branch in the early morning light, adorned with glistening dewdrops. in the background beautiful valleys, divine iridescent glowing, opelescent textures, volumetric light, ethereal, sporkling, light inside body, biotumènescence, studio photo, highly delailed, sharp Socus, photorealism, pholorealism, 8k,, best quality, ultra delail, hyper detail, hdr, hyper detail.
Drone view of waves crashing against the rugged eliff's along Big Sur's Garay Point beach. The crashing blue waters create white-tipped woves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sils in the distance, and green shrubbery covers the cliff's edge. The steep drop from the road down to the beach is dramatic feat, with the cliff's edges jutting out over the sea. This is view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highoway.
Image of a jade green and gold coloured Fabergé egg, 16k resolution, highly detailed, product pho- tography, trending artstation, shorp focus, studio photo, intricate details, fairly dork background, perfect lighting, perfect composition, sharp features, Miki Asai Macro photography, close-up, hyper detailed, trending on artstation, sharp focus, studio photo, intricate details, highly detailed, by greg rutkowski.
Astronaut in jungle, cold color palette, muted colors, detailed, 8k.
D.2 Prompts of Figure
Prompts of Figure] from left to right:
small cactus with a happy face in the Sahara desert;
A delicate apple(universe of stors inside the apple) made of opal hung branch in the early morning light, adorned with glistening dewdrops. in the background beautiful valleys, divine iridescent glowing, opolescent texlures, volumetric light, ethereal, sparkling, light inside body, bioluminescencee, studio photo, highly detailed, sharp focus, photorealism, photorealism, 8k, best quality, ultra detail, hyper delail, hdr, hyper detail;
Drone view of waves crashing against the rugged cliffs along Big Sur's Garay Point beach. The crashing blue walers creale while-lipped waves, while the golden light of the selling sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff's s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff's edges jutting out over the sea. This is view that captures the raw beauty of the coast and the rugged landscupe of the Pacific Coast Highwayy;
31

---

Published in Transactions on Machine Learning Research (11/2024)
Astronaut in fungle, cold color palette, muted colors, detailed, 8k;
parrot with a pearl earring, Vermeer style;
D.3 Prompts for Figure 3
prompt for first row of Figure 3 A small cactus with a happy Jace in the Sahara desert.
prompt for second row of Figure An image of a jade green and gold coloured Fabergé egg, 16k resolution, highly detailed, product photography, trending artstation, sharp focus, studio photo, intricate details, fairly dark background, perfect lighting, perfect composition, sharp features, Miki Asai Macro photography, close-up, hyper detailed, trending on artstation, sharp focus, studio photo, intricate details, highly detailed, by greg rutkowski.
prompt for third row of Figure
Boby playing with toys in the snow.
D.4 Prompts for Figure
The answer for the left. three columns:
the first row from left to right is the one-step model (4.5 CFG and 1.0 reward); the PixelArt-a diffusion with 30 generation steps; the one-step model (4.5 CFG and 10.0 reward);
the PixelArt-a diffusion with 30 generation steps; the PixelArt-a diffusion with 30 generation steps; the one-step model (4.5 CFG and 10.0 reward); the first row from left to right is the one-step model (4.5 CFG and 1.0 reward);
the PixelArt-a diffusion with 30 generation steps; the first row from left to right is the one-step model (4.5 CFG and 1.0 reward); the first row from left to right is the one-step model (4.5 CFG and 10.0 reward);
The prompts from up to down are:
A dog that has been meditating all the time;
Drone view of waves crashing against the rugged cliff's along Big Sur's Garay Point beach. The crashing blue waters create white-tipped woves, while the golden light of the setting sun illuminates the rocky shore. A small island with lighthouse sits in the distance, and green shrubbery covers the cliff's edge. The sleep drop from the road down to the beach is dramatic feat, with the cliff's edges jutting out over the sea. This is view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highoway;
A delicate apple(univers) of stars inside the apple) made of opal hunng a branch in the early morn- ing lighl, adorned with glistening dewdrops. in the background beauliful valleys, divine iridescent glowing, opalescent tertures, volumetric light, ethereal, sporkling, light inside the body, biolumines- cence, studio photo, highly detailed, sharp focus, photorealism, photorealism, 8k, best quality, ultra detail, hyper detail, hdr, hyper detail.
32


Reviews:
[{"review_id": "akA5Tx1CEX", "reviewer": "Reviewer_o47F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n* the approach seems to work\n* while the idea of learning a reward is known, the math formulation that leads to an implementable solution is novel and valuable.\n\nCons\n* The idea is straightforward. Learning a reward has long been studied.\n* The paper is a bit hard to follow. It seems to me much of the content can be explained in a much easier way, however, the current organization makes it confusing.\n* Theorem 3.1 / 3.2 should not be theorems. if they are theorems they should be more self-contained.\n* experimental validation is weak. Currently validation is a bit clueless. I think the paper can be improved with more focused evaluation in both qualitative and quantitative.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The contribution of the paper is\n* the development of Diff-Instruct++ (DI++), a method for aligning one-step text-to-image generator models with human preferences. \n* DI++ learns human reward functions and aligns the model with an Integral Kullback-Leibler divergence to a reference diffusion model. \n* This method enables fast-converging and image data-free human preference alignment for one-step text-to-image generators, leading to significant image generation quality improvements.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the paper's flow needs to be majorly revised and re-organized. By reading the paper, I don't think the math part has significant contribution, so most of them can be moved to appendix.\n\nThe presentation of theorems should be re-considered. And they are not theorems.\n\nThe experiment validation needs to be improved. Current experiment section is very hard to understand and hard to tell what's the major benefits of the paper. I suggest more focused evaluation in both qualitative and quantitative results.\n\nI do not see many of the math parts contribute to the significance of the paper, like how to compute the gradients which I believe they are implementation details. So they should be moved to appendix.", "broader_impact_concerns": "The collection of human preferences may introduce biases. I don't see the paper has consideration of that issue.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qsk3mfoEZo", "reviewer": "Reviewer_UNo6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The method seems solid and allows to perform the task on a small budget\n- The method allow to perform the reward modeling without the need for training data\n\nWeaknesses:\n- The paper lack better qualitative analysis in the main paper. The supp mat figure should be moved to the main paper. I would also like to see failure cases\n- The results focus only on the reward scores. However, i think this models should also evaluated with more traditional image generation metrics (FID, CLIPScore, PRDC, GenEval). Indeed, this paper shows that the rewards are maximized but is it at the cost of diversity? Does the model loose clipscore?\n- Since the authors use a very high guidance for their best results i suspect a collapse of the model. What about a mix of different guidances during training?\n- Also, how does this metrics evolve with training?\n- I'm not sure why this method would only work with one step models? If not, would be interested to see results in multi steps setups\n- Lack comparaison to other RLHF methods for text to image methods such as Diffusion DPO, and other methods. Indeed this method only compare to none RLHF methods. This method should be compared on Pixart Alpha to have a fair comparaison\n- One of the advantages of the method is that it's online. However, if we follow the trends of RLHF in LLMs they tend to be converted to offline methods with training datasets for better efficiency. Would this method work on an offline manner? Or even with a replay buffer for faster iterations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to perform RLHF on one step generators. To do so they propose a data free formulation that allow for efficient tuning of a model towards certain rewards. They achieve competitive results aligning Pixart Alpha towards the Image Reward", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "HEre are the changes prioritize from most critical to least critical:\n- Compare to other RLHF methods\n- Compare on standard Image generation metrics\n- Improve paper qualitative results\n- Multi step diffusion\n- Evolution of the different metrics during training (Supp mat maybe)\n- Offline", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0nt2CdSq6N", "reviewer": "Reviewer_v88H", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thanks the authors for the great work on the text-to-image generation model.\nStrengths: Besides the technical contributions mentioned above, the paper is also well-polished with adequate theoretical proof. I guess the authors put effort into that.\n\nWeakness: \nThough overall a great work, I still think there is some space to improve.\n1. Since the Diff-Instruct++ requires a reference diffusion model to initialize, I don't think it's a fair comparison, like in Table 1, in which you compared the DI++ with SDXL-Lightning, SD15-TCD, etc, which initialized from another pre-trained model. Since DI++ is compatible with different reference diffusion models if I understand correctly, the initialized model should be kept the same. For example, if you want to compare DI++ with SD15-TCD, the DI++ should also initialize from SD15.\n\n2. The paper provides some quantitative metrics like Image Reward, Pick Score, etc. However, there are still some other datasets or metrics proposed in these two years for human preference evaluation, like the HPS dataset [1], and the MPS dataset [2] (not sure whether they released their dataset. If not, then forget about it). I would like to see the performance of DI++ on these datasets.\n\n3. The authors mainly compare DI++ with some few-step text-to-image generation methods, which kind of make sense. However, since another target is to generate images aligned with human preference, it's better to compare with some human preference-aligned text-to-image generation methods like [1,2,3,4], although they require multi-steps during generation. If the authors can demonstrate that DI++ can get competitive results with these multi-step human preference-aligned text-to-image generation methods, while the training and inference time is much less, it will strengthen the paper a lot.\n\n[1] Xiaoshi Wu, etal. Better Aligning Text-to-Image Models with Human Preference, ICCV 2023.\n[2] Sixian Zhang, etal. Learning Multi-dimensional Human Preference for Text-to-Image Generation, CVPR 2024.\n[3] Xiaoshi Wu, etal. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis.\n[4] Jiazheng Xu, etal. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation, NIPS 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method for one-step text-to-image generation that aligns with human preference. Leveraging the knowledge of reinforcement learning using human feedback (RLHF), the authors attempt to maximize the expected human reward functions and introduce a fast-converging and image data-free human preference alignment method for one-step text-to-image generators. Through diffusion distillation and CFG, the experimental results show their superiority results on some human preference metrics compared with other few-step text-to-image generation models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I highly recommend the authors to solve my 1st and 3rd concerns in the weakness part.", "broader_impact_concerns": "If the author can make their code public available, it will be beneficial to the whole community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While there was limited consensus between reviewers with respect to the breadth of impact of the method, they generally agreed that the paper was correct, and contained enough novel elements and empirical evidence to warrant acceptance. A noted weakness was a lack of investigative experiments ("_why_ does this method work?"), which was counterbalanced by a large number of additional settings and comparative experiments performed during the rebuttal phase.
Decision: Accept (poster)