Reward Guided Latent Consistency Distillation
Authors: Jiachen Li
Weixi Feng
Wenhu Chen
William Yang Wang
Venue: Accepted by TMLR ()

Abstract: Latent Consistency Distillation (LCD) has emerged as a promising paradigm for efficient text-to-image synthesis. By distilling a latent consistency model (LCM) from a pre-trained teacher latent diffusion model (LDM), LCD facilitates the generation of high-fidelity images within merely 2 to 4 inference steps. However, the LCM's efficient inference is obtained at the cost of the sample quality. In this paper, we propose compensating the quality loss by aligning LCM's output with human preference during training. Specifically, we introduce Reward Guided LCD (RG-LCD), which integrates feedback from a reward model (RM) into the LCD process by augmenting the original LCD loss with the objective of maximizing the reward associated with LCM's single-step generation. As validated through human evaluation, when trained with the feedback of a good RM, the 2-step generations from our RG-LCM are favored by humans over the 50-step DDIM samples from the teacher LDM, representing a 25-time inference acceleration without quality loss.

As directly optimizing towards differentiable RMs can suffer from over-optimization, we take the initial step to overcome this difficulty by proposing the use of a latent proxy RM (LRM). This novel component serves as an intermediary, connecting our LCM with the RM. Empirically, we demonstrate that incorporating the LRM into our RG-LCD successfully avoids high-frequency noise in the generated images, contributing to both improved Fréchet Inception Distance (FID) on MS-COCO and a higher HPSv2.1 score on HPSv2's test set, surpassing those achieved by the baseline LCM.

Project Page: https://rg-lcd.github.io/

Published in Transactions on Machine Learning Research (10/2024)
Reward Guided Latent Consistency Distillation
Jiachen Li
jiacknn Riccs Rt.cs.ussb..... edu
University of California, Santo. Barbara
Weixi Feng
weinifeng@es. wew.:/swwss.sss.oooo edu
University of California, Santa Barbara
Wenhu Chen
wenhu, chen
University of Waterloo
William Yang Wang
william www..uc.b.... edu
University of California, Santa Barbara
Reviewed on OpenReview: https: openreviee.......... net/ forum? 1d=
Abstract
Latent Consistency Distillation (LCD) has emerged as a promising paradigm for efficient text-lo-image senthesis. By distilling a latent consistency model (L.C)) from a pre-trained teacher latent diffusion model (LDM)), LCD facilitates the generation of high-fidelity images within merely 2 to inference steps. However, the LCM's efficient inference is obtained at the cost of the sample quality. In this paper, we propose compensating the quality loss by aligning LCM's output with human preference during training. Specifically, we introduce Reward Guided LCD (RG-LCD), which integrates feedback from a reward model (RM) into the LCD process by augmenting the original LCD loss with the objective of maximixing the reward associated with LCM's single-step generation. As validated through human evaluation, when trained with the feedback of a good RM, the 2-step generations from our RG-LCM are favored by humans over the 50-step DDIM (Song et al., 2020a) samples from the teacher LDM, representing 25-time inference acceleration without quality loss.
As directly optimizing towards differentiable RMs suffer from over-optimization, take the initial step to overcome this difficulty by proposing the use of latent proxy RM (LRM). This novel component serves as an intermediary, connecting our LCM with the RM. Empirically, we demonstrate that incorporating the LRM into our RG-LCD successfully avoids high-frequency noise in the generated images, contributing to both improved Fréchct Inception Distance (FID) on MS-COCO (Lin et al. 2014) and a higher HPSv2.1 score on HPSv2 (Wu al. 2023a 's set, surpassing those achieved by the baseline LCM.
Project Page: https: //tg www.ll.....hbiii..
1
Introduction
In the realm of modern generative Al (GenAl) models, computational resources are typically allocated across three key areas: pretraining (Brown et al.
2020 Achiam et al.
2023a Li al. 2022b Radford et al.) 2021
Rombach et al. 2022
Saharia et al.) 2022 Betker al. 2023a])
alignment (Ziegler et al. 2019 Stiennon et al, Rafailov ct al. 2024h, and inference (Zhang
2020 Ouyang et al. 2022: Hu et al. 2024 Clark et al.
2023
Shih et al.] 2024). 3 Normally, increasing the computational budget across these areas leads to improvements in sample quality. For instance, the most advanced text-to-image (T2I) models, such as DALLE-3 (Betker et al., (023a), Imagen (Saharia al., 2022). and Stable Diffusion (Rombach et al.) 2022) are built from diffusion models (DMs) (Sohl-Dickstein
& Chen 2022 Feng et al. 2023 Vijayakunar el al. 2016
al. 2015 Ho et. al.] 2020 Song & Ermon 2019 These models pretrained massive web-scale

---

Published in Transactions on Machine Learning Research (10/2024)
50-step
SD SD v2.1
2-step
LCM
RG-LCM (Ours)
CLIP Score
LU
RG-LCM (Ours)
HPS v2.1
4-step
LCM
RG-LCM (Ours)
CLIP Score
WUFFM
RG-LCM (Ours)
HPS v2.1
Figure 1: Even with merely 2-4 sampling steps, our RG-LCMs that learned from the CLIP Score and 1PSv2.1 1 can produce high-quality images.
2

---

Published in Transactions on Machine Learning Research (10/2024)
datasets (Schuhmann et al.)
2022 Changpinyo et al. 2021 aligned with human preference curated high-
quality images (Dai et al. 2023
Rombach et al. 2022), and benefit. from DMs' iterative sampling process.
However, DM's iterative sampling requires performing 10 2000 sequential function evaluations (FEs) (Ho
ct al. 2020 Song et al. 2020a): thus impeding rapid inference. While there have been many works proposed
to address this issue (I.u et al. 2022a bb Zhang et al. 2023 Sauer et al. 2023; Geng et al. 2024 Nguyen
& Tran 2023 Song et al. 2023), consistency model (CM) [Song et al. 2023) emerges as new family of
GenAl model to facilitate fast sampling. Specifically, a CM is trained to perform single-step generation while
supporting multi-step sampling to trade compute for sample quality, We can distill CM from pretrained
DM, a process known as consistency distillation (CD). For instance, [uo et al. (2023a) distill Latent CM
(LCM) from a pretrained Stable Diffusion (Rombach et al., 2022), achieving high-fidelity image generation
in just 2 to 4 FE steps. However, the sample quality of LCM is inherently constrained by the pretrained
LDM's capabilities (Song & Dhariwal 2023 Additionally, the reduced inference computational resources stemming from the limited number of FE steps compromise LCM's sample quality.
In this paper, we aim to offset LCM's sample quality by dedicating additional computational resources to the training process. Recent advancements in large language models (Achiam et al. 2023b Team et al. 2023) have shown that aligning GenAI model with a reward model (RM) that mirrors human preferences can substantially improve sample quality by reducing undesirable outputs (Ouyang et al., 2022 Rafailov et al. 2024). Thus, we are motivated to align the learned LCM with human preferences by optimizing towards off-the-shelf text-image RMs. Instead of designing a separate aligument phase, we leverage the single-step generation that naturally arises from computing the LCD loss and implement training objective maximize its associated rewards given by a differentiable RM through gradient descent. Notably, our approach obviates the need for backpropagating gradients through the complicated denoising procedures, which is typically required by previons methods when optimizing a DM (Clark et al., 2023 Xu et al. 2024 Prabhudesai et al. 2023 Yang et al., 2024). We dub our method Reward Guided Latent Consistency Distillation (RG-LCD). Human evaluation shows that our RG-LCM significantly outperforms the LCM derived from standard LCD. Remarkably, our 2-step generation is favored by humans over the 50-step generation from the teacher LDM,
representing a 25-fold inference acceleration without compromising image quality.
While our RG-LCD is conceptually simple and already achicves impressive results, it can suffer from reward overestimation (Kim et al.; 2023b: Zhang el al., 2024) due to direct optimization with the gradient from the RM. As shown in the top row of Fig. 3 performing RG-LCD with ImageReward (Xu et al.) 2024) causes high-frequency noise in the generated images. In this paper, we take an initial step to tackle this challenge. We propose learning a latent proxy RM to serve as the intermediary that connects our LCM with the RM. Instead of directly optimizing towards the RM, we optimize the LCM towards the LRM while finetuning the LRM to match the preference of the expert RM in each RG-LCD iteration. This novel strategy allows us to optimize the expert RM indirectly, even allowing for learning from non-differentiable RMs. We empirically verify that incorporating the LRM into our RG-LCD successfully eliminates the high- frequfrey noise in the generated image, contributing to improved FID on MS-COCO (Lin et al. (0044] and higher HPSv2.1 score on HPSv2's test set (Wu et al. 2023a), purperrrningg the baseline LCM.
In summary, our contributions are threefold:
Introduction of RG-LCD framework, which incorporates feedback from an RM that mirrors human preference into the LCD process.
Introduction of the LRM, which enables indirect optimization towards the RM, mitigating the issue of reward over-oplimization.
A 25 times inference acceleration over teacher LDM (Stable Diffusion v2.1) without compromising sample quality.
2 Related Work
Accelerating DM inference. Centering around DM's SDE formulation (Song et al. 2020b), various methods have been proposed to accelerate the sampling process of DM. For example, faster numerical
3

---

Published in Transactions on Machine Learning Research (10/2024)
ODE solvers (Song al. 2020a
Lu al.
2022a b Zheng et al. 2022 Dockhorn et al. 2022
Jolicoeur-
Martineau et al. 2021 and distillation techniques Luhmaa & Luhman 2021 Salimans & Ho 2022 Meng
et al. 2023 Zheng et al. 2023).
Recent advances explore enhancing the single-step generation quality by incorporating an adversarial loss [Sauer ct al. 2023) or by distillation (Nguyen & Tran 2023)
Consistency 2023) is also trained for single-slep generation. We leverage this property and directly maximize the reward of this single-step generation given by differentiable RM, avoiding the complexities of backpropagating gradients through the iterative sampling process of DM.
Model (Song et al.
Consistency Model has emerged a new family of GenAl model (Song al. 2023 that facilitates fast inference. While it is trained to perform singlo-step generation by mapping arbitrary points in the PF-ODE trajectory to the origin, CM also supports multi-step sampling, allowing for trading compute for better sample quality. On the one hand, a CM can be trained as a standalone GenAI model (consistency training). Recently, Song & Dhariwal (2023) proposed improved techniques to support better consistency training. On the other hand, a CM can also be distilled from a pretrained DM (Kim al., 2023a). For instance, Luo
eet al. (2023a) learn an LCM by distilling from a pretrained Stable Diffusion (Rombach al. 2022 We
defer more technical details to Sec. 3
Vision-and-language reward models. Motivated by the significant success of reinforcement from human leedback (RLHF) in training the LLMs, there have been many works delving into training an RM to mirror human preferences on a pair of text and image, including HPSv1 (Wu et al. 2023b). HPSv2 (Wu et al. 2023a ImageReward (Xu et al. 2024),, and PickScore (Kirstain et al. 2024). These RMs are normally derived by finetuming a vision-and aade--nd--lnnng and-langugge foundation model, e.g., CLIP (Radford et al., 2021) and BLIP (Li et al, 2022a), on human preference data. Since these RMs are differentiable, our RG-LCD augments the standard LCD with the objective of maximizing the differentiable reward associated with its single-step generation during training.
Aligning DMs to Human preference has been extensively studied recently, including RL based meth- 2024 Prabhudesai et al.) 2023 Zhang et al. 2024) and reward finetuning methods (Clark
ods (Tan et al. ett al.]
2023 Xu et al., 2024). Recently, Diflusion-DPO (Wallace et al. 2023a) is proposed by extending DPO Rafailov et al., 2024) to train DMs on preference data. Moreover, other works focus on modifying
the training data, distribution (Wu et al.] 2023b Lee et al. 2023 Dong et al. 2023 Sun et al., 2023 Dai
et a..
2023 to finetune DMs on visually appeellig and lextually cohered data. Additionally, alternative
techniques (Betker et al. 2023b Segalis et al.] 2023) re-caption pre-collected web images to enhance text accuracy. On the other hand, DOODL (Wallace et al. 2023b) is proposed to optimize the RM during in- ference time. However, its improvement is made at the cost of inference speed. While we also propose to directly finetune our model with the gradient given by an RM, finetuning an L.M during LCD is much simpler than finetuning a DM, as we only tackle the single-step generation, circumventing the need to pass gradients through the complicated iterative sampling process of a DM.
3 Background
3.1
Diffusion Model
Diffusion models (DMs) (Sohl-Dickstein et al. 2015 Song & Ermon 2019
Ho et al. 2020 Nichol & Dhariwal
2021 progressively inject Gaussian noise into data in the forward process and sequentially denoise the data to create samples in the reverse denoising process. The forward process perturbs the original data distribution Pdata (x) = px(x)) to the marginal distributional (x:). From a continuous- time perspective, we can represent
the forward process with stochastic differential equation (SDE) (Song et al., 2020b Karras et al., 2022)
dx, = ptt)x;dtt + (t)dwa Xo ~ Pdata (Xo)
(1)
where pa(-) and (-) are the drift and diffusion coefficients respectively, and w, denotes the standard Wiener process. The reverse time SDE above corresponds to an ordinary differential equation (ODE) (Song et al. 2020b), named Probability Flow (PF-ODE), which is given by
dx, = (1) (1)))V log (xx) dt, XT ~ pr(x))-
(2)
4

---

Published in Transactions on Machine Learning Research (10/2024)
PF-ODE's solution trajectories sampled at t are distributed the same px())). Empirically, we learn demoising model <g(x.. 1) to fit. V log (x+) (score function) via score matching (Hyvärinen & Dayan 2005
Song & Ermon 2019 Ho et al. 2020). During sampling, we start from the sample XT ~ (N 8²1) and
follow the empirical PF-ODE below
1 (t) X: - ) dt, xT N(0,00,
dx.-
(3)
In this paper, we focus on conditional LDM that operates on the image latent space 2 and includes text prompt с passed to the denoising model 6a(Z,, c, (), where 2, - {x,) E 2 is encoded by a VAE (Kingma et al. 2021 encoder E. Moreover, we utilize Classifier Free Guidance (CFG) (Ho Salimans 2022 to improve the quality of conditional sampling by replacing the noise prediction with linear combination of conditional and unconditional noise prediction for denoising, i.c., Eg (z, c, = (1 + weo (z,, c,t) weg(z, 2,t), where is the CFG scale.
3.2 Consistency Model
Consistency model (CM) (Song et al. 2023 is proposed to facilitate elficient generation. At its core, CM learns a consistency function f (x,,t) t) x. that can map any point x2 the same PF-ODE trajectory to the trajectory's s origin, where is fixed small positive number. Learning the consistency function involves enforcing the
property
f(x,, f(x,, 1'), VL, [e,
(4)
where x2 and x/ belong to the same PF-ODE. The consistency function f is modeled with a CM fo. To ensure fx( e) x, So is parameteried as
fx( t) Cakip (t)x + Cout (t),
(5)
where Cskip (f) and Cout(t) (t) are differentiable functions with Cskip (c) 1 and Cont(c) 0, and Fø is neural network. We can learn a CM Se by distilling from pretrained DM, known consistency distillation
(CD) (Song et al. 2023). The CD loss is given by
LCD (0,0 :
Ex.e [d (Fo (Xtm411(a41) , fo-
kn).
(6)
where :( measures the distance between two samples. 0- is the parameter of the larget CM fe-, updated by the exponential moving average (EMA) of 0, ie., 0- +- stop_grad (nD + -=)00 x is estimation of xt, from Xt,-ı using the one-step ODE solver o:
+ Xl,+1 + (tn - (x+1) &b (xt.41: f)
Note that the parameter á corresponds to the parameter of the pretrained DM, which is used to construct the ODE solver Φ. We includes the algorithm for sampling from learned CM in Appendix B
3.3 Latent Consistency Model
Luo al. (Luo et al. Specifically, a Latent CM (LCM) fo (Ze, c, zo is trained to minimized the LCD loss
2023a extends CM to work on latent space 2 and focuses on conditional generation.
LLCD (0,0 v) Ez,,,,, ( fo w. c, tn+k) fo (2%."" ,w, c, c, ).]
(8)
where 2) an estimate of 2t,, obtained by the numerical augmented PF-ODE solver and l is skipping interval. L
N.
Zentt + +1 + w) (Zt, tn+k tn, v) w WV tntk, tn Ø; v).
(9)
In this paper, we use DDIM (Song et al. 2020a) as the ODE solver v to distill from pretrained Stable Diffusson (Rombach al. 2022) and refer interested readers the original LCM paper for formula of the DDIM solver. We use huber loss as our distance function d(
5

---

Published in Transactions on Machine Learning Research (10/2024)
Forward diffusion
fe
E
D
R R J(0)
Zt n
fo (tntt w, c, tn+k)
Reward
ODE T
Photo <PERSON> on a mounlain
Solver
LLCD (0, (W, Distillation Loss
fo
Gradient from the RM Gradient from the LCD
V 2 In
(2 W.w fg
w C,
Figure 2:
Overview of our RG-LCD. We integrate leedback from a differentiable RM into the standard LCD procedures by training the L.C to maximize the reward associated with single-step generation..
4 Reward Guided Latent Consistency Distillation
In this section, we start by presenting the core components our RG-L.CD framework, which augments the standard LCD loss equation with an objective towards maximizing differentiable RM, as shown in Fig. 2 (Sec. 4.1.. We then motivate the development of latent proxy RM (LRM) to support indirect RM oplimization by illustrating the risk of sulfering from reward over-oplimization when directly optimizing towards the RM with gradient-based method. Following this, we then detail the procedure to pretrain and finetune the LRM to match the preference of the RGB-based RM during RG-LCD (Sec. 4.2)
4.1 RG-LCD with Differentiable RMs
Recall that each LCD iteration samples timestep Eatt and construct the noisy latent Zt,-x by perturb the image latent. z = {x) with Gaussian noise, given a sampled CFG scale w and text prompt c. As the LCM fo maps the zt to the PF-ODE origin 2n = fo (Zt W, in=k), we construct the following objective to maximize the reward associated with D(2o)
J(() Ez,,,,,n [R (D (fo (Zr, in c, tn)))))) ,c]]
(10)
where R is differentiable RM that calculates the rewards associated with pair of text and image. We define the training loss of our RG-LCD by linear combination of the L.CD loss in equation and J(Ø) with weighting parameter
LRG-LCD (0, ; D) LL.D (0,0 3J(
(11)
Appendix
B includes pseudo-codes for our RG-LCD training.
4.2 RG-LCD with a Latent Proxy RM
When training the LCM fo towards (0) with a gradient-based method, we may suffer from the issue of reward over-optimization. As shown in the top row of Fig, performing RG-LCD with ImageReward (Xu et al. 2024) causes high-frequency noise in the generated images. To mitigate this issue, we propose learning latent proxy RM R. to serve as an intermediary to connect fo and the expert RGB-based RM RE where the stands for "Expert". Specifically, we train fo to optimize the reward given by R. while simultaneously finetuning the R.F to matches the preference given by the expert RM R that process RGB images.
Ideally, the LRM R/ should be capable of accessing the text-image pair even at the beginning of RG-LCD. We thus initialize R. with a pretrained CLIP (Radford et al. 2021) text encoder, complemented by pretraining its latent encoder from scratch. This latent encoder is pretrained following the same methodology used for CLIP visual encoders, ensuring it aligns effectively with the text encoder's representation.
After pretraining, we finetune Rh match the preference of R Note that we do not need to assume differentiable RE anymore, allowing us to learn from the feedback from wider range of RGB-based

---

Published in Transactions on Machine Learning Research (10/2024)
fluffy sentient Axor, androneda galaxy have large innocent eyes digitigrade, high detail.
astronaut riding
Figure 3: (Top) Optimizing the RG-LCM with the gradient from ImageReward (xu et al. 2024 results in high-frequency noise the generated images. (Bottom) Indirectly optimizing the ImageReward through the latent proxy RM eliminates the high-frequency noise, avoiding reward over-optimization.
RM, e.g., LLMScore [u et al., 2024), VIEScore (Ku al. 2023) and DA-score (Singh & Zheng 2024
Next, will derive the finetuning loss for our LRM(O). Given Zo z, Z1 fo (Zi W, C, (a+k): and So- (Zt,, w, c,4n) in each RG-LCD iteration, (2). We then use Rh and R to compute the rewards for each latent. For each latent pair (z, zy), the probability of Re preferring over 2, is modeled as:
group them into three pairs: (zo, Zi), (Zo, 22) and
exp (Rl (z,, c) /TL) (1) exp (Rl (z, /TL) exp (Rl (z,, /TL)
is the temperature parameter. Similarly, with the temperature the probability of E preferring over 2j can be modeled as:
exp (R (D (z:) c) /TT)
(1,(1) exp (RE (2,),, /TE) + exp (R (D (D())), /TE)
And thus, we have
(m) & exp ! (zm: c) 2)/TL) TL), Qi)(m) & exp (R (zm) /TE) m E {i.s}.
We can construct the KL divergence between the distribution Pe and (i,; for each (2,, 23) pair. Our LRM (a) derived by summing the KL divergence for all three latent pairs as below
DKL (Pr, |stto_grad (Qt,))
LaMa(o) = Ez,,,nn
(12)
İ= j=111
Appendix B.3 includes pseudo-codes for training our RG-LCM with an LRM in Algorithm 4 supports matching R that only output preference two images. Tn this case, we set TE small
LRM(O) also

---

Published in Transactions on Machine Learning Research (10/2024)
Parti Prompts: General Preference
Parti Prompis: Visual Appeal
Parti Prompts: Prompt Aligament
(CLIC
43.4
#15
step
50 steps
(CLIP
step
50 steps
RG-LCM (CLIP)
52.8
58.3
63.0
LCM steps
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
RG-LCM (IIPS)
53.8
5BA
53.1
SD
50 steps
RG-LCM (ITPS)
54.7
61.7
53.6
step
steps
RG-LCM (HPS)
766.7
step
steps
20
40
60
100
20
100
Preference (%)
Proference [%)]
Preference [%)
Figure 4: Human evaluation results on the PartiPrompt (1632 prompts) across three evaluation questions. Top row evaluates the RG-LCM (CLIP). Bottom row evaluates the RG-LCM (HPS).
positive number and only give a non-zero positive reward to the sample favored by the expert. Moreover, since 2n = z corresponds to the latent a real image, we can increase likelihood for Qo. to prefer 0.
While calculating (D)(z)) c) still requires decoding the latent, the application of the stop_gras operation eliminates the need for gradient transmission through D, leading Moreover, optimizing Rl with (o) is independent from optimizing fa with TRG-LCD- Therefore, use smaller batch size to optimize R. without affecting the batch size used to optimize fo.
substantial reduction in memory usage.
In essence, our LRM acts as proxy connecting the LCM fo and the expert RM R. As will show Sec. 5.2 using this indirect feedback from the expert mitigates the issue of reward over-optimization, avoiding figh-fregeeccy noise in the generated images.
5
Experiment
We perform thorough experiments to demonstrate the effectiveness our RG-LCD. Sec. 5.1 conducts
human evaluation to compare the performance of our methods with baselines. Sec. 5.2 further increases the
experiment scales experiment with wider array of RMs with automatic metrics. By connecting both evaluation results, identify problems with the current RMs. Finally, Sec. 5.3 conducts ablation studies critical design choices.
Settings Our training are conducted on the CC12M datasets (Changpinyo et al. 2021 , as the LAION- Aesthetics datasets (Schuhmann et al. 2022) used by the original LCM (Luo el al. 2023a) are no longer accessible We distill our LCM from the Stable Diffusion-v2.1 (Rombach et al. 2022) by training for 10K iterations on 8 NVIDIA A100 GPUs without gradient accumulation and set the batch size to reach the maximum capacity of our GPUs. We follow the hyperparaneter settings listed in the diffusers (von Platen et al. 2022 library by setting learning rate le 6, FMA rate 1 0.95 and the guidance scale range Wmin Wmax [5, 15]. As mentioned in Sec. 3.3 use DDIM (Song et al., 2020a)
our ODE solver W with a skipping step k = 20. We include more training details in Appendix A Appendix D further includes experiment results with diverse teacher T2I models, including Stable Diffusion 1.5 and Stable Diffusion XL.
5.1 Evaluating RG-LCD with Human
We train RG-LCM (HPS) and RG-LCM (CLIP) utilizing feedback from HPSv2.1 Wu et al. 2023a) and CLIPScore (Radford et al. 2021), respectively. CLIPScore evaluates the relevance between text and images, whereas HPSv2.1, derived by fine-tuning CLIPScore with human preference data, is expected to mirror
https://laion wwwp:/as/aaaaannnaaeeeeeaaeaeoee
8

---

Published in Transactions on Machine Learning Research (10/2024)
HPSv2 Test: General Preference
HPSv2 Test: Visual Appeal
HPPV2 Test: Prompt Alignment
45.0
2 step
50 steps
(CLIP) 4 step
50 steps
RG-LCM (CLIP)
81.7
78-4
81.6
LCM steps
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
RG-LCM (IIPS)
52.0
65.6
$1.8
SD
50 steps
RG-LCM (ITPS)
57.1
86.3
55.1
step
steps
RG-LCM (HPS)
69.0
72.6
LCM
step
steps
20
40
60
100
20
100
100
Preference (%)
Proference [%)]
Preference [%)
Figure 5: Human evaluation results on the HPSv2 test set (3200 prompts) across three evaluation questions. Top row evaluates the RG-LCM (CLIP). Bottom row evaluates the RG-LCM (HPS).
human preferences more accurately. We choose the teacher LDM (Stable Diffusion v2.1) and standard LCM distilled from the same teacher LDM as the baseline methods. To demonstrate the efficacy of our methods, we compare the performance of our RG-LCMs over 2-step and 4-step generations against the 50- step generations from the teacher LDM and evaluate the 4-step generation quality of our RG-LCMs against the standard LCM.
We follow a similar evaluation protocol as in (Wallace et al., 2023a to generate images by conditioning on prompts from Partiprompt (Yu al.) [202)) (1632 prompts) and of HPSv2's test set (Wu al.) 2023a (3200 prompts). We hire labelers from Amazon Mechanical Turk for head-to-head comparison of images based three criteria: Q1 General Preference (Which image do you prefer given the prompt?), Q2 Visual Appeal (Which image is more visually appealing, irrespective of the prompt?), and Q3 Prompt Alignment (Which image better matches the text description?).
The full human evaluation results Fig. and 5 show that the 2-step gencrations from RG-LCM (CLIP) are generally preferred (Q1) over the 50-step generations of the teacher LDM in both prompt sets, representing 25-fold acceleration in inference speed. Even with CLIPScore feedback, the 4-step generations from our RG-LCM are generally preferred (Q1) over the bascline methods. This indicates noteworthy achievement, given that CLIPScore does not train human preference data. Surprisingly, the HPSv2 prompt set, the 4-step generations from the RG-LCMM (CLIP) are more preferred (59.4% against 50-step DDIM samples from SD and 81.7% against 4-step LCM samples) compared to the 4-step generations of the RG-LCM (CLIP) (57.1% against 50-step DDIM samples from SD, and 69.0% against 4-step LCM samples).
To investigate this phenomenon, we observe that both RG-LCMs score similarly General Preference (Q1) and Prompt Alignment (Q3). However, the RG-LCM (CLIP) is rated slightly lower in Visual Appeal (Q2) than in the other criteria, whereas the RG-LCM (HPS) is rated significantly higher for Q2 compared to Q1 and Q3. This distinction highlights that CLIPScore's primary contribution is enhancing text-image alignment, whereas an RM like HPSv2.1 particularly focuses on improving visual quality. Thus, when over- optimizing towards HPSv2.1, the RG-LCM (HPS) can be biased in generating visually appealing samples by sacrificing prompt alignment.
5.2 Evaluating RG-LCD with Automatic Metrics
In this section, we further train RG-LCD (IngRwd) and RG-LCD (Pick) by leveraging feedback from ImageReward [uu al.) 2024) and PickScore (Kirstain et al.] 2024). Both of these RMs are trained on human preference data. We will use automatic metries to perform large-scale evaluation of the performance of different models. As have human evaluation results for RG-LCD (HPS) and RG-LCD (CLIP),
9

---

Published in Transactions on Machine Learning Research (10/2024)
Models
NFEs
Human Preference Score v2.1 f
FID-30K + MS-COCO
Anime
Photo
Concept-Art
Paintings
LCM
4
22.40
19.17
18.86
20.55
19.05
Stable Diffusion v2.1
50
25.66
24.37
24.58
25.72
12.66
RG-LCM (CLIP)
2
26.32
25.01
25.27
26.71
18.06
RG-LCM (CLIP)
4
27.80
26.92
27.04
28.11
19.22
RG-LCM (Pick)
2
26.44
28.26
28.24
29.04
22.84
RG-LCM (Pick)
4
27.33
29.42
29.29
30.26
22.02
RG-LCM (Pick) LRM
2
23.82
21.31
21.90
22.99
15.91
RG-LCM (Pick) + LRM
25.17 4
23.06
22.90
24.87
16.27
RG-LCM (ImgRwd)
2
29.65
31.03
31.15
32.00
32.12
RG-LCM (ImgRwd)
4
30.26
31.83
31.88
32.73
42.69
RG-LCM (ImgRwd) + LRM
2
25.64
25.61
25.82
25.75
17.57
RG-LCM (ImgRwd) LRM
4
26.84
26.72
26.72
27.30
17.20
RG-LCM (HPS)
2
30.85
33.66
33.35
33.66
24.04
RG-LCM (HPS)
4
31.83
34.84
34.43
34.75
25.11
RG-LCM (HPS) + LRM
2
27.58
25.94
26.77
27.24
16.71
RG-LCM (HPS) + LRM
4
28.53
27.49
27.94
28.87
17.52
Table 1: Evaluation of RG-LCMs the HPSv2 test. prompts and MS-COCO datasets. NFEs denote the number of function evaluations during inference. We train RG-LCMs with CLIPScore, PickScore, Im- ageReward (ImgRwd) and HPSv2.1. We employ the HPSv2.1 to evaluate the generations on the HPSv2 Benchmark's 's lest set. We calculate the FID of the generations on the MS-COCO. Except trained with CLIPScore, our RG-LCMs achieve better HPSv2.1 scores on HPSv2 test prompts at the expense of higher FIDs on MS-COCO. Integrating a LRM into our RG-LCD process allows for simultaneous improvement on HPSv2.1 scores on HPSV2 test prompts and FID on MS-COCO against the baseline LCM.
also evaluate the quality of the automatic metrics. For each RG-LCD, we collect their 2-step and 4-step generations by conditioning on prompts from HPSv2's test set and measuring the HPSv2.1 score associated with the samples. To comprchensively understand the sample quality from different modcls, we further generale images conditioned the prompts of MS-COCO [Lin et. al. 2014) and measure their Fréchet Inception Distance (FID) to the ground truth images.
Table presents the full evaluation results with the automatic metrics. Except for RG-LCM (CLIP), all the other RG-LCMs achieve higher HPSv2. scores than the baseline LCM but the expense of higher FID values on the MS-COCO dataset. Specifically, the RG-LCM (IngRwd) model exhibits notably high FID value, yet it still secures an impressive HPSv2.1 score when evaluated HPSv2 test prompts. The elevated FID value aligns with expectations, as Figure 3 Dillustrates that optimization directed towards ImageReward tends to introduce significant amount of high-frequency noise into the generated images. Surprisingly, these high-frequency noises do not adversely affect the HPSv2.1 scores. Furthermore, the HPSv2. scores do not capture the human preference for the 4-step samples from RG-LCM (CLIP) by giving the highest score to RG-LCM (HPS)'s 4-step samples, contrary to what depicted in human evaluation shown Fig.
These observations suggest that the HPSv2.1 score, as metric, has limitations and requires further refine- ment. We conjecture that the Resize operation, which happens during the preprocessing phase, causes the HPSv2.1 model to overlook the high-frequency noise during reward calculation. As illustrated in Fig. the high-frequency noise becomes less perceptible when images are reduced in size. Although resizing operations enhance efficiency in lasks such as image classification (Lu & Weng 2007 Deng et al. 2009 He et al. 2016) and facilitate high-level text-image understanding (Radford et al. 2021), they prevent the model from cap- turing critical visual nuances that are vital for accuratoly reflecting human preferences. Consoquently, we advocate for future RMs exclude the Resize operation. One potential approach could involve training LRM, as in paper, to learn human preferences in the latent. space without resizing input images.
10

---

Published in Transactions on Machine Learning Research (10/2024)
LCM
RG-LCM
dog is reading thick book
Mountains range with waterfall, purple by greg nutkowski and magali villeneuve, artstation.
photo beautiful mountain with realistic and blue lake,, highly detailed, masterpiece
HPSv2.1 #=1
HPSv2.1 100
PickSoore B f=5
PickScore B = 50
CLIPScore =
CLIPScore 100
Figure Ablating the reward scale 3 for different reward functions. All samples are generated with steps. We observe that over-optimizing RMs trained with preference data prioritizes visual appeal over lext alignment, whereas over-optimizing CLIPScore compromises visual attractiveness in favor of text alignment.
Connecting Table ] with the human evaluation results in Fig. 5 suggests that images that achieve high HPSv2.1 score and low FID on MS-COCO are more aligned with human preferences. Morcover, this desirable outcome can be accomplished by integrating an LRM into our RG-LCD, Although these correlations do not imply causality, they underscore the potential benefits of utilizing an LRM in the RG-LCD process. As depicted the bottom row in Fig. the images generated by RG-LCM (IngRwd) that integrates an LRM do suffer from high-frequency noise, contributing to their improved FID on MS-COCO. In Appendix C we include additional samples for each RG-LCMM in Table
5.3 Ablation Study
Ablation on the reward scale 3. We use the hyperparameter 3 to determinc the optimization strength towards the RM. We are especially interested the impact of an extremely large 8, which can lead reward over-optimization (Kim et al. 2023b).. We already know that over-optimizing the ImageReward can lead to the introduction of high-frequency noise in the generated images. To expand our understanding, conduct experiments wider array of RMs including HPSv2.1, PickScore and CLIPScore and evaluate whether over-oplimizing these RMs will also leads to similar high-frequency noise.
The results in Figure reveal that an extremely large 3 value does not introduce the high-frequency noise when using 1, PickScore, and CLIPScore, even though all these metrics resize input images to 224x224 pixels as in ImageReward. Notably, over-optimization of HPSv2.1 leads to generating images with repetitive objects as described in the text prompts and increases color saturation. Conversely, over-optimization of PickScore tends to result in images with more muted colors. On the other hand, excessive optimization of CLIPScore results in images where the text prompts are visibly incorporated into the imagery. These findings align with the discussions in Sec. 5.1 suggesting that optimizing towards proference-trained RM generally prioritizes visual appeal over text alignment. In contrast, over-optimizing CLIPScore compromises visual attractiveness in lavor of text. alignment. We include additional image samples in Appendix C
Ablation on the training iterations. In total, we train each model for 10K iterations. We take chock- points from 1K, 2K, 4K, and 10K iterations and sample images with the same prompts and seeds. We can observe performing RG-L.DD with RMs that facilitate the visual appeal of the generated images also results
11

---

Published in Transactions on Machine Learning Research (10/2024)
1K
2K
4K
10K
IK
2K
4K
10K
1K
2K
4K
10K
RG-LCM (HPS), B
RG-LCM (Pick), / =
RG-LCM (CLIP), & =
Figure 7: Ablation study on the number of training iterations. We generate all samples with 4 steps. We observe that RG-LCM, which learns from an RM that prioritizes visual appeal, can generate high-quality images with fewer training iterations.
in fast training, as the checkpoint at the 2K iterations can already produce high-quality images. In rontrast the images generated by RG-LCM (CLIP) still generate blurry images after training for 2K iterations.
6 Conclusion
In this paper, we introduce RG-LCD, a novel strategy that integrates feedback from an RM into the LCD process. The RG-LCM learned via our method enjoys better sample quality while facilitating fast inference, benefiting from additional computational resources allocated align with human preferences. By evaluating using prompts from the HPSv2 (Wu al. 2023a) test and PartiPrompt (Yu et al., (022), we empirically show that humans favor the 2-step generations of RG-LCD (HPS) over the 50-step DDIM generations of the teacher LDM. This represents 25-fold increase in terms of inference speed without a loss in quality. Moreover, even when using CLIPScore- model not fine-tuned on human preferences our method's 4-step generations still surpass the 50-step DDIM generations from the teacher LDM.
We also identify that directly optimizing towards an imperfect RM, e.g., ImageReward, can cause high- frequency noise in generated images. To reconcile the issuc, we propose integrating an LRM into the RG- LCD framework. Notably, our methods not only prevents reward over-optimization but also avoids passing gradients through the VAE decoder and facilitates learning from non-differentiable RMs.
7 Limitation and Impact Statement
While our RG-LCD marks critical advancement in the realm of efficient text-to-image synthesis, introducing acceleration in the generation process without compromising image quality, it is important to recognize certain limitations. The approach relies on employing a reward model that reflects human preference, which, while effective in improving image quality metrics, may introduce additional costs in the training pipeline and nocessitate fine-tuning to adapt to various domains or datasets. Despite these challengos, the impact of RG-LCD profound, offering a scalable solution that significantly enhances the accessibility and practicality of generating sigh-fidelity images a remarkable speed. This innovation not only broadens the potential applications in fields ranging from digital for future research in text-to-image synthesis, emphasizing the importance of human-centric design in the development of generative AI Lechnologies.
visual content creation but also sets new benchmark
12

---

Published in Transactions on Machine Learning Research (10/2024)
Acknowledgement
The work was funded by unrestricted gift from Google. We would like to thank Google for their generous sponsorship. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the sponsors' official policy, expressed or inferred.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lgge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXit:2303.08774 2023a.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXit:2303.087 08774, 2023b.
James Betker, Gabriel Goh; Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/ /alles.- pdf, 2(3):8, 2023a.
James Betker, Gabriel Goh, Li Jing, TimBrooks, Jianfeng Wang, Linjie Li, LongOuyang, JuntangZhuang, JoyceLee, YufeiGuo, WesamManassra, PrafullaDharival, CaseyChu, YunxinJiao, and Aditya Ramesh. Improving image generation with better captions. 2023b. URL httpss//appi semanticscholar org/
CorpusID: 264403222
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Aduances in neural information processing systems, 33:1877-1901, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pro-training to recognize long-tail tail visual concepts. In Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognition, pp. 3558 3568, 2021.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differ- entiable rewards. arXiv preprint ar Xix:22200. 17400, 2023.
Xiaoliang Dai, Ji Hou, Chih- Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Van- denhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309. 15807, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. Advances in Neural Information Processing Systems, 35:30150 30166, 2022.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ar preprint arXiv:2304.06% 06767, 2023.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lec, and Kimin Lec. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero tree-scarch guide large language model decoding and training. arXiv preprint arXiv:2309.177 1109999997719911 17179, 2023.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Zhengyang Geng, Ashwini Pokle, and J Zico Kolter. One-step diffusion distillation via deep equilibrium models. Advances in Neural Information. Processing Systems, 36, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference on computer vision and pattern recognation, pp-
2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 --854,, 2020.
Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. Instruct-imagen: Image generation with multi-modal instruction. arXiv preprint ar arXic:2401.011 019552 2024.
Aapo Hyvärinen and Peter Dayan. Estimation non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Alexia
Ke Li, Rémi Piché Taillefer, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diflusion-based generative models. Advances in Neural Information Processing Systems, 35:26565 -6577,, 2022.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. aix preprint orXx:23310.0222 2023a.
Kyuyoung Kim, Jonghoon Jcong, Minyong An, Mohammad Ghavamzadeh, Krishnamurthy Dj Dvijotham, Jinwoo Shin, and Kimin Lee. Confidence-aware reward optimization for fine-tuning text-to-image models. In The Twelfth International Conference on Learning Representations, 2023b.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Advances in neural information processing systems, 1121696-217011 2021.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXie:2312. 200111..1116.. 14867, 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Moham- mad Ghavamzadeh, and Shixiang Shane Gu. Aligning toxt-to--mmae models using human feedback. arXiv preprint ar 2rX::2200.. 12192, 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022a.
Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jun Nic, and Ji-Rong Wen. Pretrained language models for text generation: A survey. Xiv preprinl ur urXin:2201.05273 2022b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Aision-ECOY 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling around 10 steps. Advances in Neural Information Processing Systems, 35:5775-577 2022a.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. D +-: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint orXiv:221 1. 1.01095. 2022b.
Dengsheng Lu and Qihao Weng. A survey of image classification methods and techniques for improving classification performance. International journal of Remote sensing, 28(5):823 870, 2007.
Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36, 2024.
Eric Luhman and Troy Luhman. Knowlodge distillation in itcrative generative models for improved sampling speed. aixiv preprind or 0r::::2001. 02388,, 2021.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint urXie:2310.04376 2023a.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. ar preprinl arXiv:2311.054 2023b.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusson model with variational score distillation. arXiv preprint arXio:2312.05939, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. International Conference on Machine Learning, pp.
Improved denoising diffusion probabilistic models.
PMLR, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feodback. Advances Neural Information Processing Systems, 11227700227444 2022.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.0373) 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highhresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 36494, 2022.
Tim Salimans and Jonathan Ho. Progrossive distillation for fast sampling of diffusion models. orXiv preprint ar arXiv:2202.0051: 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint or Xio:2311. 17042, 2023.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large- scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 55:25278-25294, 2022.
Eyal Segalis, Dani Valevski, Danny Lumen, Yossi Matias, and Yaniv Leviathan. A picture is worth thousand words: Principled recaptioning improves image generation. arXiv preprint arXiv:2310.16656 2023.
Andy Shih, Suncel Belkhale, Stefano Ermon, Dorsa Sadigh, and Nima Anari. Parallel sampling of diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Jaskirat Singh and Liang Zheng. Divide, evaluate, and refine: Evaluating and improving lext-lo-image alignment with iterative vqa feedback. Advances in Newral Information Processing Systems, 36, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised lcarning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Jiaming Song, Chenlin Mong, and Stefano Ermon. Denoising diffusion implicit models. ar preprint arXiv:2010.088 02502, 2020a.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. orXiv preprint arXiv:2300. 1499 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Yang Song, Prafulla Dhariwal, Mark Chen, and Nya Sutskever. Consistency models. International conference on machine learning, 2023.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-300 100083021,-200 2020.
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, et al. Dreamsyne: Aligning text-to-image generation with image understanding feedback. arXiv preprint arXio:2811.19946 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu,, Jean- Baptiste Alayrac, Jiahui Yu, Radu Sori- cut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint (Tt:::61000242.. 2016.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambort, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-at diffusion models. https://github. com/ huggingface/ dificeer/ffffssrrrs 2022.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint ar Xim:2311. 12908, 2023a.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXio:2311.12900 2023b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprinl Xiv:23306 09341, 2023a.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF Internalional Conference on Computer Vision, pp. 2096-2100 2105, 2023b.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong, Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Newral Information Processing Systems, 36, 2024.
Shentao Yang, Tianqi Chen, and Mingyuan Zhou. A dense reward view on aligning text-to-image diffusion with preference. arXiv preprint arXiv:2402.000: 08265, 2024.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexan- der Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to- image generation. Xiv preprint arXic:2206. 10789, 2(3):5, 2022.
Kexun Zhang, Xianjun Yang, William Yang Wang, and Lei Li. Redi: Efficient learning-free free diffusion inference via trajectory retrieval. arviv preprint ar arXiv:2302. 02285, 2023.
Qinsheng Zhang and Yongxin Chen. Fast sampling diffusion models with exponential integrator. arXiv preprint ar arXio:2204.1800 18902, 2022.
Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint 0rXiv:4401.12444 2024.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference Machine Learning, pp. 42390- 42402. PMLR, 2023.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial anto-cncoders. arXiv preprint arXic:2202.0000 2022.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. Fino-tuning language models from human preferences. arXiv preprint urXiv:1909.08500 2019.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Appendix
In the main paper, distill our RG-LCD from the Stable Diffusion-v2.11 (768 768) (Rombach et al. 2022 Fig. 8 further shows the samples from our RG-LCM (HPSv) distilled from Stable Diffusion-v2. 1221-bbss base (512
x 512)
(Rombach et al. 2022 The rest of the appendix is structured as below
Appendix A details the experimental setup and hyperparameter configurations.
Appendix B elaborates on the training processes and sampling techniques from a (latent) CM.
Appendix C shows extra samples generated by various models.
18

---

Published in Transactions on Machine Learning Research (10/2024)
1-step
2-step
4-step
Figure 8: Samples from our RG-LCM (HPSv2.1) with the teacher Stable Diflusion --.seb The renolution
is 512 x 512.
19

---

Published in Transactions on Machine Learning Research (10/2024)
CLIPScore
PickScore
ImageReward
HPSv2.1
3
5.0
5.0
1.0
1.0
Table 2: 8 for different RG-LCMs when training with different RMs.
A Additional Experimental Details and Hyperparameters (HPs)
For qualitative evaluation, we ensure consistency across all methods by using the same random seed for head-to-head image comparisons.
As mentioned in Sec. 5 our training are conducted on the CC12M datasets (Changpinyo et al. 2021 the
datasets (Schuhmann et al. 2022) used in the original LCM paper (Luo et al. 2023a) are not accessible] We train all LCMs (including RG-LCMs and the standard LCM) by distilling from the teacher LDM Stable Diflusion-v2.1 (Rombach el al. 2022) for 10K gradient steps on 8 NVIDIA A100 GPUs. When learning the standard LCM, we use a batch size 32 on each GPU (256 effective batch size). For RG-LCMs, we use a batch size on each GPU (40 effective batch size). Interestingly, we observe that different batch sizes do not impact the final performance too much.
We use the same set of hyperparameters (HP) for training RG-LCM and the standard LCM by following the settings listed in the diffusers (von Platen et al. 2022) library, except that RG-LCM has unique HP 3. Specifically, we set the learning rate le - 6, EMA rate μ 0.95 and the guidance scale range [max] [5, 15]. We include more training details in Appendix A Following the practice in (Luo et al., 2023a), we initialize fo with the same paramcters as the teacher LDM. We further encode the CFG scale w by applying the Fourier embedding to w and integrate into the LCM backbone by adding the projected w-embedding into the original embedding as in (Meng al. 2023).
As mentioned in Sec. 3.3 use DDIM (Song et al. 2020a) our ODE solver vp with skipping step k == 20, the formula of the DDIM ODE solver VDDIM from Luth to bn is given below (Luo et al. 2023a
Ctt Ztntk
Btenss Ctt,
1) En (Zenger C, bn4x) -n
VDDIM (Zingee bntks bn: c)
Brn
(13)
Ot.,
Cit, in Ben
DDIM Estimated Ze..
where EN denotes the noise prediction model from the teacher LDM, at,, and Be, specify the noise schedule. For the forward process SDE defined in equation 1 we have
d dlog((() dt
d)
d dlogga()) (t).
a(t)
2
(14)
dt
dt
As a result, we have (x(x:) = (x)) We refer interested readers to the original LCM paper Luo
ct al. 2023a) for further dotails.
Reward scale 3 for different RG-LCMs with different RMs. In Sec. 5.1 and 5.2 we train our
RG-LCMs with different RMs, including CLiPScore (Radford et al. 2021), PickScore (Kirstain et al. 20224), ImageReward (Xu et al.) 2024) and HPSv2.1 (Wu et al.) 2023a Table 2 shows the B we used for different
RMs when obtaining the results in Fig. 5 and Table 1.
Details for integrating an LRM into RG-LCM As discussed in Sec. 4.2 the LRM admits similar architecture as the CLIP (Radford et al. 2021 model, with the distinction of replacing the visual encoder with a latent encoder. We retain the original pretrained text encoder and focus on pretraining the latent encoder from scratch. This process mirrors the CLIP's pretraining approach, minimizing the same contrastive loss on the CC12M datasets (Changpinyo al., 2021 The image latent extracted with the same VAE encoder used in the teacher LDM Stable Diffusion-v2. Upon completing the pretraining phase, the LRM demonstrates promising initial results, achieving zero-shot Top-1 Accuracy of 38.8% and Top-5 Accuracy of
https: http:/aaii...ii/ttssslliiaaooooo
20

---

Published in Transactions on Machine Learning Research (10/2024)
66.47% on the ImageNet validation set (Deng et al, 2009 These metrics underscore the model's fundamental capability in understanding text-image alignments.
During the RG-LCD process, we finetune the LRM to match the preference of an expert RM. We train the last layer the latent encoder and the last 5 layers the text encoder. We set the learning rate to 0.0000033 following (Wu al. 2023a). Note that we do not perform heavy HP searches to determine their optimal values.
As are finetuning our LRM, there is a potential risk of overfitting the model to the training datasets, which could degrade the quality of generated outputs if training continues indefinitely. We emphasize that this is unlikely to pose a problem for our method. In practice, we use large and diverse text-image dataset, such as CC12M. We also fixed the training to 10K iterations and observed stable performance without encountering any training instability. We hypothesize that performance degradation would only occur training exceeds one full epoch of the datasct. However, even with a large batch size of 256, one epoch would require 12M / 256 47.9K iterations, which far beyond the 10K iterations we used. Therefore, early stopping is not critical concern for approach.
Nonetheless, we could still implement a stopping criterion by monitoring the average rewards of training batches. We can stop the LRM training when the average rewards converge to specific value, ensuring the LRM is not overtrained.
21

---

Published in Transactions on Machine Learning Research (10/2024)
B Training and Sampling from (Latent) CM
B.1 Multistep sampling from learned CM and LCM
We provide the pscudo-codes for multistep consistency sampling (Song et al. 2023 and multistep latent consistency sampling {Luo et al. 2023a) procedures in Algorithm [ and A Algorithm respectively. The multistep sampling procedures alternate between the consistency mapping and noise- injection steps, trading additional computation resources for better sample quality. In the n-th iteration, we first perturb the predicted sample x (or z) with Gaussian noise obtain &r, (or 2.,). We then map the noisy sample & (or 2,,) to obtain new (or 2).
Algorithm 2 Multistep Latent Consistency Sam-
Algorithm Multistep Consistency Sampling
pling
Require: CM fe, steps N, timestep sequence TI
Require: LCM fo, steps N, timestep sequence TI 72 >
T2 > .... TN-1, noise schodule at)), (t)).
TN-1, noise schedule a(t), 3(t), text prompt CFG scale w, VAE decoder D. Sample initial noise 2T (0,
Sample initial noise äT ~ IO,
x fo(kr,T)
for -.....,, N....,N-11.. do
2 + fe(22, w, c)
Sample &r, ~ ~N(a(T))x
for n n= 1= 1...,,..NN N do
x x Tn)
Sample 27 ~ (((((TT) (z, 3 (T,)
end for Return x
2 2 fo w, c,
end for Return D(z)
B.2 Training procedures of RG-LCD
Algorithm 3 Reward Guided Latent Consistency Distillation
Require: dataset D, initial model parameter 0, learning rate 7, ODE solver V, distance metric d, EMA rate μ, learning rate 7,, noise schedule att), B(t,, guidance scale [Wmins Wmax], skipping interval k, VAE encoder
E, decoder D, reward model R, reward scale 8
Encoding training data into latent space: D: {(2, c) == E(xr), (x. ED
0 + 0
repeat
Sample (z,c) ~ Dz, n U[I,N - k] and w ~ [wminn Wmax]
Sample Ztnth N (a (fn+x) 2; o² (In+s)
20w Zens + (1 + WI (Zenpe) botks bm; e) wi (ztwee ba k,, 23)
L (0, (0000::0 1) + d fe (Ztmazs W. e, (.tkt)) fo a
wee,tt) 3. R (D (fo (Z e, tn+x)) cc)
0 - 0 - (0,00
0- + stop_grad (10 + (1 11-11) 1))
until convergence
Algorithm 3 presents the pseudo-codes for our RG-LCD. We use the red color to highlight the difference between our RG-LCD and the standard LCD (Luo et al. 2023a
B.3 Training procedures of RG-LCD with Latent Proxy RM
Algorithm g
presents the training codes for our RG-LCD with Latent Proxy RM.
22

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 4 Reward Guided Latent Consistency Distillation with Latent Proxy RM
Require: dataset D, initial model parameter 0, learning rate v, ODE solver V, distance metric d, EMA rate μ. learning rates y f22 noise schedule a(t), [(t), guidance scale Wiin Wmax]: skipping interval k, VAE encoder &, decoder D, reward scale B, expert RMM RE, LRM R. temperature TE and TL
Encoding training data into latent space: D: {(z, c) E(x), (x
0- +
repeat
# Calculate the Training loss for fo
Sample (z, Dz... n U[1, N k] and w [iminn Wmax]
Sample Zt,t N (a (In+k) 0; (In+k) I)
2 + 2t,1k + (1 W (Zinju bntks bn; c) wi (2t, in batkki In, R)
Detach the parameter a of R
C (0,0-; VV,)) < d (Fo (2 e, batx) So (2t.. ,w, (.,csha)) B RE (2e c)
# Calculate the Training loss for Re
Zo Z, Z1 + fo (Zinte w, c, tntk), 22 t fo- (2) ( (0,
for i E 10, do
for j (i, do
Derive the preference distribution: piy(m) (m) x exp (Rl (Zm, c)/Tl), m E {i,}}
Derive the preference distribution: (i)(m) exp (RE (D (2m) , c) /TE), E (i,5) end for end for
LRM(O)+ + ELo ELO Ej=41 DKL (stoppsstpgraad (Qiy))
# Update the learnable parameters via gradient descent
- - 7/2 VoL 0-)
0- + stop_stteeeep _arr__gadddd (no (1 - p)
o 1/2 (oRM(())
until convergence
23

---

Published in Transactions on Machine Learning Research (10/2024)
C Additional Qualitative Results
We provide additional samples from our RG-LCMs with different RMs compared with the baseline LCM and teacher Stable Diffusion v2.1 in Fig. g
and 10
The prompts for images in Fig. 9 in the left-to-right order are given below
Van Gogh painting of teacup the desk
Impressionist painting of a cat, textured, hypermodern
photo of a kid playing, snow filling the air
A fluffy owl sits atop stack of antique books in detailed and moody illustration.
a deer reading a book
photo of monkey wearing glasses in suit
The prompts for images in Fig. 10 in the left-to-right order are given below
ornate archway inset with matching fireplace in room
Poster of a mechanical cat, techical Schematics viewed from front
portrait of a person with Cthulhu features, painted by Bouguereau.
serene nighttime cityscape with lake reflections, fruit trees
Teddy bears working on new AI research on the moon in the 1980s.
A cinematic shot of robot with colorful feathers
Fig. 11 presents image samples when integrating a latent proxy RM (LRM) into our RG-LCD procedures.
The prompts the left-to-right order are given below
man in brown blazer standing in front of smoke, backlit, in the style of gritty hollywood glamour, light brown and emerald, movie still, emphasis on facial expression, robert bevan, violent, dappled
a cute pokemon resembling a blue duck wearing a puffy coat
Highly detailed photograph of meal with many dishes.
Fig. 12 further qualitatively compares RG-LCM (HPS) + LRM and the standard LCM. The prompts in the top- tostto-botiee bottom order are given below
(Masterpiece:1. 5), RAW photo, film grain, (best quality:1. 5), (photorealistic), realistic, real picture, intricate details, photo of full body cute cat in medieval warrior costumo, ((wastclands background)), diamond crown on head, ((dark background))
back view of woman walking at Shibuya Tokyo, shot on Afga Vista 400, night with neon side lighting
Fall And Autumn Wallpaper Daniel Wall Rainy Day In Autumn Painting Oil Artwork
bird-eye shot photograph of New York City, shot on Lomography Color Negative 800
painting of forest and pond
24

---

Published in Transactions on Machine Learning Research (10/2024)
SD v2.1 50-steps
LCM 4-steps
R6-LCM (CLIP) 2-step
Mostegellle Moork
Arhro Mosany
RG-LCM (CLIP) 4-step
Phart
Mosua itsktop
Mevech
RG-LCM (Pick) 2-step
RG-LCM (Pick) 4-step
RG-LCM (HPS) 2-step
RG-LCM (HPS) 4-step
RG-LCM (ImgRwd) 2-step
RG-LCM (ImgRwd) 4-step
Figure 9: Samples from our RG-LCMs with different RMs compared with the baseline LCM and teacher Stable Diffusion v2.1.
25

---

Published in Transactions on Machine Learning Research (10/2024)
SD v2.1 50-steps
LCM 4-steps
RG-LCM (CLIP) 2-step
RG-LCM (CLIP) 4-step
RG-LCM (Pick) 2-step
RG-LCM (Pick) 4-step
RG-LCM (HPS) 2-step
RG-LCM (HPS)
4-step
RG-LCM (ImgRwd) 2-step
RG-LCM (ImgRwd) 4-step
Figure 10: More samples from our RG-LCMs with different RMs compared with the baseline LCM and teacher Stable Diffusion v..1.
26

---

Published in Transactions on Machine Learning Research (10/2024)
RG-LCM (ImgRwd)
RG-LCM (ImgRwd) LRM
RG-LCM (Pick)
RG-LCM (Pick) +LRM
RG-LCM (HPS)
RG-LCM (HPS)
+LRM
Figure 11: Effect of the Latent proxy RM (LRM). Integrating the LRM into RG-LCD procedures makes the generated images natural, corresponding to the lower FID in Table Moreover, it helps eliminate the high-frequency noise in the generated images.
27

---

Published in Transactions on Machine Learning Research (10/2024)
RG-LCM (HPS) + LRM
LCM
BIGER
Figure 12: Comparison between RG-LCM (HPS) + LRM with LCM. The samples from RG-LCM (HPS) +
LRM are visually appealing while remaining natural, corresponding to the high HPSv2.1 score and low FID in Table
28

---

Published in Transactions on Machine Learning Research (10/2024)
LCM
RG-LCM
SPAAGLE BUND
GURGODEE
TOURY T000
HPSv2.1
HPSv2.1 10000
PickScore
PickScore B
CLIPScore $ B=5
CLIPScore
f =
Figure 13: Additional images to study the impact of the reward scale 3. We generate all samples with 4 steps.
Fig. includes additional samples for the ablation on the reward scale B. The prompts in the top-to-bottom bottom order are given below
Ultra realistic photo of single light bulb, dramatic lighting
Pirate ship trapped in cosmic maelstrom nebula
golden retriever wearing VR goggles.
Highly detailed portrait of woman with long hairs, stephen bliss, unreal engine, fantasy art by greg rutkowski.
A stunning beautiful oil painting of lion, cinematic lighting, golden hour light.
29

---

Published in Transactions on Machine Learning Research (10/2024)
A raccoon wearing tophat and suit, holding briefcase, standing in front of city skyline.
Fig. 14 and 15 provide additional qualitative comparisons for RG-LCM (CLIP) with / e (5, 100), RG-L.CM (HPS) with 3 1, and RG-LCM (HPS) LRM with 3 100.
The prompts for Fig. 14 in the top-to-bottom order are given below
game screenshot featuring Woolie Madden with dreadlocks in Mass Effect.
Two girls holding hands while watching the world burn
A full-body portrait of female cybered shadowrunner with dark and cyberpunk atmosphere created by Echo Chernik in the style of Shadowrun Returns PC game.
A portrait of skeleton possessed by spirit with green smoke exiting its empty eyes.
A counter in a coffee house with choices of coffee and syrup flavors.
The prompts for Fig. 15 in the top-to-bottom order given below
Black and white portrait of Thabo Mbeki with highly detailed ink lines and a cyberpunk flair, created for the Inktober challenge as part the Cyberpunk 2020 manual coloring pages.
The image features big white cliff, cargo favela, wall fortress, neon pub, and some plants, with vivid and colorful style depicted in hyperrealistic CGI.
An albino lion wearing Mafia hat, digitally painted by multiple artists, trending Artstation.
A galaxy-colored DnD dice is shown against a sunset over sea, in artwork by Greg Rutkowski and Thomas Kinkade that is trending on Artstation.
A pirate skeleton.
Fig. 16 additional quantitative results for RG-LCM (ImgRwd) with images resized to low resolution of 224x224. Please use Adobe Acrobat Reader and set the zoom to 100% (actual size). At this resolution, high-frequency noise becomes less noticeable. The prompts, listed in order from top to bottom and left to right, are provided below:
A creepy cartoon rabbit wearing pants and shirt, with dramatic lightning and cinematic atmo- sphere.
A beaver in formal attire stands beside books in a library.
A pencil sketch of Victoria Justice drawn in the Disney style by Milt Kahl.
Portrait of a male furry Black Reindeer anthro wearing black and rainbow galaxy clothes, with wings and tail, in an outerspace city at night while rains.
galaxy-colored DnD dice shown against a sunset over a sea, artwork by Greg Rutkowski and Thomas Kinkade that trending on Artstation.
Architecture render with pleasing aesthetics.
An empty road with buildings each side.
A computer monitor glows wooden desk that has black computer chair near it.
A man standing by his motorcycle is looking out to take in the view.
A koala bear dressed as ninja in a kayak.
Baby Yoda depicted in the style of Assassination Classroom anime.
puppy is driving a car in a film still.
30

---

Published in Transactions on Machine Learning Research (10/2024)
CLIP Score, B =
CLIP Score, B = 100
HPSv2.1, B =
HPSv2.1 1, B = 100, LRM
DRESA
DRASA
ViON
BAIRN
WURE
BIRRBIONN
BURLIK
BAYR
BAIRR
3R WERR
BER SUR
RIN
B
BURN
BURB
BLOEEN
DAIRR
BURN
BURRIN
BUFFIN
BURN
BUR
SHOLAWOED HARBEE
SHHAOW
SHIMA AREEE
EECHH
SHHAWGEN
SHAHAOWER
AHE WHONMEN
SHAAWWERR REROWER!
SHAPPWINS
SHANWIER
ECHCHO
SHRIVO
CREWHER
Sprin
COVE COFR
CROFFEE CooffE
COFFE HOPFE
S ST
COFFFE COFFFER GHOFFF CHOFEE:
SEUPPE
COFFEE
COFFE COFFEE
CoofeeR A Coffee
Catres
COFFE COFFEE
CHOFEESE OPFISS COFFEEE
Car
& FP
OFFEE
& & SFEOP
CoFe Corning
Cartes Crlp
COFFFES & Cuerries
GATIE COFFEE
COFFE HOCEE CRAFFEE
LofFE
COrTS SAA
CUTTALL
CofFE
PV
Copple CATP
COFFEE
COFFEES
CL COFLE SPON
CAFIP
CAUREE
Figure 14: We present additional qualitative comparisons between RG-LCM (CLIP) with 3 E {5, 100). RG-LCM (HPS) with I = 1, and RG-LCM (HPS) + LRM with B = 100.
31

---

Published in Transactions on Machine Learning Research (10/2024)
CLIP Score, 5
CLIP Score, B 100
HPSv2.1, B =
HPSv2.1, & = 100, LRM
TERS
MEBER
THE CARRGIO
HIPEE
CRYPTESS DE
CLAT
HIWEO'
WATRE WAIRETLLLLLT
CLUFE
WELL BARAGO
CELIMS
ANIN ARRA
APAA NALAFF
MMAAFIFA
MAFW
MARHAA
MALFFTUINA
AB aherre
SALLAY
A
AT GARY HANK
IDN
Figure 15: We present additional qualitative comparisons between RG-LCM (CLIP) with 3 E {5, 100).
RG-LCM (HPS) with I = 1, and RG-LCM (HPS) + LRM with B = 100.
32

---

Published in Transactions on Machine Learning Research (10/2024)
S
Figure 16: Additional quantitative results for RG-LCM (ImgRwd) with images resized to low resolution of 224x224. Please use Adobe Acrobat Reader and set the zoom to 100% (actual size). At this resolution, high- frehffreyenccy noise becomes less noticeable.
33

---

Published in Transactions on Machine Learning Research (10/2024)
D Experiments with Additional Teacher T21 Models
In this section, we conduct additional experiments using different teacher T2I models, including Stable Diffusion 1.5 (SD 1.5) and Stable Diffusion XL (SDXL). For each teacher model, we train both the baseline LCM and our RG-LCM (HPS) by learning from 1PSv2.1, using DDIM as defined in equation 13 as our ODE solver V. The CC12M (Changpinyo et al. 2021) dataset serves as our training dataset. For SDXL, due to GPU memory constraints, we apply LCM-LoRA [Luo et al., 2023b) on top of SDXL to construct our RG-LCM and baseline LCM instead of performing full-model training. Additionally, we fix the weighting to 1, consistent with the settings for Stable Diffusion 2.1.
parameter B in equation 11
Empirically, we evaluate different methods using 3,200 HPSv2 test prompts and employ VIEScore (Ku et al. 2023) as our evaluation metric with the GPT4o backbone. VIEScore achieves a high Spearman correlation of 0.4 with human ovaluations, close to the human-to-human correlation of 0.45. Given a tcxt-image pair, VIEScore provides Semantic Score, Quality Score, and Overall Score, reflecting text-image alignment, visual quality, and overall human preference, respectively. We compare the 4-step generation of our RG-LCM with the 4-step generation of the baseline LCM, well with the 4-step and 25-step generations from the teacher T2I model using DPM-Solver++ (Lu et al. 2022b) with CFG guidance (Ho & Salimans
2022 and
negative prompts. DPM-Solver is a high-order fast ODE solver that accelerates inference from diffusion models. It is important to note that we can also integrate DPM-Solver++ and negative prompts into our RG-LCM training. We leave it for future work.
Table and present the evaluation results. In both cases, the 4-step generation of our RG-LCM (HPS) outperforms other 4-step baselines. When using SD 1.5 as the teacher model, our 4-step generation even surpasses the 25-step gencration achieved using DPM-Solver++ with CFG and negative prompts. With SDXL the teacher model, our 4-step generation slightly underperforms but still matches the 25-step generation from the teacher. We believe this performance drop may be due to 1) the LoRA training and 2) the absence of high-quality image datasets. Therefore, we expect our RG-LCM to perform even better with full-model training and access to datasets with high image aesthetics, e.g. Aestheis V2
6.5+ (Schuhmann al. 2022
SD-1.5 as the Teacher Model
NFEs
Semantic Score
Quality Score
Overall Score
DPM-Solver+++ Negative Prompt +, Negative Prompt
6.06
4.98
5.23
25
6.77
6.63
6.45
LCM
6.75
5.88
6.02
RG-LCM (HPS)
7.55
7.02
7.11
Evaluation of different methods using Stable Diffusion 1.5 as the teacher model on the HPSv2 test prompts. NFEs denote the number of function ovaluations during inference. We employ VIEScore as the evaluation metric. By learning from the feedback of HPSv2.1, the 4-step generation of our RG-LCM (HPS) not only outperforms other 4-step baselines but also surpasses the 25-step generation achieved using DPM-Solver++ when sampling from the teacher model with CFG and negative prompts.
Table 3:
SDXL as the Teacher Model
NFEs
Semantic Score t Quality Score 1
Overall Score t
DPM-Solver++, Negative Prompt
4
6.63
4.99
5.52
DPM-Solver+++, Negative Prompt LCM RG-LCM (HPS)
25
8.23
7.73
7.83
4
7.26
6.43
6.65
8.1
7.46
7.64
Table 4: Evaluation of different methods using Stable Diffusion XL the teacher model on the HPSv2 test prompts. NFEs denote the number of function evaluations during inference. We employ VIEScore as the evaluation metric. By learning from the feedback of HPSv2.1, the 4-step generation of our RG-LCM (HPS) not only outperforms other 4-step basclines but also matches the performance of the 25-step generation when using + to sample from the teacher model with CFG and negative prompts.
34


Reviews:
[{"review_id": "f765tLSmmx", "reviewer": "Reviewer_ZfZw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- It is good to see that the suggested algorithm is very clear while performing good quality.\n\nWeakness\n- It is unclear why latent reward function can resolve the high-frequency issue presented in Figure 3. Simply saying replacing the evaluation from pixel space to latent space solves the issue does not make sense to me. I need more experimentally grounded or theoretically solid evidence for this. This is not the novelty concern. This is rather the correctness and completeness concern.\n- Let's suppose that the CLIP text encoder is infinitely flexible and we have amazing optimizer. If my understanding is correct, if $\\tau_{L}$ and $\\tau_{E}$ are the same, then the optimal solution of $L_{RM}(\\sigma)$ (when we optimize Eq. (12)) should be $R_{\\sigma}^{L}(z,c)=R^{E}(D(z),c)$. In that optimal solution, Figure 3 will happen again. Correct me if I'm wrong.\n- Do you have any experimental result if you fine-tune too much the CLIP text encoder? If it falls into the global optimum, $L_{RM}(\\sigma)=R_{\\sigma}^{L}(z,c)=R^{E}(D(z),c)$, and Figure 3 will happen again. What is happening in the optimization procedure? Either algorithmic explanation would also be possible if it is plausible.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces fine-tuning the few-step generator with auxiliary loss that is computed at the latent space. The proposed auxiliary loss, the negative of reward surrogate function, deviates the denoising trajectory from the trajectory from PF-ODE, in preferable directions.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. ImageGen -> Imagen\n2. Eq. (11) should be re-written for the reward surrogate function at the latent space. It is extremely confusing to understand.\n3. I don't understand Eq. (12). What does i and j denote for?\n4. Replace $\\sigma$ into $\\phi$ or something else that most people get immediately as the parameter. $\\sigma$ is usually designated for the noise level in diffusion community.\n5. What does $E$ stand in $R^{E}$? Please recheck all terms carefully and define properly.", "broader_impact_concerns": "The deep fake generation is always on the concerning list in this kind of works. However, it would be unfair to impose such ethical concern for this particular paper. I would say there is no ethical concern for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wnGXdzPNYL", "reviewer": "Reviewer_a19z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel method, RG-LCD, which addresses the quality loss issue in LCD and achieves high-quality image generation with significant inference acceleration.\n2. The use of a reward model and the integration of feedback from the RM into the LCD process are innovative and practical approaches to improve the sample quality. The introduction of a latent proxy RM (LRM) as an intermediary to connect the LCM with the RM is a clever solution to avoid reward over-optimization.\n3. The experimental results demonstrate the effectiveness of the proposed method, with improved image quality and outperformance of the baseline LCM.\n\nWeakness:\n1. Clarify the Training Process of Latent Proxy RM: The paper does not clearly explain the process of training the Latent Proxy RM, making it difficult to understand the exact steps involved. A more detailed and structured explanation or pseudo code would help in comprehending how the LRM is trained and integrated into the RG-LCD framework.\n2. The reward function appears to be heuristic, with many available options that significantly impact the results. We hope author provide more discussion on this part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Reward Guided Latent Consistency Distillation (RG-LCD), a method that integrates feedback from a reward model (RM) into the Latent Consistency Distillation (LCD) process for text-to-image synthesis. The authors propose compensating for the quality loss of LCD by aligning the output of the latent consistency model (LCM) with human preference during training. They demonstrate that when trained with the feedback of a good RM, the RG-LCM achieves high-quality image generation with a significant inference acceleration. They also propose the use of a latent proxy RM (LRM) to overcome reward over-optimization. Experimental results show that incorporating the LRM into RG-LCD improves image quality and outperforms the baseline LCM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. We hope the author can provide detailed training process of latent proxy RM. Including the pseudo code would help in comprehending how the LRM is trained.\n2. Provide more discussion on how to choose the reward function and its influence.", "broader_impact_concerns": "The broader impact of this work is generally positive, as it enables faster and more efficient generation of high-quality images, which can benefit various applications, from digital art to content creation. However, there are some concerns regarding the potential misuse of such technology in generating misleading or harmful content, especially if the reward models are not properly aligned with ethical considerations. We hope author can provide more insightful discussion in this part.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X3HScwYBJc", "reviewer": "Reviewer_UYSc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pons:\n\n- The idea of involving reward in the LCM distillation process is interesting. The core contribution is to address the reward overestimation issue by the use of a latent proxy RM (LRM). The technical novelty is moderate.\n\n- The proposed strategy can work on the adopted models and achieve superior performance compared to the original LCM.\n\n- User studies are performed on PartiPrompt and HPSv2 test sets for each baseline comparison. The ablation\n\nCons:\n\nMy main concerns are raised from the evaluations.\n\n- The baselines are the SDv2.1 plus DDIM and LCM. While the best performance is usually achieved based on some other base models (SDXL, SD1.5), samplers (such as DPM-like samplers), and some other wildly used techniques (such as CFG with certain negative prompts). I wonder if the proposed method can achieve comparable performance on the commonly used settings, or it only works on the standard  SDv2.1 + DDIM/LCM settings which are quite limited for the community.\n\n- It is mentioned in Table 1 that “Except trained with CLIPScore, our RG-LCMs achieve better”, where I suppose the CLIP reward model can achieve good results without the proposed RG-LCMs. Besides, as illustrated in Figure 6, I think the CLIPScore with Beta=100 achieves the best quality. HPSv2.1 leads to color over-saturation issues, and PickScore leads to haze issues. More visual results on the comparisons between RG-LCM (CLIP), RG-LCM (CLIP) + LRM, RG-LCM (HPS), and RG-LCM (HPS) + LRM would be helpful.\n\n- It is also questionable if the HPSv2.1 metric can reflect the quality of different method results. Besides, the HPSv2.1 score is used to boost model performance in the RG-LCM (HPS) + LRM. It could be a bit unfair in this evaluation.\nHuman evaluation is a good solution. I would also recommend using VLM models for evaluation (like GPT4V or miniGPT4 adopted in [1])\n\n- It is unclear what the D means in Section 4.1. The explanation should be provided following equation (10).\n\n[1] T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation\n\nMinor:\n\nI suppose there is a typo in section 3.1: where µ(·) and µ(·) are the drift and diffusion", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work addresses the quality loss by aligning LCM’s output with the human preference reward model during training. A reward-guided latent consistency distillation framework is proposed to enhance the LCM with reward model feedback. The core contribution addresses the reward overestimation issue due to direct optimization with the gradient from the RM. With the proposed method, 2-step diffusion inference can achieve better results than the classical 50 steps on SDv2.1.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Comparisons on some other base models (SDXL, SD1.5), samplers (such as DPM-like samplers), with some wildly used techniques (such as CFG with certain negative prompts).\n\nMore visual results on the comparisons between RG-LCM (CLIP), RG-LCM (CLIP) + LRM, RG-LCM (HPS), and RG-LCM (HPS) + LRM.\n\nA better evaluation metric would be more convenient.", "broader_impact_concerns": "The impact statement has been discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are all in agreement that the paper has substantial novelty and significance, and all remaining concerns/questions were addressed adequately during the discussion period.
Decision: Accept (poster)